Raghavendra Narayana

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

5,023 reads

Blogs

Using Azure Data Factory Mapping Data Flows to populate Data Vault

By

(2019-May-24) Data Flow as a data transformation engine has been introduced to the Microsoft Azure...

New Azure “SQL Server settings” blade in the Azure Portal

By

I just noticed today that there is a new blade in the Azure portal...

Mass Backup All Sessions

By

Migrating Extended Event Sessions from one server to another should be a simple task....

Read the latest Blogs

Forums

The Change Failure Rate

By Steve Jones - SSC Editor

Comments posted to this topic are about the item The Change Failure Rate

Buffer Pool calculation

By avpco

Hello Colleagues. Could you tell me about how MS SQL Server 2016 SE determines...

SSIS Scale out deployment in a high availability cluster

By KobusV

Good day all, Just wondering if it will be possible to have a scale-out...

Visit the forum

Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers