Raghavendra Narayana

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

5,171 reads

Blogs

Azure SQL Database – Error Logs?

By

Ok, so Azure SQL doesn’t really have its own error log based somewhere on...

Pluralsight Free April

By

It’s a few days into April, but not too late I hope to mention...

Daily Coping 7 Apr 2020

By

I’ve started to add a daily coping tip to the SQLServerCentral newsletter, which is...

Read the latest Blogs

Forums

For loop in C# script component

By rohitkocharda

Hi Everyone,I have a script task calling a sequence container. What i would like...

More Time to Certify

By Steve Jones - SSC Editor

Comments posted to this topic are about the item More Time to Certify

Difference Between Lists

By BTylerWhite

Comments posted to this topic are about the item Difference Between Lists

Visit the forum

Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers