Raghavendra Narayana

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

5,428 reads

Blogs

Daily Coping 18 Sep 2020

By

I started to add a daily coping tip to the SQLServerCentral newsletter and to...

Optimize for Unknown for Inline Table-Valued Functions

By

I had a curious performance issue today. An inline table-valued function (iTVF) was performing...

Daily Coping 17 Sep 2020

By

I started to add a daily coping tip to the SQLServerCentral newsletter and to...

Read the latest Blogs

Forums

SQL Query - FULL JOIN multiple tables but return NULL results

By glennyboy

Good day! Need help with my sql query code; first FULL JOIN "ILEtransfer" no...

Coping with a working life as an imposter

By Phil Factor

Comments posted to this topic are about the item Coping with a working life...

SQL Select with condition

By Dee Dee-422077

Hi, In my sql table,  there is a column which have string of text...

Visit the forum

Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers