Raghavendra Narayana


SQLServerCentral Article

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

6,585 reads

Blogs

WIT/DEI Mental Health and Wellness Day Challenge to Present

By

Back in December 2019, I started presenting my session on mental health with lots...

Daily Coping 24 Mar 2023–The Final Tip

By

Today’s coping tip is to discover the joy in the simple things in life....

Experimenting with RedisGraph on Docker and Windows

By

I have been experimenting a bit with graph databases, trying to learn more about...

Read the latest Blogs

Forums

dbatools - Connect-DbaInstance

By rothj

I have my SQL servers name, UserID (UID) and Password (PWD) in a table...

Extracting the string before and after the given character

By sathishkm

Hi All, I want to extract the string before and after the given characters....

BULK INSERT CSV with a fixed value for a field not in the CSV

By PS_Richard

Well, I hope the title explains what it is.  Currently, I have the following...

Visit the forum

Ask SSC Logo Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers