Extract, Transform, Load (ETL)

Parallel Processing of Large Volume ETL Jobs

  • Article

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

5,447 reads

Blogs

Fixing the PoSh Hotkey in ADS

By

I installed the PowerShell (PoSh) extension in ADS recently. This is available in the...

Capture Execution Plans Only For Long Running Queries

By

I love questions. Most of all, I love questions I can answer. I spotted...

Daily Coping 28 Sep 2020

By

I started to add a daily coping tip to the SQLServerCentral newsletter and to...

Read the latest Blogs

Forums

Keeping Your Staff

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Keeping Your Staff

Backing up to Azure

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Backing up to Azure

When one of your DBA colleagues leaves the company, what is your checklist?

By Cláudio Silva

Comments posted to this topic are about the item When one of your DBA...

Visit the forum

Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers