Extract, Transform, Load (ETL)

Parallel Processing of Large Volume ETL Jobs

  • Article

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

5,030 reads

Blogs

Check Query Progress with Live Query Stats

By

This is something I touched on back in 2017 a little after the Live...

SQL Server 2019 on Docker no longer runs as root by default

By

In my home lab I have an Ubuntu virtual machine that runs both SQL...

SQL Server 2019 – Compatibility Level in Azure

By

I am not sure when this became available but for Azure SQL Database 150...

Read the latest Blogs

Forums

Is Data the Future of the Vibrant Web?

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Is Data the Future of...

SSIS catalog deployment from source server to target - powershell

By iKvfpMWsaoC1Gy8

I am looking for a good solution for automating the deployment of SSIS packages...

XML Data singleton value not fixed????

By jbalbo

Hi Hope to explain this correctly I have some XML data that can vary...

Visit the forum

Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers