Raghavendra Narayana


Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

5,715 reads

Blogs

Goal Progress–Feb 2021

By

I set goals at the beginning of the year, and I’m tracking my progress...

How to: exec sp_help on temp tables?

By

Say you have a temp table and you want to see the columns names. For...

Daily Coping 26 Feb 2021

By

I started to add a daily coping tip to the SQLServerCentral newsletter and to...

Read the latest Blogs

Forums

Why are DBAs not paid for knowledge and other IT Professionals are?

By Engineer

DBAs are being asked to know everything for 6 figures which cap out for...

Common Table Expression Misunderstandings

By Kathi Kellenberger

Comments posted to this topic are about the item Common Table Expression Misunderstandings

Distributed AG DB movement

By Syed_SQLDBA

I have a DAG between On prem and Azure. We have a maintenance window...

Visit the forum

Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers