Raghavendra Narayana


SQLServerCentral Article

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

6,309 reads

Blogs

Scary Scalar Functions - Part Two: Performance

By

Scary Scalar Functions series overview Part One: Parallelism Part Two: Performance Foreword In the second part of this...

Goal Progress for June 2022

By

I set goals at the beginning of the year, and I’m tracking my progress...

Using Sessionize for Your User Group – Part 2

By

Now that your CFS has been setup it’s time to get the word out...

Read the latest Blogs

Forums

Add a column with default value

By pedroccamara

Hi guys I have a query with a SELECT statement, that joins 2 other...

Help with Pivoting Table

By carlton 84646

Hi all, Can someone please help Pivot a table. The table currently looks like...

Help Transposing SQL Table Pt3

By carlton 84646

Hello Community, Can someone please help transpose a table. The table currently looks like...

Visit the forum

Ask SSC Logo Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers