Raghavendra Narayana

Parallel Processing of Large Volume ETL Jobs

by Raghavendra Narayana

SQLServerCentral.com

Extract, Transform, Load (ETL)

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(9)

You rated this post out of 5. Change rating

2007-11-08

7,094 reads

Discuss

All authors

Read the latest Blogs

Visit the forum

Question of the Day

The Read Committed Snapshot Isolation behaviour

I am currently working with Sql Server 2022 and AdventureWorks database. First of all, let's set the "Read Committed Snapshot" to ON:

use master;
go

alter database AdventureWorks set read_committed_snapshot on with no_wait;
go

Then, from Session 1, I execute the following code:

--Session 1
use AdventureWorks;
go

create table ##t1 (id int, f1 varchar(10));
go

insert into ##t1 values (1, 'A');

From another session, called Session 2, I open a transaction and execute the following update:

--Session 2
use AdventureWorks;
go

begin tran;
update ##t1 
set f1 = 'B'
where id = 1;

Now, going back to Session 1, what happens if I execute this statement?

--Session 1
select f1
from ##t1
where id = 1;

See possible answers

Raghavendra Narayana

Parallel Processing of Large Volume ETL Jobs

Blogs

Advice I Like: Art

Why Optimize CPU for RDS SQL Server is a game changer

Performance tuning KubeVirt for SQL Server

Forums

The AI Bubble and the Weak Foundation Beam

data type gets lost in data flow

i noticed the sqlhealth extende event is on by default , so can i reduce

Question of the Day

The Read Committed Snapshot Isolation behaviour