Raghavendra Narayana

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

2007-11-08

5,026 reads

Blogs

Azure Cosmos DB: real-time data movement using Change Feed and Azure Functions

By

In this article, we will focus on creating a data pipeline to ETL (Extract,...

Quick 6 month check – how are those learning goals going…?

By

So in January I wrote a blog post on some goals I had this...

Read the latest Blogs

Forums

Select n records in table X from a field on table Y table

By luisalfonso70

Hi, hope someone can help with this. I've the table gifts_to_give: Id_give      number_of_gifts          name ...

Error: 17892 Severity: 20 State: 1

By ismahar

Hi, I've been struggling for 5 days to solved this issue. We have a...

Mass replace of DB codes

By berki

Hi, I have a column in a table that contains variable semi-colon DB codes...

Visit the forum

Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers

Question of the Day

Intersection

See possible answers