Raghavendra Narayana


SQLServerCentral Article

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

You rated this post out of 5. Change rating

2007-11-08

7,061 reads

Blogs

Creating a GitHub Actions Self-Hosted Runner

By

I had to demo the Flyway Autopilot system recently and created a GitHub Actions...

Migrating from Azure PG Single Server to Flex with Azure VM and pg_dump/restore

By

This is more complicated than using the Azure Migration method, but because it’s maxed...

PowerShell (and T-SQL) and Elastic Jobs

By

I’ve covered how to create Elastic Jobs in the portal (this one is important...

Read the latest Blogs

Forums

SSMA for Oracle get error for assembly 'Oracle.ManagedDataAccess'

By Xiaogang Zheng

Hi, I am using the SSMA for Oracle v10.0. I have installed the SSMA...

Avoiding duplicates stop and restart an ssis package that is using batch inserts

By Neo

Hi, I an using VS 2022 to build a package that selects and then...

Contracting DBA - Permissions for initial analysis

By StephenSQL

Accidental DBA here.  After looking at all the work that needs to be done,...

Visit the forum

Question of the Day

AG Rules for Databases

Which of these is not a requirement for a database to be added to an Availability Group (AG)?

See possible answers