Extract, Transform, Load (ETL)

SQLServerCentral Article

The Art of Data Ingestion - Part 1

  • Article

Data movement is a fundamental piece of a data engineer’s duties, and recently I’ve been thinking about the art of data movement. What are some of the most important pieces that a data engineer needs to think about when confronted with data ingestion? There is of course data exporting as well, and in that case, […]

5 (8)

You rated this post out of 5. Change rating

2022-11-11

5,951 reads

SQLServerCentral Article

Parallel Processing of Large Volume ETL Jobs

  • Article

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

You rated this post out of 5. Change rating

2007-11-08

7,071 reads

Blogs

The Book of Redgate–Being Reasonable

By

As a part of the Book of Redgate, we have a series of (red,...

Azure Data Factory Pipeline Debugging Fails with BadRequest

By

I recently had a new pipeline fail. It was actually a copy of an...

Unlocking Real-Time Financial Insights: The Power of Microsoft Fabric

By

Microsoft Fabric is transforming real-time analytics for financial institutions. It provides a unified data...

Read the latest Blogs

Forums

How to get result from deleted row

By mashispano

Hi Guys I have a store proc that delete a row from a and...

Count of Distinct Values

By jagjitsingh

Hi If value is 21000030 it should return 4 not 5 . In first...

Joining against min date only

By darrell.woolaver 91826

This is one of those things very hard (for me)to explain in words. See...

Visit the forum

Question of the Day

Rolling Back CUs

Can I rollback a cumulative update and remove it from my SQL Server instance?

See possible answers