Extract, Transform, Load (ETL)

SQLServerCentral Article

The Art of Data Ingestion - Part 1

  • Article

Data movement is a fundamental piece of a data engineer’s duties, and recently I’ve been thinking about the art of data movement. What are some of the most important pieces that a data engineer needs to think about when confronted with data ingestion? There is of course data exporting as well, and in that case, […]

5 (8)

You rated this post out of 5. Change rating

2022-11-11

5,622 reads

SQLServerCentral Article

Parallel Processing of Large Volume ETL Jobs

  • Article

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

You rated this post out of 5. Change rating

2007-11-08

6,915 reads

Blogs

First Time Speaking with a Mask at an Event

By

Over the past couple of months, I have started losing my ability to talk...

A New Word: Apolytus

By

apolytus– n. the moment you realize you are changing as a person, finally outgrowing...

Tired of Tedious SQL Queries? AI Tools Can Automate The Boring Parts

By

If you've ever wrestled with complex SQL code, you know the drill. Hours spent...

Read the latest Blogs

Forums

Do you listen to music when you work?

By Louis Davidson (@drsql)

Comments posted to this topic are about the item Do you listen to music...

Slow update on temp table

By dpaproskijr

I have a job running each night starting at 4:30AM and usually finishing at...

Table Partitioning - remove null range

By goher2000

hi guys,   I have a partitioned table AM_ITM_INSTNC_MSS and I want to  get...

Visit the forum

Question of the Day

Read Only Replica in SQL Server Standard

Our environment runs using SQL Server Standard. We are implementing Availability groups. Our database has been experiencing high read volumes, so I want to let the application read the Synchronized Secondary replica, as I read that HADR does this. Can we implement this?

See possible answers