Extract, Transform, Load (ETL)

SQLServerCentral Article

The Art of Data Ingestion - Part 1

  • Article

Data movement is a fundamental piece of a data engineer’s duties, and recently I’ve been thinking about the art of data movement. What are some of the most important pieces that a data engineer needs to think about when confronted with data ingestion? There is of course data exporting as well, and in that case, […]

5 (8)

You rated this post out of 5. Change rating

2022-11-11

5,951 reads

SQLServerCentral Article

Parallel Processing of Large Volume ETL Jobs

  • Article

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

2.44 (9)

You rated this post out of 5. Change rating

2007-11-08

7,071 reads

Blogs

Unlocking Real-Time Financial Insights: The Power of Microsoft Fabric

By

Microsoft Fabric is transforming real-time analytics for financial institutions. It provides a unified data...

Understanding Lakehouse Permissions in Microsoft Fabric

By

Understanding how permissions work in Microsoft Fabric can be essential for anyone managing access...

The Cost of Not Having a DBA

By

Does skipping a DBA save money? Wait until your system grinds to a halt,...

Read the latest Blogs

Forums

Rolling Back CUs

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Rolling Back CUs

A Domain for Data

By Steve Jones - SSC Editor

Comments posted to this topic are about the item A Domain for Data

Windows Clustered MSDTC and SQL Server 2019 mishap

By Johan Bijnens

Comments posted to this topic are about the item Windows Clustered MSDTC and SQL...

Visit the forum

Question of the Day

Rolling Back CUs

Can I rollback a cumulative update and remove it from my SQL Server instance?

See possible answers