Raghavendra Narayana

All authors

SQLServerCentral Article

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(9)

You rated this post out of 5. Change rating

2007-11-08

7,097 reads

All authors

Blogs

Another Change

By

Today Redgate announced that we are partnering with Bregal Sagemount, a growth-focused private equity...

Using Prompt AI to Help Setup Data Analysis

By

I used Claude to build an application that loaded data for me. However, there...

NVMe vs SCSI: Real-World Performance Testing for SQL Server Workloads on VMware

By

End-to-end NVMe vs SCSI testing over NVMe/TCP to a Pure Storage FlashArray: TPC-C and...

Read the latest Blogs

Forums

Simplifying WHERE Condition with LIKE test on multiple columns

By Reh23

Good Evening, Is there a simpler way to rearrange the following WHERE condition: [Column_1]...

Which Table I

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Which Table I

Using Python notebooks to save money in Fabric: The Fabric Modern Data Platform

By John Miner

Comments posted to this topic are about the item Using Python notebooks to save...

Visit the forum

Question of the Day

Which Table I

I have this code in SQL Server 2022: 

CREATE SCHEMA etl;
GO
CREATE TABLE etl.product
(
    ProductID INT,
    ProductName VARCHAR(100)
);
GO
INSERT etl.product
VALUES
(2, 'Bee AI Wearable');
GO
CREATE TABLE dbo.product
(
    ProductID INT,
    ProductName VARCHAR(100)
);
GO
INSERT dbo.product
VALUES
(1, 'Spiral College-ruled Notebook');
GO
CREATE OR ALTER PROCEDURE etl.GettheProduct
AS
BEGIN
    SELECT  ProductName
    FROM product;
END;
GO
When I execute this code as a user whose default schema is dbo and has rights to the tables and proc, what is returned?

See possible answers