Raghavendra Narayana

Parallel Processing of Large Volume ETL Jobs

by Raghavendra Narayana

SQLServerCentral.com

Extract, Transform, Load (ETL)

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(9)

You rated this post out of 5. Change rating

2007-11-08

7,097 reads

Discuss

All authors

Read the latest Blogs

Visit the forum

Question of the Day

Which Result II

I have this code in SQL Server 2022:

CREATE SCHEMA etl;
GO
CREATE TABLE etl.product
(
    ProductID INT,
    ProductName VARCHAR(100)
);
GO
INSERT etl.product
VALUES
(2, 'Bee AI Wearable');
GO
CREATE TABLE dbo.product
(
    ProductID INT,
    ProductName VARCHAR(100)
);
GO
INSERT dbo.product
VALUES
(1, 'Spiral College-ruled Notebook');
GO
CREATE OR ALTER PROCEDURE etl.GettheProduct
AS
BEGIN
    exec('SELECT ProductName FROM product;')
END;
GO

When I execute this code as a user whose default schema is dbo and has rights to the tables and proc, what is returned?

See possible answers

Raghavendra Narayana

Parallel Processing of Large Volume ETL Jobs

Blogs

Learn Better: Pause to Review More

Azure SQL Managed Instance Next-Gen: Bring on the IOPS

SQL, MDX, DAX – the languages of data

Forums

Faster Data Engineering with Python Notebooks: The Fabric Modern Data Platform

Which Result II

JSON Has a Cost

Question of the Day

Which Result II