Raghavendra Narayana

All authors

SQLServerCentral Article

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

2.44 (9)

You rated this post out of 5. Change rating

2007-11-08

7,087 reads

All authors

Blogs

From Couch-Potato to Triathlete – and What This Means for Your SQL Server

By

Do you know if your SQL Server is really running at its best? To...

Retro Data 2025 – Slidedeck

By

You can find the slides of my session on the €100 DWH in Azure...

The Book of Redgate: We Value Teams

By

This value is something that I still hear today: our best work is done...

Read the latest Blogs

Forums

Is there a way for SP to know who called it?

By water490

Hi everyone I am writing an SP where there is logic inside the SP...

Planning for tomorrow, today - database migrations

By John Martin

Comments posted to this topic are about the item Planning for tomorrow, today -...

Bottlenecks on SQL Server performance

By runarlan

We have a BI-application that connects to input tables on a SQL Server 2022...

Visit the forum

Question of the Day

The Tightly Linked View

I try to run this code on SQL Server 2022. All the objects exist in the database. 

CREATE OR ALTER VIEW OrderShipping
AS
SELECT cl.CityNameID,
       cl.CityName,
       o.OrderID,
       o.Customer,
       o.OrderDate,
       o.CustomerID,
       o.cityId
 FROM dbo.CityList AS cl
 INNER JOIN dbo.[Order] AS o ON o.cityId = cl.CityNameID
GO
CREATE OR ALTER FUNCTION GetShipCityForOrder
(
    @OrderID INT
)
RETURNS VARCHAR(50)
WITH SCHEMABINDING
AS
BEGIN
    DECLARE @city VARCHAR(50);
    SELECT @city = os.CityName
    FROM dbo.OrderShipping AS os
    WHERE os.OrderID = @OrderID;
    RETURN @city;
END;
go
What is the result?

See possible answers