ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

Importing Poorly Formatted Text Files using SSIS

by Additional Articles

SQLServerCentral

Integration Services (SSIS)

Importing text files is a common task for DBA's and Developers, but what happens when the text file is not properly formatted?

SQL Puzzle 7

by Additional Articles

SimpleTalk

SQL Puzzles

After a gap of sixteen months, Lionel Clarke, the creator of the only 3-D maze game to be written in TSQL, returns with a new SQL puzzle. It is very simple. You have to move as much of the data as you can from the source tables to the destination tables.

Using SQL Server 2008's MERGE statement

by Additional Articles

SQLServerCentral

SQL Server 2008

SQL Server 2008's new MERGE statement allows you to insert, update, or delete data based on certain join conditions in the same statement.

Which Platform Do You Work On?

by Steve Jones

SQLServerCentral.com

Editorial

The growth of SQL Server's subsystems leaves Steve Jones wondering how should we be thinking about the product.

XML Workshop X - Working with namespaces

by Jacob Sebastian

SQLServerCentral.com

With the tenth installment of his series on XML, Jacob Sebastian turns his attention to the art of namespaces and how you can avoid ambiguity in interpreting various elements.