The Art of Data Ingestion - Part 1

Data movement is a fundamental piece of a data engineer’s duties, and recently I’ve been thinking about the art of data movement. What are some of the most important pieces that a data engineer needs to think about when confronted with data ingestion? There is of course data exporting as well, and in that case, […]

TSV

Function to find non-printable characters

The function is used to find non-printable ASCII characters in an input string.

Kimball ETL Part 1 – Data Profiling via SSIS Data Flow

A new series sponsored by Actuality Business Intelligence on data warehousing. In part 1, the data flow in SSIS packages are used to profile the source data and determine how it should be handled in the process.

ETL for Free-Form Data

This article demonstrates how to extract, transform and load free-form data into a structured form like a staging table by way of XML conversions.

Slowly Changing Dimensions

Bulk Import Formats

Importing Text-based data: Workbench

Robyn and Phil return with some fresh ideas about how to import text files into SQL Server, without resorting to DTS or SSIS scripting. They go on to show how much can be done in TSQL

Parallel Processing of Large Volume ETL Jobs

ETL processing, generally involves copying/moving, transforming, cleaning the records/transactions from one or multiple sources. Most of the batch processing or warehousing projects involve such data processing in millions on daily/weekly basis. Typically, there is a Staging area and production area. Records are cleaned, transformed, filtered and verified from staging to production area. This demands SQL Set theory based queries, parallel processing with multiple processors/CPU. The article focuses on need of SQL Set theory approach and parallel processing while processing large volume of ETL records using programming approach.

The Subsystems of ETL Revisited

These 34 subsystems cover the crucial extract, transform and load architecture components required in almost every dimensional data warehouse environment. Understanding the breadth of requirements is the first step to putting an effective architecture in place.