The Art of Data Ingestion - Part 1

Data movement is a fundamental piece of a data engineer’s duties, and recently I’ve been thinking about the art of data movement. What are some of the most important pieces that a data engineer needs to think about when confronted with data ingestion? There is of course data exporting as well, and in that case, […]

minimally logged operations

We Loaded 1 Terabyte in 30 Minutes with SSIS, and So Can You

In February 2008, Microsoft announced a record-breaking data load using Microsoft® SQL Server® Integration Services (SSIS): 1 TB of data in less than 30 minutes. That data load, using SQL Server Integration Services, was 30% faster than the previous best time using a commercial ETL tool. This paper outlines what it took: the software, hardware, and configuration used. We will describe what we did to achieve that result, and offer suggestions for how to relate these techniques to typical scenarios.

Function to find non-printable characters

The function is used to find non-printable ASCII characters in an input string.

The Data Loading Performance Guide

This document described techniques for bulk loading large data sets into SQL Server. It covers both the available techniques as well as methodologies to performance tune and optimize the bulk loading process.

Getting Data out of SQL Server from Powershell

In the first article, we learned how to move data into SQL Server. This time Chad Miller shows us how we can use Powershell to extract data out of SQL Server.

Using OPENROWSET to read large files into SQL Server

OPENROWSET is a T-SQL function that allows for reading data from many sources including using the SQL Server’s BULK import capability.

Auditing Changes in ETL with SSIS

This article briefly demonstrates a methodology for auditing changes made to data flows during the ETL process.

Loading Data With Powershell

Moving data in and out of SQL Server is a fundamental task for any developer or DBA. Chad Miller brings us a new way that we can load Powershell output into a SQL Server database.

ETL Auditing

This Article shows a method to audit ETL-Processes to be able to retrace processes and affected data.