Surrounding the ETL Requirements

November 13, 2004 / Issue TOC

Surrounding the ETL Requirements

Before designing an ETL system, you must first understand all of your business needs.

By Ralph Kimball , Margy Ross

Ideally, the design of your extract, transform, and load (ETL) system begins with one of the toughest challenges: surrounding the requirements. By this we mean gathering in one place all the known requirements, realities, and constraints affecting the ETL system. The list of requirements is pretty overwhelming, but it's essential to lay them on the table before launching a data warehouse project.

The requirements are mostly things you must live with and adapt your system to. Within the framework of your requirements, you'll have many places where you can make your own decisions, exercise your judgment, and leverage your creativity, but the requirements are just what they're named. They are required.

Code Generation: Using SQL to Generate SQL and Other Code

SQL Server is a great platform against which to develop, but it can also be used as a code generation tool with a little creativity. Haidong Ji brings us a few techniques that can be used to automatically build T-SQL code and bcp code for importing and exporting data.

Using Parent Package Variables in Package Configurations

Package configurations are now the prescribed way of being able to set values within your package from an outside source. One of the options for the source is Parent Package Variable. The name is perhaps a little misleading so this article is meant to guide you through this slight confusion and into using them. It also helps to explain a key concept in SQL Server Integration Services

SQL Server 2000 Reporting Services Activation Process

SQL Server 2000 Reporting Services uses cryptography to protect account data. Learn how SQL Server 2000 Reporting Services generates and stores the keys required to encrypt data.

Using a Subquery in a T-SQL Statement

Sometimes the criteria for determining which set of records will be affected by a SELECT, UPDATE, DELETE and/or INSERT statement cannot be obtained by hard coding the selection criteria. Occasionally there is a need to use the results of a SELECT statement to help determine which records are returned or are affected by a T-SQL statement. When a SELECT statement is used inside another statement, the inside SELECT statement is known as a subquery. Subqueries can help to dynamically control the records affected by an UPDATE, DELETE or INSERT statement, as well as to help determine the records that will be returned by a SELECT statement. This article will show different examples of how to use a subquery to help clarify the records affected or selected by a T-SQL statement.

Monitoring Blocks

SQL Server excels at quickly acquiring and releasing locks to allow as much concurrency as possible on the server. However there are times that one user will block another, which can be a source of great user frustration as well as making the system appear to be slow. Leo Peysakhovich brings us some great information and code on how you can monitor and deal with blocking on your SQL Server 2000 server.

Full Text Search on SQL 2000 Part 3

In the previous month's articles, Full Text Searching was introduced as a way to query strings with more refinement than the usual TSQL "like" or equal operator statements. With Full Text Searching, a new file system structure is created, storing key words from selected fields into Catalogs. In addition to storing typical character fields from databases, Microsoft Office documents that have been saved as binaries can also be entered into the Catalogs. In this month's edition, we will begin with the maintenance issues required to keep these Catalogs current. Once created, these catalogs can be interrogated for key words that are near each other's proximity, the singular and plural versions of a word, or the noun and verb variations of a word. In addition, search result rankings, or weights, that usually accompany internet search engine returns can also be requested.

Rebuilding SQL Server Cluster Nodes

Active/Passive SQL Server 2000 clustering gives more reliability and fault tolerance to Production SQL Server environments. When a failure occurs, all of the resources fail over from the active node to the passive node and make the passive node active. This article explains how to rebuild the node that failed and attach it back to the cluster.

Optimizer Join Methods

Understanding the different types of joins used by the optimizer will help developers and DBAs understand how the optimizer is routing their queries. Developers often create queries without knowing that it would only take a few tweaks to produce an execution plan that utilizes one optimizer join method over another. These small tweaks can have dramatic effects on the optimization of the query and the ultimate satisfaction of the query by the end-users.

Full Text Search on SQL 2000 Part 2

In last month's article, Full Text Searching was introduced as a way to query strings with more refinement than the usual TSQL "like" or equal operator statements. With Full Text Searching, a new file system structure is created, storing key words from selected fields into Catalogs. In addition to storing typical character fields from databases, Microsoft Office documents that have been saved as binaries can also be entered into the Catalogs. Once created, these catalogs can be interrogated for key words that are near each other's proximity, the singular and plural versions of a word, or the noun and verb variations of a word can all be searched for. In addition, search result rankings, or weights, that usually accompany internet search engine returns can also be requested.