The idea of a data hub in your company is interesting to Steve Jones. He talks about this concept, and how it might help you increase data quality, and perhaps get closed to a single view of the truth.  Read more...

A primer on how to reduce network and source system load when reading a relational source into the data warehouse.  Read more...

On Thursday December 15th at 8PM Pacific, Jeremy will discuss the strengths and disadvantages of a row-versioned data warehouse design in the context of a real world case study, sharing lessons learnt and demonstrating some of the technologies and techniques used to build a row-versioned data warehouse   Read more...

Of all the technical solutions to the problem of slowly changing dimensions, the T-SQL MERGE statement is one of the most elegant.   Read more...

SQL Server 2008 introduced many new functional and performance improvements for data warehousing, and SQL Server 2008 R2 includes all these and more. This paper discusses how to use SQL Server 2008 R2 to get great performance as your data warehouse scales up. We present lessons learned during extensive internal data warehouse testing on a 64-core HP Integrity Superdome during the development of the SQL Server 2008 release, and via production experience with large-scale SQL Server customers. Our testing indicates that many customers can expect their performance to nearly double on the same hardware they are currently using, merely by upgrading to SQL Server 2008 R2 from SQL Server 2005 or earlier, and compressing their fact tables. We cover techniques to improve manageability and performance at high-scale, encompassing data loading (extract, transform, load), query processing, partitioning, index maintenance, indexed view (aggregate) management, and backup and restore.  Read more...

Data warehousing and general reporting applications tend to be CPU intensive because they need to read and process a large number of rows. To facilitate quick data processing for queries that touch a large amount of data, Microsoft SQL Server exploits the power of multiple logical processors to provide parallel query processing operations such as parallel scans. Through extensive testing, we have learned that, for most large queries that are executed in a parallel fashion, SQL Server can deliver linear or nearly linear response time speedup as the number of logical processors increases. However, some queries in high parallelism scenarios perform suboptimally. There are also some parallelism issues that can occur in a multi-user parallel query workload. This white paper describes parallel performance problems you might encounter when you run such queries and workloads, and it explains why these issues occur. In addition, it presents how data warehouse developers can detect these issues, and how they can work around them or mitigate them.  Read more...

A data mart provides the primary access to the data stored in the data warehouse or operational data store. It is a subset of data sourced from the data warehouse or operational data store specifically focused on a business function or set of related business functions. Read on to learn the answers to fundamental questions about data marts.  Read more...

David Poole takes a look at the Massively Parallel Processing world for database servers.  Read more...

The start of a new series from Leo Peysakhovich that looks at some of the issues with moving data around between systems and ensuring that it is in sync between them.  Read more...

One of the most integral components and critical success factors of any enterprise data warehousing initiative is the Solutions Architecture document, a high-level conceptual model of a data warehousing solution. Learn why this collaborative effort that addresses the needs of all major stakeholders, including both the business units and Information Technology (IT), is essential.  Read more...

The staging area tends to be one of the more overlooked components of a data warehouse architecture, and yet it is an integral part of the ETL component design. Learn why it is best to design the staging layer right the first time, enabling support of various ETL processes and related methodology, recoverability and scalability.   Read more...

Denise Rogers discusses the essential tasks in conducting effective software evaluations revolving around data warehousing and business intellegence. Each step has a dependency on the previous one, starting with establishing the framework of the evaluation and adding progressively elaborate data that facilitates a decision making process that is resolute.   Read more...

Data warehouse loads can be time consuming - this method can be used in some instances to help speed things up.  Read more...

Why neglecting slowly changing dimensions, failing to capture metadata and overlooking scope creep can be the undoing of a dimensional data warehousing initiative.  Read more...

Architecture and data warehousing are not static. From the first notion of a data warehouse to a full-blown analytical processing architecture that includes data marts, ETL, near line storage, exploration warehouses, and other constructs, data warehousing and its associated architecture continue to evolve. In 2008, the book on the latest evolution of data warehousing appeared – DW 2.0: The Architecture for the Next Generation of Data Warehousing (Morgan Kaufman). In that book the general architecture for data warehousing in its highest evolved form appeared.  Read more...

When you need to insert a large set of data using T-SQL, what can you do? New author Melinda Sansone brings us a best practice when using T-SQL to load data for data marts, or warehouses.  Read more...

Recently, I was at a conference, and I heard the following discussion about what a data warehouse was. One person suggested that a data warehouse was really all the old legacy systems connected by software that could access the data. By calling such a contraption a data warehouse, the organization could avoid having to do the hard and complex work of integration. There are so many problems with this federated approach to a data warehouse that they are almost not worth repeating here. But (once again!) here goes.   Read more...

Best-practice advice on software vs. coding, where to integrate, how to capture changed data, when to stage data, where to correct data and what latency levels to shoot for.  Read more...

Given that companies spend on average between 7 and 12 percent of their annual budgets on energy – a focus on reducing energy consumption can have significant bottom-line impact  Read more...

This article outlines a more objective and analytical approach to the ETL and data flow architecture selection based on a set of variables with an objective to enhance the reliability of the overall data warehousing solution.  Read more...