InstantForum.NET v2.9.0SQLServerCentralhttp://www.sqlservercentral.com/Forums/notifications@sqlservercentral.comTue, 18 Jun 2013 19:49:17 GMT20http://www.sqlservercentral.com/Forums/Topic448805-361-1.aspxHi Jay,After Running a similar sized Datawarehouse (without Cubes) the following is the main advice I can give you.1) Business rules for validating and cleaning the Data take the bulk of the ETL time.2) Pay particular attention to your disk configuration, make good use of filegroups and multiple disk arrays (we did not have a SAN but inherited 100 disks with 6 Raid controllers), make sure your NTFS allocations are correct, I found 64K was best performance for our configuration. Would recommend using Mount points for you disk arrays as it makes for easier configuration and easier restores to other systems i.e. dev, test.3) Backup Compression software in our case was a must for to meet backup windows and reduce disk space. (used SQL Lite Speed with no problems)4) The Warehouse was based around Kimballs dimensional model, found that we had to add back some of the Natural keys to some of the very large Fact and dimensional tables for performance reasons, the joins between the large tables was killing performance.5) Appropriate indexing and Up to date statistic's (we updated nightly for all but the largest table) makes a huge performance impact.6) Setting the Warehouse to Read only after the ETL made a huge difference to reporting speedI trust this is helpful, can give further details if you would like.CheersBrandonSun, 03 Feb 2008 16:02:00 GMTfruitloophttp://www.sqlservercentral.com/Forums/Topic448805-361-1.aspxIf you have an ETL database for staging the data you would not have indexes on tables. The point of the ETL is to pull the information out and prep it for loading into your datawarehouse not to query against. If you use the datawarehouse as the collection of everything for historic purposes then you should think about implementing data marts or small subsets of the data using snapshots for your different internal business clients and leave the datawarehouse for your BI folks, and I don’t mean marketing, I mean your internal person who understands data mining algorithms and who can perform predictive analytics. The reduced number of data columns tailored to the specific needs of the internal client group could save you from index problems. The thing to keep in mind with indexes is that if the query returns more then 1% of your total data the optimizer will not use the index < this tidbit comes from Kimberly Trip at the SQLConnections conference (her site is excellent BTW http://www.sqlskills.com/). Your indexes on a DW will add roughly 3-5 times the storage space of the data and keep in-mind that to do online index rebuilds you need to have space equal to the size of the index being rebuilt. A couple of things to keep in mind for planning purposes, first the 5-10 min ETL as a Service Level Agreement, may not be workable, 80% of the effort in the build of datawarehouse and data mining is getting the data clean and in the format you need. Only you are the expert as to the quality of your data but until you run through this process I would not commit to 5-10 mins. Additionally, only you can realistic assess if a 5-10 minute refresh makes business sense too. Marketing does just fine on a 12 or 24 hour old view of the data and inventory managment does fine with 30 min views of the data so long as you have a method to notify customers that you are out of stock when they have already placed an order.Also, keep in mind that in order to data mine the data you need to use nvarchar and ntext as data types, varchar and varchar(max) won’t work. This could double your initial storage requirements and you want to store data in this type instead of converting on the fly which you can in BI-Studio because of the additional memory overheard associated with the conversion which leaves you with less memory for the predictive analytics which use a lot of memory; this alone is often reason enough to move to a 64 bit version for DW needs. Based on the size of your data normalizing of 3NF should work fine the advantage to a star schema is that it is more intuitive for the end user clients so if you are looking to use the data mining plug-in for Excel to allow end-user to query data themselves this may work to your advantage > the advantage being that it shifts the report monkey duties to them and away from you leave you to do the heavy lifting such as predictive analytics and multichannel analysis.Last recommendation is to use the following tool which I seem to be recommending a lot lately to automatically figure out your index needs based on usage patterns. This is from an earlier post but holds true here too.What you want to do is pull the data from the missing index dmv and the script below will do this for you by creating a database called AUTOINDEXRECS that polls the missing index dmv and sucks in the info which leaves you to come back at a later time and look at the table to determine what indexes you need to create, and which can be dropped, and on what tables. You need to have sa permission to do this. This comes from the query team at microsoft and you should download the .zip here http://blogs.msdn.com/queryoptteam/archive/2006/06/01/613516.aspx.I found it on Paul's former storage engine blog. When you query the recommendation table the results will look like the following:CREATE INDEX _MS_Sys_1 ON [Database_name].[dbo].[tbl_name]([ResponseID]) INCLUDE ([ResponseText])This is without a doubt the best tuning tool for a database server, works wonders in OLAP environments where you don't know what the reports are going to be before hand, and I am baffled why this is not more widely known.Lastly, make sure to use upsert statements if you will be updataing as well as inserting data and wrap everything in one transaction statement this will save you from the overhead of a row by agonizing row committ for all inserts. And make sure your temp DB has one file for each CPU core on the box to prevent contention issues. The partitioning by month is a good plan but take a look here at a better design methodology to use >From Kim Tripp(Entire post here http://www.sqlskills.com/blogs/kimberly/2007/10/03/SQLServer2008OffersPartitionlevelLockEscalationExcellentBut.aspx) "As a result, I would suggest a slightly different architecture. Instead of using only a single partitioned table for both read-only and read-write data, use at least two tables. One table for read-only data and another for read-write data. If you think this might be defeating the purpose of partitioning... then look at these benefits:* the read-only portion of the table (which is typically the *much* larger portion of the table - can still be managed with partitioning)* the read-only portion - once separated from the read-write - can have additional indexes for better [range] query performance* the read-only portion of the table can actually be partitioned into multiple partitioned tables - to give better per-table statistics (statistics are still at the table-level only so even if your partitioning scheme is "monthly" you might want to have tables that represent a year's worth of data...especially if your trends seem to change year to year)* large range queries against the read-only portion of the data will only escalate to the "table" (which is now separated from the read-write data)* the read-write portion of the data can have fewer indexes* the read-write portion of the data can be placed on different disks (MORE fault tolerant disks) due to the importance/volatility of the data* finally, and most importantly, the read-write portion of the data can be maintained completely separately from the read-only portion with regard to index rebuilds"Hope this helps,--Dave/****************************************************************************// Copyright (c) 2005 Microsoft Corporation.//// @File: AutoIndex.sql// // @test: //// Purpose:// Auto create or drop indexes//// Notes:// //// @EndHeader@*****************************************************************************/CREATE DATABASE AUTOINDEXRECSgoUSE AUTOINDEXRECSgo-- Table to store recommendationsIF object_id(N'dbo.recommendations', N'U') IS NOT NULLDROP table [dbo].[recommendations]GOcreate table [dbo].[recommendations] ( id int IDENTITY primary key,recommendation nvarchar(400), type char(2), initial_time datetime,latest_time datetime,[count] int, status nvarchar(20))GO-- Table to store recommendation historyIF object_id(N'dbo.recommendations_history', N'U') IS NOT NULLDROP table [dbo].[recommendations_history]GOcreate table [dbo].[recommendations_history] ( id int,operation nvarchar(20),time datetime,db_user_name sysname,login_name sysname )GO-- Table to store index recommendations detailsIF object_id(N'dbo.recommendations_details_index', N'U') IS NOT NULLDROP table [dbo].[recommendations_details_index]GOcreate table [dbo].[recommendations_details_index] ( id int,database_id int,table_id int,table_modify_time datetime)GO------------------------- add_recommendation_history ---------------------------------------------------------- SP for adding a recommendation into the recommendations_history table.IF OBJECT_ID (N'dbo.add_recommendation_history', N'P') IS NOT NULLDROP PROC [dbo].[add_recommendation_history];GOcreate procedure [dbo].[add_recommendation_history]@id int,@operation nvarchar(20),@time datetimeASBEGINdeclare @db_user_name sysnameselect @db_user_name = CURRENT_USERdeclare @login_name sysnameselect @login_name = SUSER_SNAME()insert into recommendations_history values (@id, @operation, @time, @db_user_name, @login_name)ENDgo------------------------- add_recommendation---------------------------------------------------------- SP for inserting a new recommendation into the dbo.RECOMMENDATIONS table.------ If the same entry already exists, it just changes latest_create_date to the latest time------ and increase the count by oneIF OBJECT_ID (N'dbo.add_recommendation', N'P') IS NOT NULLDROP PROC [dbo].[add_recommendation];GOcreate procedure [dbo].[add_recommendation]@recommendation nvarchar(max), @type_desc char(2),@id int OUTPUTASBEGINdeclare @create_date datetimeset @create_date = getdate()IF ( @recommendation not in (select recommendation from dbo.recommendations))BEGINinsert into dbo.recommendations values (@recommendation, @type_desc, @create_date, @create_date, 1, N'Active')select @id = @@identity-- add it into the recommendation historyexec [dbo].[add_recommendation_history] @id, N'ADD', @create_datereturn 0ENDELSEBEGINselect @id = idfrom dbo.recommendationswhere @recommendation = recommendationupdate dbo.recommendations set latest_time = @create_date, [count] = [count] +1where id = @id-- add it into the recommendation historyexec [dbo].[add_recommendation_history] @id, N'UPDATE', @create_datereturn 10ENDENDgo------------------------- disable_recommendation---------------------------------------------------------- SP for disabling a recommendation in the RECOMMENDATIONS table.IF OBJECT_ID (N'dbo.disable_recommendation', N'P') IS NOT NULLDROP PROC [dbo].[disable_recommendation];GOcreate procedure [dbo].[disable_recommendation]@id intASBEGINBEGIN TRANSACTION xDisableRecommendationdeclare @create_date datetimeset @create_date = getdate()update recommendations set status = N'Inactive'where id = @id-- add it into the recommendation historyexec [dbo].[add_recommendation_history] @id, N'DISABLE', @create_dateDECLARE @Error intSET @Error = @@ERRORIF @Error <> 0BEGINROLLBACK TRANSACTION xDisableRecommendationRETURN @ErrorEND COMMIT TRANSACTION xDisableRecommendationENDgo------------------------- enable_recommendation---------------------------------------------------------- SP for enabling a recommendation in the RECOMMENDATIONS table.IF OBJECT_ID (N'dbo.enable_recommendation', N'P') IS NOT NULLDROP PROC [dbo].[enable_recommendation];GOcreate procedure [dbo].[enable_recommendation]@id intASBEGINBEGIN TRANSACTION xEnableRecommendationdeclare @create_date datetimeset @create_date = getdate()update recommendations set status = N'Active'where id = @id-- add it into the recommendation historyexec [dbo].[add_recommendation_history] @id, N'ENABLE', @create_dateDECLARE @Error intSET @Error = @@ERRORIF @Error <> 0BEGINROLLBACK TRANSACTION xEnableRecommendationRETURN @ErrorEND COMMIT TRANSACTION xEnableRecommendationENDgo------------------------- execute_recommendation---------------------------------------------------------- SP for executing a recommendation in the RECOMMENDATIONS table.IF OBJECT_ID (N'dbo.execute_recommendation', N'P') IS NOT NULLDROP PROC [dbo].[execute_recommendation];GOcreate procedure [dbo].[execute_recommendation]@id intASBEGINdeclare @recommendation nvarchar(max)declare @status nvarchar(20)-- exec the recommendationselect @recommendation = recommendation, @status = statusfrom [recommendations]where id = @id-- check recommendation statusif (@status = 'Inactive')beginprint N'Error: Recommendation ' + cast ( @id as nvarchar(10)) + ' is Inactive. Change the status to Active before execution'return 1end-- check whether the schema has changed for the table declare @database_id intdeclare @object_id intdeclare @stored_modify_date datetimeselect @database_id = database_id, @object_id = table_id, @stored_modify_date = table_modify_timefrom [dbo].[recommendations_details_index]where id = @iddeclare @database_name sysnameselect @database_name = db_name(@database_id)-- create temporary table to store the current table schema versioncreate table [#tabSchema] ( modify_date datetime)truncate table [#tabSchema]declare @exec_stmt nvarchar(4000)select @exec_stmt = 'use '+ @database_name + '; insert [#tabSchema] select modify_date from sys.objects where object_id = ' + cast ( @object_id as nvarchar(10)) --print @exec_stmtEXEC (@exec_stmt)declare @modify_date datetimeselect @modify_date = modify_date from #tabSchemaif (object_id('[#tabSchema]') is not null)begindrop table [#tabSchema]endif (@modify_date > @stored_modify_date)beginprint N'Error: Recommendation ' + cast ( @id as nvarchar(10)) + ' might be invalid since the schema on the table has changed since the recommendation was made'return 1enddeclare @create_date datetimeset @create_date = getdate()BEGIN TRANSACTION xExecuteRecommendationexec (@recommendation)-- add it into the recommendation historyexec [dbo].[add_recommendation_history] @id, N'EXECUTE', @create_dateDECLARE @Error intSET @Error = @@ERRORIF @Error <> 0BEGINROLLBACK TRANSACTION xExecuteRecommendationRETURN @ErrorEND COMMIT TRANSACTION xExecuteRecommendationENDgo------------------------- add_recommendation_details_index ---------------------------------------------------------- SP for adding index recommendation details into the recommendations_details_index table.IF OBJECT_ID (N'dbo.add_recommendation_details_index', N'P') IS NOT NULLDROP PROC [dbo].[add_recommendation_details_index];GOcreate procedure [dbo].[add_recommendation_details_index]@id int,@database_id int,@table_id intASBEGINdeclare @database_name sysnameselect @database_name = db_name(@database_id)-- create temporary table to store the current table schema versioncreate table [#tabSchemaVer] ( modify_date datetime)truncate table [#tabSchemaVer]declare @exec_stmt nvarchar(4000)select @exec_stmt = 'use '+ @database_name + '; insert [#tabSchemaVer] select modify_date from sys.objects where object_id = ' + cast ( @table_id as nvarchar(10)) --print @exec_stmtEXEC (@exec_stmt)declare @tabVer datetimeselect @tabVer = modify_date from #tabSchemaVerinsert into recommendations_details_index values (@id,@database_id, @table_id, @tabVer) if (object_id('[#tabSchemaVer]') is not null)begindrop table [#tabSchemaVer]endENDgo---------------------------- auto_create_index ------------------------------IF OBJECT_ID (N'dbo.auto_create_index', N'P') IS NOT NULLDROP PROC [dbo].[auto_create_index];GOcreate procedure [dbo].[auto_create_index]as-- NOTE: This sp will create indexes recommended by the Missing Index DMVs.-- set nocount on-- required for creating index on ICC/IVsset ansi_warnings onset ansi_padding onset arithabort onset concat_null_yields_null onset numeric_roundabort offdeclare @exec_stmt nvarchar(4000)declare @table_name nvarchar(521)declare @column_name sysnamedeclare @column_usage varchar(20)declare @column_id smallintdeclare @index_handle intdeclare @database_id intdeclare @object_id int-- find the top 5 indexes with maximum total improventdeclare ms_cri_tnames cursor local static forSelect Top 5 mid.database_id, mid.object_id, mid.statement as table_name, mig.index_handle as index_handlefrom (select (user_seeks+user_scans) * avg_total_user_cost * (avg_user_impact * 0.01) as index_advantage, migs.*from sys.dm_db_missing_index_group_stats migs) as migs_adv,sys.dm_db_missing_index_groups mig,sys.dm_db_missing_index_details midwheremigs_adv.group_handle = mig.index_group_handle andmig.index_handle = mid.index_handleand migs_adv.index_advantage > 10order by migs_adv.index_advantage DESC-- create temporary table to store the table names on which we just auto created indexescreate table #tablenametab( table_name nvarchar(521) collate database_default )truncate table #tablenametabopen ms_cri_tnamesfetch next from ms_cri_tnames into @database_id, @object_id, @table_name, @index_handle--print @table_namewhile (@@fetch_status <> -1)begin-- don't auto create index on same table again-- UNDONE: we may try to filter out local temp table in the futureif (@table_name not in (select table_name from #tablenametab ))begin-- these are all columns on which we are going to auto create indexesdeclare ms_cri_cnames cursor local for select column_id, quotename(column_name,'['), column_usagefrom sys.dm_db_missing_index_columns(@index_handle)-- now go over all columns for the index to-be-created and-- construct the create index statementopen ms_cri_cnamesfetch next from ms_cri_cnames into @column_id, @column_name, @column_usagedeclare @index_name sysnamedeclare @include_column_list nvarchar(517)declare @key_list nvarchar(517)select @index_name = '_MS_Sys'select @key_list = ''select @include_column_list = ''declare @num_keys smallintdeclare @num_include_columns smallintselect @num_keys = 0select @num_include_columns = 0while @@fetch_status >= 0begin-- construct index name, key list and include column list during the loop-- Index Name in the format: _MS_Sys_colid1_colid2_..._colidnif (@column_usage = 'INCLUDE') beginif (@num_include_columns = 0)select @include_column_list = @column_name elseselect @include_column_list = @include_column_list + ', ' +@column_nameselect @num_include_columns = @num_include_columns + 1endelse beginif (@num_keys = 0)select @key_list = @column_nameelseselect @key_list = @key_list + ', ' +@column_nameselect @num_keys = @num_keys + 1select @index_name = @index_name + '_'+cast ( @column_id as nvarchar(10))endfetch next from ms_cri_cnames into @column_id, @column_name, @column_usageendclose ms_cri_cnamesdeallocate ms_cri_cnames--print @index_name--print @table_name--print @key_list--print @include_column_list-- construct create index statement-- "CREATE INDEX @INDEX_NAME ON @TABLE_NAME (KEY_NAME1, KEY_NAME2, ...) INCLUDE (INCLUDE_COL_NAME1, INCLUDE_COL_NAME2, ...) WITH (ONLINE = ON)" (Note: for recommendation mode, we don't use online option)if (@num_include_columns > 0)select @exec_stmt = 'CREATE INDEX ' + @index_name + ' ON ' + @table_name + '(' + @key_list + ') INCLUDE ('+ @include_column_list + ')'-- WITH (ONLINE = ON)'elseselect @exec_stmt = 'CREATE INDEX ' + @index_name + ' ON ' + @table_name + '(' + @key_list + ')'-- WITH (ONLINE = ON)'--print @exec_stmtdeclare @id intdeclare @create_date datetimeBEGIN TRANSACTION xAddCreateIdxRecommendationDECLARE @result int;EXEC @result = dbo.add_recommendation @exec_stmt, 'CI', @id OUTif (@result <> 10)EXEC dbo.add_recommendation_details_index @id, @database_id, @object_id DECLARE @Error intSET @Error = @@ERRORIF @Error <> 0BEGINROLLBACK TRANSACTION xAddCreateIdxRecommendationRETURN @ErrorEND COMMIT TRANSACTION xAddCreateIdxRecommendation--EXEC (@exec_stmt)-- insert the table name into #tablenametab insert into #tablenametab values (@table_name)endfetch next from ms_cri_tnames into @database_id, @object_id, @table_name, @index_handleenddeallocate ms_cri_tnamesreturn(0) -- auto_create_indexgo---------------------------- sp_autodropindex ------------------------------IF OBJECT_ID (N'dbo.auto_drop_index', N'P') IS NOT NULLDROP PROC [dbo].[auto_drop_index];GOcreate procedure [dbo].[auto_drop_index]as-- NOTE: This sp will drop indexes that are automatically created and -- are no longer very useful in a cost efficient manner based on feedbacks -- from index usage DMVs.set nocount ondeclare @database_id intdeclare @object_id intdeclare @index_id intdeclare ms_drpi_iids cursor local static forSelect Top 3 database_id, object_id, index_idfrom sys.dm_db_index_usage_statswhere user_updates > 10 * (user_seeks+user_scans) and index_id > 1order by user_updates / (user_seeks+user_scans+1) DESCopen ms_drpi_iidsfetch next from ms_drpi_iids into @database_id, @object_id, @index_id-- create temporary table to store the table name and index namecreate table #tabIdxnametab( table_name nvarchar(1000) collate database_default,index_name nvarchar(521) collate database_default )while (@@fetch_status >= 0)begindeclare @exec_stmt nvarchar(4000)declare @database_name sysnameselect @database_name = db_name(@database_id)truncate table #tabIdxnametab-- insert the table name and index name into the temp tableselect @exec_stmt = 'use '+ @database_name + ';'+'insert #tabIdxnametab select quotename(''' + @database_name+''', ''['')+ ''.'' +quotename(schema_name(o.schema_id), ''['')+''.''+quotename(o.name,''['') , i.namefrom sys.objects o, sys.indexes i where o.type = ''U'' and o.is_ms_shipped = 0 and i.is_primary_key = 0 and i.is_unique_constraint = 0 and o.object_id =' + cast ( @object_id as nvarchar(10))+' and o.object_id = i.object_id and index_id = '+ cast ( @index_id as nvarchar(10)) --print @exec_stmtEXEC (@exec_stmt) -- get the table_name and index_namedeclare @table_name nvarchar(1000)declare @index_name sysnameselect @table_name = table_name, @index_name = index_name from #tabIdxnametab--use name convention to recognize auto-created indexes for now--in the future, we will add a special bit inside metadata to distinguish--if (substring(@index_name, 1, 8) = '_MS_Sys_') --begin-- construct drop index statement-- "DROP INDEX @TABLE_NAME.@INDEX_NAME"--select @exec_stmt = 'drop index '+@index_name+' on '+@table_name--print @exec_stmt--EXEC (@exec_stmt)--end--else--print 'User Index: '+@table_name + '.'+ @index_nameIF (@index_name IS NOT NULL)beginselect @exec_stmt = 'drop index '+@index_name+' on '+@table_namedeclare @id intdeclare @create_date datetimeBEGIN TRANSACTION xAddDropIdxRecommendationDECLARE @result int;EXEC @result = dbo.add_recommendation @exec_stmt, 'DI', @id outif (@result <> 10)EXEC dbo.add_recommendation_details_index @id, @database_id, @object_id DECLARE @Error intSET @Error = @@ERRORIF @Error <> 0BEGINROLLBACK TRANSACTION xAddDropIdxRecommendationRETURN @ErrorEND COMMIT TRANSACTION xAddDropIdxRecommendationendfetch next from ms_drpi_iids into @database_id, @object_id, @index_idendif (object_id('[#tabIdxnametab]') is not null)begindrop table [#tabIdxnametab]enddeallocate ms_drpi_iidsreturn(0) -- auto_drop_indexgo---- JOBs for Executing [auto_create_index] and [auto_drop_index]-- DECLARE @jobId BINARY(16)EXEC msdb.dbo.sp_add_job @job_name=N'SQL MDW: Auto Index Management', @job_id = @jobId OUTPUTGOEXEC msdb.dbo.sp_add_jobstep @job_name=N'SQL MDW: Auto Index Management', @step_name=N'Auto Create Index', @step_id=1, @subsystem=N'TSQL', @command=N'EXECUTE [dbo].[auto_create_index]',@on_success_action = 3, -- on success, go to next step@database_name=N'AUTOINDEXRECS'GOEXEC msdb.dbo.sp_add_jobstep @job_name=N'SQL MDW: Auto Index Management', @step_name=N'Auto Drop Index', @step_id=2, @subsystem=N'TSQL', @command=N'EXECUTE [dbo].[auto_drop_index]', @database_name=N'AUTOINDEXRECS'GOEXEC msdb.dbo.sp_add_jobserver @job_name=N'SQL MDW: Auto Index Management'GODECLARE @schedule_id intEXEC msdb.dbo.sp_add_schedule@schedule_name = N'SQL MDW: Auto Index Management' ,@freq_type = 4, -- daily@freq_interval = 1, -- every day@freq_subday_type = 4, -- subday interval in minutes@freq_subday_interval = 30, -- every 30 minutes@schedule_id = @schedule_id OUTPUTEXEC msdb.dbo.sp_attach_schedule@job_name=N'SQL MDW: Auto Index Management',@schedule_id = @schedule_idgoThu, 31 Jan 2008 13:04:48 GMTdtevlinhttp://www.sqlservercentral.com/Forums/Topic448805-361-1.aspxIndexes can be, but aren't always, a performance problem when performing ETL. Best answer to that question is for you to test your load both ways, running with the indexes on and then running with a drop & recreate on the indexes. The one thing you can do to speed up either load is to, where possible, ensure that the data being loaded is in the same order as your clustered index. That helps regardless of whether you recreate the indexes or not.Wed, 30 Jan 2008 05:28:12 GMTGrant Fritcheyhttp://www.sqlservercentral.com/Forums/Topic448805-361-1.aspxYes , we will be implementing a horizontal partition on the main tables , with a partition for each month. regarding the indexes , we will be loading data once every 5 - 10 minutes, so i was wondering if i should use the indexes since the tables will have frequent inserts happening . I also need to implement full text indexing as well. I need the ETL to complete in 5 -10 minutes before it starts all over again.Indexes i think could become a performance bottle neck during etl , am i right ?Thanks again for your help.Tue, 29 Jan 2008 23:21:50 GMTJayanth_Kuruphttp://www.sqlservercentral.com/Forums/Topic448805-361-1.aspxYou want to normalize the data as much as you need to. There really isn't a "too far" or a "not far enough." Meet the business requirements in the best way possible. Remember that normalization not only increases data accuracy, but it reduces the amount of data stored. For example, you can create all the address information with 50 million customers, repeating addresses over and over again, or you can link to an address table and radically reduce the amount of data stored. That two table join is not going to seriously impact performance. Three, four, and 15 table joins won't seriously impact performance either if you've got good indexes, especially good clustered indexes. Flattening the structure reduces joins and simplifies queries, but it could make for poorer performance (you'll need to index more columns on the table and maintain that data on that one table with more page splits, more index rebuilds...). If flat files were better, we'd never have gone to relational databases in the first place.Tue, 29 Jan 2008 09:13:22 GMTGrant Fritcheyhttp://www.sqlservercentral.com/Forums/Topic448805-361-1.aspxWhat about table partitioning, will it work in your case?Tue, 29 Jan 2008 08:42:45 GMTLonerhttp://www.sqlservercentral.com/Forums/Topic448805-361-1.aspxHi We are planning to implement a database which etl's about 2GB of data daily.Estimating the overall size on the server to be close to 1 TB on a yearly basis.We are going to be using an ETL database and a Datawarehouse , from which we will load the data into cubes. Considering the ETL and the overall size of the database are there any specifics that i need to consider when designing the database. I am sorry i cant provide much detail right now but we are looking for alternatives in the database design so that we dont tradeoff too much on space or performance.I am thinking that I dont want to normalize the database too far. Am i right ???Thanks JayTue, 29 Jan 2008 06:31:51 GMTJayanth_Kurup