SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


SQL Server Index and Performance tuning ETL


SQL Server Index and Performance tuning ETL

Author
Message
avinash jobanputra
avinash jobanputra
SSC-Enthusiastic
SSC-Enthusiastic (137 reputation)SSC-Enthusiastic (137 reputation)SSC-Enthusiastic (137 reputation)SSC-Enthusiastic (137 reputation)SSC-Enthusiastic (137 reputation)SSC-Enthusiastic (137 reputation)SSC-Enthusiastic (137 reputation)SSC-Enthusiastic (137 reputation)

Group: General Forum Members
Points: 137 Visits: 468
HI Im working as a SQL DBA where im currently supporting a lage volume of data migration from a legacy system to a brand new product.

One of the migrations consist of a large volume of data with billions of rows of data in some staging tables. I have a couple of questions here which im hoping the experts can contribute to in terms of performance tuning.

1) We have created a number of indexes to improve the read time of some queries. However these tables then have large inserts going into them. Does indexing not slow down the insert.
2) This specific migration is going to be running consecutively for over 24 hours. As a test I did a rebuild and reorganise half way through and this helped. Is this something valid to suggest in production.
Thanks
GilaMonster
GilaMonster
SSC Guru
SSC Guru (418K reputation)SSC Guru (418K reputation)SSC Guru (418K reputation)SSC Guru (418K reputation)SSC Guru (418K reputation)SSC Guru (418K reputation)SSC Guru (418K reputation)SSC Guru (418K reputation)

Group: General Forum Members
Points: 418528 Visits: 47141
avinash jobanputra (1/30/2011)
1) We have created a number of indexes to improve the read time of some queries. However these tables then have large inserts going into them. Does indexing not slow down the insert.


Yes. Indexing is a tradeoff. Test and see what kind of hit you take on the inserts. Usually a small number of indexes is fine, but do test.

2) This specific migration is going to be running consecutively for over 24 hours. As a test I did a rebuild and reorganise half way through and this helped. Is this something valid to suggest in production.


Absolutely. You should be doing regular index maintenance. Not rebuild and reorg, the latter is redundant, but something. This script's good if the DB is large. http://sqlfool.com/2010/04/index-defrag-script-v4-0/

Gail Shaw
Microsoft Certified Master: SQL Server, MVP, M.Sc (Comp Sci)
SQL In The Wild: Discussions on DB performance with occasional diversions into recoverability

We walk in the dark places no others will enter
We stand on the bridge and no one may pass


free_mascot
free_mascot
SSChampion
SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)SSChampion (12K reputation)

Group: General Forum Members
Points: 12163 Visits: 2250
When you are transfering large data to new table; I suggest you create just basic structure of the table i.e. without indexes, rules etc.

Once transfer is done you can create the required indexes depending on your queries.

This will be much faster.

HTH

---------------------------------------------------
"Thare are only 10 types of people in the world:
Those who understand binary, and those who don't."
Grant Fritchey
Grant Fritchey
SSC Guru
SSC Guru (165K reputation)SSC Guru (165K reputation)SSC Guru (165K reputation)SSC Guru (165K reputation)SSC Guru (165K reputation)SSC Guru (165K reputation)SSC Guru (165K reputation)SSC Guru (165K reputation)

Group: General Forum Members
Points: 165237 Visits: 33203
free_mascot (1/31/2011)
When you are transfering large data to new table; I suggest you create just basic structure of the table i.e. without indexes, rules etc.

Once transfer is done you can create the required indexes depending on your queries.

This will be much faster.

HTH


Maybe, maybe not. Inserts into tables with a well-constructed clustered index are faster than inserts into a heap table. It's all the additional non-clustered indexes and foreign key constraints that can slow down inserts. But even these may benefit inserts depending on how the inserts are being done. You really have to address each situation individually and not use a blanket approach.

----------------------------------------------------
The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood...
Theodore Roosevelt

The Scary DBA
Author of: SQL Server Query Performance Tuning and SQL Server Execution Plans
Product Evangelist for Red Gate Software
TheSQLGuru
TheSQLGuru
SSC Guru
SSC Guru (58K reputation)SSC Guru (58K reputation)SSC Guru (58K reputation)SSC Guru (58K reputation)SSC Guru (58K reputation)SSC Guru (58K reputation)SSC Guru (58K reputation)SSC Guru (58K reputation)

Group: General Forum Members
Points: 58896 Visits: 8830
Given that you have massive volumes of data to work with and yet are asking basic questions I really must encourage you to get a professional to help you with your migration efforts.

I will throw out that it can often be much more efficient to drop (some or all) indexes on tables that are going to have large amounts of data added to them in an ETL process and then rebuild them after load completion.

Best,
Kevin G. Boles
SQL Server Consultant
SQL MVP 2007-2012
TheSQLGuru on googles mail service
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum







































































































































































SQLServerCentral


Search