Post reply

Indexing and Update Statistic on Source Data (ETL process)

  • November 20, 2008 at 6:22 pm

    #127746

    Hi there

    We have a flat file and as apart of ETL process, we bring this flat into a mapped table (called tblSourceFlatFile for instance and it's mapped out as it is with few additional columns).

    My question is:

    Before doing any tranfrom/massaging from this table into a destination table, is it worthwile to do Indexing and Update Statistic or not on this table (tblSourceFlatFile)? The new data is changing all the time and eacload is an average of 2 million records and we need to load/tranform this data everytime.

    I'm appreciated your comment/thought.

    Thanks

  • November 21, 2008 at 9:28 pm

    #901960

    It depends. How selective is your query to pull the data? If you have a high degree of selectivity (returning only a small portion of the entire table), it could make sense to do this as part of your ETL process. If you have a low degree of selectivity or are just dumping the entire table, then this would likely be a waste of time in terms of your ETL.

    hth,

    Tim

    Tim Mitchell
    TimMitchell.net | @tmitch.net | Tyleris.com
    ETL Best Practices

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic. Login to reply