• romina (8/19/2014)


    Thanks Koen and Phil,

    I haven't got too far with this. Still struggling to load the data. I am not sure what else to do. It's getting to my nerves.

    Any other ideas?

    Basically I am trying to read a table with over 4 million records (96,000 employee's data that has been extracted weekly over the last 56 weeks - a lot of duplicate data for the same employee!), apply an SCD (which will reduce the number of records dramatically by getting rid of duplicates) and load it into a Data warehouse.

    Your input will be highly appreciated ...

    Thanks

    Romina

    Do you have any control on the process that inserts the data into the staging table?

    Anyway, 4 million rows is not that much for SQL Server. A few decent written TSQL statement can take care of your problem in a few minutes.

    Need an answer? No, you need a question
    My blog at https://sqlkover.com.
    MCSE Business Intelligence - Microsoft Data Platform MVP