romina (8/19/2014)
Thanks Koen and Phil,I haven't got too far with this. Still struggling to load the data. I am not sure what else to do. It's getting to my nerves.
Any other ideas?
Basically I am trying to read a table with over 4 million records (96,000 employee's data that has been extracted weekly over the last 56 weeks - a lot of duplicate data for the same employee!), apply an SCD (which will reduce the number of records dramatically by getting rid of duplicates) and load it into a Data warehouse.
Your input will be highly appreciated ...
Thanks
Romina
Do you have any control on the process that inserts the data into the staging table?
Anyway, 4 million rows is not that much for SQL Server. A few decent written TSQL statement can take care of your problem in a few minutes.
Need an answer? No, you need a question
My blog at https://sqlkover.com.
MCSE Business Intelligence - Microsoft Data Platform MVP