Abu Dina (2/22/2013)
By the way, the ID column appears to contain a mixture of integers and GUIDs. Should I create a new Identity column on the big table and use that as my ID? Would that help?
yes... and I would do the same on the destination table ... and make it a clustered index because you want to make sure all the inserts go at the end of the table.
Also, if you had an identity ID on the big table you could change the "WHERE NOT EXISTS (SELECT 1 FROM dbo.Loading_B2C_keys_ as b WHERE a.ID = b.ID)" in your inserts to "WHERE ID > @NextChunkID" instead. You would increment @NextChunkID by 1000000 (or whatever your top(N) is ) in each batch loop.
If I were doing this though, I would consider doing the transformations and writing rows to a CSV file. Then Bulk insert the end results back into a table. That would be a faster than the way you are doing it I think.
The probability of survival is inversely proportional to the angle of arrival.