Great article Andy!

I found a unique way to reduce the number of rows I had to check, when I've got a SQL Server database as my source. I just add a "timestamp" column to each source table. Of course, this isn't really a timestamp its a varbinary that's database row version. Then I check for the minimum active row version, and pull only the rows that have been updated or inserted since my last incremental load.

This helps us reduce the number of rows we have to compare down to 40K or so to see if they were inserted or update. This is a great time saver when you would have to compare 100s of millions rows we have in our source database.

Thanks again - Linda