Phil Parkin (2/9/2015)
Your main problem here is that your lookup cache needs to be in-memory for optimum performance. Millions of rows in your target table = lots of RAM needed.What sort of column will you be matching on when deciding whether to insert or update ... what does the data look like?
Depends a bit on the lookup columns used.
I have used lookup transformations on data sets over 60 million rows without an issue, because I used integer columns.
Do not try this at home with GUIDS though 😀
Need an answer? No, you need a question
My blog at https://sqlkover.com.
MCSE Business Intelligence - Microsoft Data Platform MVP