I am running a pretty straight forward dataflow that is:
Source -> Lookup -> Destination
I am in the process of trying to test run an incremental load. When I have no checked columns coming from the lookup table, this process is successful in roughly 5 mins (4 mill records cached).
Now I need to check some of the columns in the lookup so I can push them out to a conditional split to check for changes in the source data. When I check some of the columns in the lookup and rerun this process, it appears that it is caching all of the records, but then the package fails with the following errors:
Error: A buffer failed while allocating 10458336 bytes.
Error: The system reports 96 percent memory load. There are 17179336704 bytes of physical memory with 526725120 bytes free. There are 4294836224 bytes of virtual memory with 159760384 bytes free. The paging file has 34356768768 bytes with 14195691520 bytes free.
[SSIS.Pipeline] Error: The Data Flow task failed to create a buffer to call PrimeOutput for output "HZ_SCORE_EXPORT" (1) on component "OLE DB Source Output" (12). This error usually occurs due to an out-of-memory condition.
Based on this, I am assuming that it's a RAM issue? Please let me know if that is not the case and if so what might be the error.
Currently, this server has 16GB of RAM. If I need to add more, how much should I add?
Thanks for the help.