romina (8/20/2014)
MMartin1 (8/20/2014)
Hi Romina,Are you still using the looping? I don't think that will help. Just use a SCD transform and select it as a 'Historical Attribute' for the change type property for those columns you want history on and fixed or changing (as appropriate) for the rest. You may have done this already but I just want to be sure. Does this work any faster on the data load? Each change type will map to a different output from the SCD transform.
I've not really used this, only read about it so my working knowledge is limited. But I understand you need a derived column with the end date for the historically tagged records that changed (after scd transform), and then do a union all after with the new records. (Are you using something like validFrom , validTo in your destination?). I just dont think the looping containers are needed and they may be whats slowing you down.
MMartin,
I stopped using the loop. The reason being was it was overloading the server's memory after about 1 hour of processing.
I am now just using the SCD but it's been now nearly 17 hours that it's been running. Of course, I have had no problem with memory or CPU usage but 17 hours to process 4 millions it's very inefficient.
Any thoughts?
Koen has already provided a link to an article which he wrote which demonstrates how to do this without using the SCD component. I think that you should invest your time in rewriting your package accordingly.
If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.