Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

How to handle "big data" or "large data" with SSIS ? Expand / Collapse
Author
Message
Posted Tuesday, October 29, 2013 1:39 PM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Thursday, May 15, 2014 8:51 PM
Points: 183, Visits: 480
I am running dynamic queries that often return HUGE (300MB - 1GB) result sets initially. Later, it should not be this big (not sure about that though) because I will be using delta loading. These result sets are then loaded into a C# data table. A script loops over these rows and then generates a query (stored in SSIS variable) to load them to the appropriate destination columns (determined by other scripts).

For small result sets, my package runs properly. But, for big ones, it simply fails due to out of memory error.
How do I resolve this problem ? Can you suggest some strategies. I guess I could fetch smaller parts of the
data at a time and then load into target. Not sure how to go about it though. Is there a recipe for this ?
Post #1509550
Posted Wednesday, October 30, 2013 9:14 AM
SSC-Addicted

SSC-AddictedSSC-AddictedSSC-AddictedSSC-AddictedSSC-AddictedSSC-AddictedSSC-AddictedSSC-Addicted

Group: General Forum Members
Last Login: Monday, August 25, 2014 11:36 AM
Points: 493, Visits: 1,867
Duplicate post. See the following thread:
http://www.sqlservercentral.com/Forums/Topic1509553-364-1.aspx
Post #1509830
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse