Click here to monitor SSC
SQLServerCentral is supported by Redgate
Log in  ::  Register  ::  Not logged in
Home       Members    Calendar    Who's On

Add to briefcase

How to handle "big data" or "large data" with SSIS ? Expand / Collapse
Posted Tuesday, October 29, 2013 1:39 PM


Group: General Forum Members
Last Login: Thursday, May 15, 2014 8:51 PM
Points: 183, Visits: 480
I am running dynamic queries that often return HUGE (300MB - 1GB) result sets initially. Later, it should not be this big (not sure about that though) because I will be using delta loading. These result sets are then loaded into a C# data table. A script loops over these rows and then generates a query (stored in SSIS variable) to load them to the appropriate destination columns (determined by other scripts).

For small result sets, my package runs properly. But, for big ones, it simply fails due to out of memory error.
How do I resolve this problem ? Can you suggest some strategies. I guess I could fetch smaller parts of the
data at a time and then load into target. Not sure how to go about it though. Is there a recipe for this ?
Post #1509550
Posted Wednesday, October 30, 2013 9:14 AM
Mr or Mrs. 500

Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500Mr or Mrs. 500

Group: General Forum Members
Last Login: Thursday, October 15, 2015 10:28 AM
Points: 504, Visits: 1,875
Duplicate post. See the following thread:
Post #1509830
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse