Loading data from 5 GB flat file

Question

Loading data from 5 GB flat file

sunder.bugatha

Ten Centuries

Points: 1227
More actions
March 27, 2015 at 9:02 am

#295492

Did anyone work on designing a solution for loading data into SQL destination from a single 5/10 GB flat file? If yes, can you let me know what kind of performance measures you have taken while designing the solution ?

Viewing 4 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply

Steve Jones - SSC Editor SSC Guru Points: 734434 More actions · Answer 1

Are you looking to load a single 5GB file into a single table?

There are a variety of ways to do this. Is there a transform? Is it a straight load? Bulk insert is about the fastest way to do this. You can also use SSIS, and there is a paper on high performance loads here:

https://technet.microsoft.com/en-us/library/dd537533%28v=sql.100%29.aspx

Need a bit more detail.

Jeff Moden SSC Guru Points: 1003859 More actions · Answer 2

I think that someone on "spam patrol" deleted the other two posts by mistake. This is the third instance I've seen of this particular request in the last hour. I even posted on of those ol' "This is a duplicate post" responses to the original post that I had answered to.

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

Jeff Moden SSC Guru Points: 1003859 More actions · Answer 3

So, what does the file look like and what does the table look like? And, does the table have any indexes or constraints on it?

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)