Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Need help in transfering 400 million from SQL Server to SQL Server


Need help in transfering 400 million from SQL Server to SQL Server

Author
Message
Rocky's
Rocky's
SSC Rookie
SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)

Group: General Forum Members
Points: 29 Visits: 341
Hi,

I have to transfer 400 million (with 8 columns) from SQL Server to SQL Server.I planned to do this in three ways.
1.
Creating an SSIS Pkg and calling it in SQL Agent Job Job step.
---or---
2.
"
insert into target_tbl
select columns from [linkserver].database.source_tbl
"
using above sql statement in SQL Job Agent Job step.
---or---
3
"
Drop table target_tbl
select columns into target_tbl from [linkserver].database.source_tbl
"
using above sql statement into SQL Job Agent Job step.

Please suggest me which one gives the best performance and why?
Thanks in advance...

Regards,
Rocky
Jeffrey Williams 3188
Jeffrey Williams 3188
SSCarpal Tunnel
SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)SSCarpal Tunnel (4.5K reputation)

Group: General Forum Members
Points: 4463 Visits: 9833
I would recommend the SSIS approach - simply for the fact that you can then control the batch and commit sizes on the OLE DB Destination (using the fast load option).

Doing this - you can actually improve the performance over a linked server. Using a linked server will require the complete statement to be completed successfully - and then it will commit it to the database. This will require the space to be available in the transaction log to handle the 400 million rows to be inserted.

Using SSIS and defining a reasonable batch size (200000 would be a good starting point) - and a reasonable commit size (200000 would also be a good start here), you then only need enough space in the transaction log to account for the 200,000 rows. This assumes that your destination database is in simple recovery model. If the destination database is in full recovery model - you can add a step in the SSIS package to kick off the transaction log backup for that database.

I would also recommend that you don't perform a delete - rather, use truncate instead to clear the table before inserting the data if you are going to be performing a full refresh of the data.

Jeffrey Williams
Problems are opportunities brilliantly disguised as insurmountable obstacles.

How to post questions to get better answers faster
Managing Transaction Logs

Rocky's
Rocky's
SSC Rookie
SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)SSC Rookie (29 reputation)

Group: General Forum Members
Points: 29 Visits: 341
Thank you Jeffrey
Jeff Moden
Jeff Moden
SSC-Forever
SSC-Forever (45K reputation)SSC-Forever (45K reputation)SSC-Forever (45K reputation)SSC-Forever (45K reputation)SSC-Forever (45K reputation)SSC-Forever (45K reputation)SSC-Forever (45K reputation)SSC-Forever (45K reputation)

Group: General Forum Members
Points: 45215 Visits: 39926
I'd probably not use any of those methods especially if it's a one off task. I'd most likely do a native BCP out on the source and a native BCP in on th destination.

No matter what you do, make sure that it does it in batches and that you have transaction log backups running at pretty close intervals to keep the log file from blowing up if you're in the full recovery mode.

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
Although they tell us that they want it real bad, our primary goal is to ensure that we dont actually give it to them that way.
Although change is inevitable, change for the better is not.
Just because you can do something in PowerShell, doesnt mean you should. Wink

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
Orlando Colamatteo
Orlando Colamatteo
SSCrazy Eights
SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)

Group: General Forum Members
Points: 8259 Visits: 14368
Just to add to what Jeffrey said, if performance is of primary concern and you have a little time to tune the SSIS you can play around with adjusting the SSIS buffer sizes on your Data Flow to get maximum throughput. Reference: DefaultBufferMaxRows and DefaultBufferMaxSize

Adjust buffer size in SSIS data flow task by SQL Server Performance Team

__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community. --Plato
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search