Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Need help in transfering 400 million from SQL Server to SQL Server Expand / Collapse
Author
Message
Posted Wednesday, May 1, 2013 12:13 PM


SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, May 21, 2014 2:16 AM
Points: 27, Visits: 267
Hi,

I have to transfer 400 million (with 8 columns) from SQL Server to SQL Server.I planned to do this in three ways.
1.
Creating an SSIS Pkg and calling it in SQL Agent Job Job step.
---or---
2.
"
insert into target_tbl
select columns from [linkserver].database.source_tbl
"
using above sql statement in SQL Job Agent Job step.
---or---
3
"
Drop table target_tbl
select columns into target_tbl from [linkserver].database.source_tbl
"
using above sql statement into SQL Job Agent Job step.

Please suggest me which one gives the best performance and why?
Thanks in advance...

Regards,
Rocky
Post #1448556
Posted Wednesday, May 1, 2013 12:44 PM
SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Thursday, October 2, 2014 12:09 PM
Points: 4,358, Visits: 9,538
I would recommend the SSIS approach - simply for the fact that you can then control the batch and commit sizes on the OLE DB Destination (using the fast load option).

Doing this - you can actually improve the performance over a linked server. Using a linked server will require the complete statement to be completed successfully - and then it will commit it to the database. This will require the space to be available in the transaction log to handle the 400 million rows to be inserted.

Using SSIS and defining a reasonable batch size (200000 would be a good starting point) - and a reasonable commit size (200000 would also be a good start here), you then only need enough space in the transaction log to account for the 200,000 rows. This assumes that your destination database is in simple recovery model. If the destination database is in full recovery model - you can add a step in the SSIS package to kick off the transaction log backup for that database.

I would also recommend that you don't perform a delete - rather, use truncate instead to clear the table before inserting the data if you are going to be performing a full refresh of the data.



Jeffrey Williams
Problems are opportunites brilliantly disguised as insurmountable obstacles.

How to post questions to get better answers faster
Managing Transaction Logs
Post #1448568
Posted Wednesday, May 1, 2013 12:58 PM


SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Wednesday, May 21, 2014 2:16 AM
Points: 27, Visits: 267
Thank you Jeffrey
Post #1448577
Posted Wednesday, May 1, 2013 4:01 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Today @ 6:45 AM
Points: 35,366, Visits: 31,902
I'd probably not use any of those methods especially if it's a one off task. I'd most likely do a native BCP out on the source and a native BCP in on th destination.

No matter what you do, make sure that it does it in batches and that you have transaction log backups running at pretty close intervals to keep the log file from blowing up if you're in the full recovery mode.


--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T." --22 Aug 2013

Helpful Links:
How to post code problems
How to post performance problems
Post #1448640
Posted Friday, May 3, 2013 10:33 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Today @ 8:21 AM
Points: 7,125, Visits: 12,721
Just to add to what Jeffrey said, if performance is of primary concern and you have a little time to tune the SSIS you can play around with adjusting the SSIS buffer sizes on your Data Flow to get maximum throughput. Reference: DefaultBufferMaxRows and DefaultBufferMaxSize

Adjust buffer size in SSIS data flow task by SQL Server Performance Team


__________________________________________________________________________________________________
There are no special teachers of virtue, because virtue is taught by the whole community. --Plato
Post #1449295
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse