SQL Clone
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Insert data in multiple batches for millions of record


Insert data in multiple batches for millions of record

Author
Message
s.ravisankar
s.ravisankar
SSC Journeyman
SSC Journeyman (77 reputation)SSC Journeyman (77 reputation)SSC Journeyman (77 reputation)SSC Journeyman (77 reputation)SSC Journeyman (77 reputation)SSC Journeyman (77 reputation)SSC Journeyman (77 reputation)SSC Journeyman (77 reputation)

Group: General Forum Members
Points: 77 Visits: 87
Hi,

I have table x, table y

table x have millions of record, so i want to insert data by multiple batches into table y.

Since I have millions of record, batch size should be configurable and after insert it should close loop based on count in table x.

Thanks in advanceSmile

Regards
Ravi
Joe Torre
Joe Torre
SSCrazy
SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)SSCrazy (2.6K reputation)

Group: General Forum Members
Points: 2626 Visits: 720
Use SSIS. It will handle inserts in batches behind the scenes without having to roll your own solution. You can do it manually using ORDER BY with OFFSET.
Steve Jones
Steve Jones
SSC Guru
SSC Guru (302K reputation)SSC Guru (302K reputation)SSC Guru (302K reputation)SSC Guru (302K reputation)SSC Guru (302K reputation)SSC Guru (302K reputation)SSC Guru (302K reputation)SSC Guru (302K reputation)

Group: Administrators
Points: 302674 Visits: 20005
Joe Torre - Thursday, December 7, 2017 9:30 AM
Use SSIS. It will handle inserts in batches behind the scenes without having to roll your own solution. You can do it manually using ORDER BY with OFFSET.


+1

Follow me on Twitter: @way0utwest
Forum Etiquette: How to post data/code on a forum to get the best help
My Blog: www.voiceofthedba.com
Jeff Moden
Jeff Moden
SSC Guru
SSC Guru (463K reputation)SSC Guru (463K reputation)SSC Guru (463K reputation)SSC Guru (463K reputation)SSC Guru (463K reputation)SSC Guru (463K reputation)SSC Guru (463K reputation)SSC Guru (463K reputation)

Group: General Forum Members
Points: 463766 Visits: 43803
s.ravisankar - Thursday, December 7, 2017 4:15 AM
Hi,

I have table x, table y

table x have millions of record, so i want to insert data by multiple batches into table y.

Since I have millions of record, batch size should be configurable and after insert it should close loop based on count in table x.

Thanks in advanceSmile

Regards
Ravi


1. How many "millions of record"?
2. How many bytes does the average "record" contain?
3. Are both tables in the same database, different databases on the same instance, different instances, or what ???

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
If you think its expensive to hire a professional to do the job, wait until you hire an amateur. -- Red Adair

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
Lynn Pettis
Lynn Pettis
SSC Guru
SSC Guru (204K reputation)SSC Guru (204K reputation)SSC Guru (204K reputation)SSC Guru (204K reputation)SSC Guru (204K reputation)SSC Guru (204K reputation)SSC Guru (204K reputation)SSC Guru (204K reputation)

Group: General Forum Members
Points: 204052 Visits: 40038
Jeff Moden - Thursday, December 7, 2017 12:07 PM
s.ravisankar - Thursday, December 7, 2017 4:15 AM
Hi,

I have table x, table y

table x have millions of record, so i want to insert data by multiple batches into table y.

Since I have millions of record, batch size should be configurable and after insert it should close loop based on count in table x.

Thanks in advanceSmile

Regards
Ravi


1. How many "millions of record"?
2. How many bytes does the average "record" contain?
3. Are both tables in the same database, different databases on the same instance, different instances, or what ???


Is table y empty at the start?
Do you need to verify that the data in x isn't already in y?

Cool
Lynn Pettis

For better assistance in answering your questions, click here
For tips to get better help with Performance Problems, click here
For Running Totals and its variations, click here or when working with partitioned tables
For more about Tally Tables, click here
For more about Cross Tabs and Pivots, click here and here
Managing Transaction Logs

SQL Musings from the Desert Fountain Valley SQL (My Mirror Blog)
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum







































































































































































SQLServerCentral


Search