Click here to monitor SSC
SQLServerCentral is supported by Redgate
 
Log in  ::  Register  ::  Not logged in
 
 
 


Load newly inserted data from millions of records on daily basis-Using SSIS


Load newly inserted data from millions of records on daily basis-Using SSIS

Author
Message
var05
var05
SSC-Enthusiastic
SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)

Group: General Forum Members
Points: 101 Visits: 437
Hi All,

I have to load data from one server to another server on daily basis using SSIS Package.

I have only one table to be populated from Source ->destination. But the source contains millions and millions of records.. I need to set up a job nightly, so that data is picked up from Source and loaded into the destination server.

Say if there are 100 new data coming in today into the source,then I wanted to look up the destinationand load only the newly inserted data....Is SSIS look up is the best option to look up the existing records in destination table and load the new ones from the source table???

Any best practices(performance wise) to implement the SSIS package to load data from Source->destination by looking up millions of records and load the new ones???

Thanks in advance.
robert.gerald.taylor
robert.gerald.taylor
Ten Centuries
Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)Ten Centuries (1.3K reputation)

Group: General Forum Members
Points: 1319 Visits: 1398
var05 (11/19/2012)
Hi All,

I have to load data from one server to another server on daily basis using SSIS Package.

I have only one table to be populated from Source ->destination. But the source contains millions and millions of records.. I need to set up a job nightly, so that data is picked up from Source and loaded into the destination server.

Say if there are 100 new data coming in today into the source,then I wanted to look up the destinationand load only the newly inserted data....Is SSIS look up is the best option to look up the existing records in destination table and load the new ones from the source table???

Any best practices(performance wise) to implement the SSIS package to load data from Source->destination by looking up millions of records and load the new ones???

The Lookup transofrmation will work fine. I'm assuming you have a unique business key that you can verify if a record is present or not with. Multiple millions of rows isn't all that many -- of course this is a generalization and depends upon your specific load window, network performance, ect....

Do you have a way to identify new and/or updated records on the source system? Something like a last updated and create date columns? If so, then you only need to pull records from the source that have changed in the last day.

HTH,
Rob
var05
var05
SSC-Enthusiastic
SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)SSC-Enthusiastic (101 reputation)

Group: General Forum Members
Points: 101 Visits: 437
Thanks....I am using the lookup transformation only...n again I don have to worry about updated records... It is just only the new records...

ThanksSmile
Phil Parkin
Phil Parkin
SSCrazy Eights
SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)SSCrazy Eights (8.3K reputation)

Group: General Forum Members
Points: 8300 Visits: 19445
var05 (11/19/2012)
Hi All,

--

Say if there are 100 new data coming in today into the source,then I wanted to look up the destinationand load only the newly inserted data....Is SSIS look up is the best option to look up the existing records in destination table and load the new ones from the source table???

Any best practices(performance wise) to implement the SSIS package to load data from Source->destination by looking up millions of records and load the new ones???

Thanks in advance.


You need to find a way of identifying newly inserted rows which does not require you to cross check all of them.

Usually, a row will have a date created column and you can compare that (by selecting rows where this date is greater than the max of this column in the destination data).

Alternatively, maybe an autoincrementing primary key on the source table - similar principle to the above.


Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

If the answer to your question can be found with a brief Google search, please perform the search yourself, rather than expecting one of the SSC members to do it for you.

Please surround any code or links you post with the appropriate IFCode formatting tags. It helps readability a lot.
Jeff Moden
Jeff Moden
SSC-Forever
SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)SSC-Forever (44K reputation)

Group: General Forum Members
Points: 44807 Visits: 39848
Instead of growing your own form of replication, why not just setup replication for that table?

--Jeff Moden

RBAR is pronounced ree-bar and is a Modenism for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
     Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column.
Although they tell us that they want it real bad, our primary goal is to ensure that we dont actually give it to them that way.
Although change is inevitable, change for the better is usually not.
Just because you can do something in PowerShell, doesnt mean you should. Wink

Helpful Links:
How to post code problems
How to post performance problems
Forum FAQs
Go


Permissions

You can't post new topics.
You can't post topic replies.
You can't post new polls.
You can't post replies to polls.
You can't edit your own topics.
You can't delete your own topics.
You can't edit other topics.
You can't delete other topics.
You can't edit your own posts.
You can't edit other posts.
You can't delete your own posts.
You can't delete other posts.
You can't post events.
You can't edit your own events.
You can't edit other events.
You can't delete your own events.
You can't delete other events.
You can't send private messages.
You can't send emails.
You can read topics.
You can't vote in polls.
You can't upload attachments.
You can download attachments.
You can't post HTML code.
You can't edit HTML code.
You can't post IFCode.
You can't post JavaScript.
You can post emoticons.
You can't post or upload images.

Select a forum

































































































































































SQLServerCentral


Search