Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase

Load newly inserted data from millions of records on daily basis-Using SSIS Expand / Collapse
Author
Message
Posted Monday, November 19, 2012 4:22 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Saturday, February 01, 2014 2:14 PM
Points: 100, Visits: 435
Hi All,

I have to load data from one server to another server on daily basis using SSIS Package.

I have only one table to be populated from Source ->destination. But the source contains millions and millions of records.. I need to set up a job nightly, so that data is picked up from Source and loaded into the destination server.

Say if there are 100 new data coming in today into the source,then I wanted to look up the destinationand load only the newly inserted data....Is SSIS look up is the best option to look up the existing records in destination table and load the new ones from the source table???

Any best practices(performance wise) to implement the SSIS package to load data from Source->destination by looking up millions of records and load the new ones???

Thanks in advance.
Post #1386286
Posted Monday, November 19, 2012 7:02 AM
Ten Centuries

Ten CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen CenturiesTen Centuries

Group: General Forum Members
Last Login: Today @ 11:52 AM
Points: 1,135, Visits: 1,151
var05 (11/19/2012)
Hi All,

I have to load data from one server to another server on daily basis using SSIS Package.

I have only one table to be populated from Source ->destination. But the source contains millions and millions of records.. I need to set up a job nightly, so that data is picked up from Source and loaded into the destination server.

Say if there are 100 new data coming in today into the source,then I wanted to look up the destinationand load only the newly inserted data....Is SSIS look up is the best option to look up the existing records in destination table and load the new ones from the source table???

Any best practices(performance wise) to implement the SSIS package to load data from Source->destination by looking up millions of records and load the new ones???

The Lookup transofrmation will work fine. I'm assuming you have a unique business key that you can verify if a record is present or not with. Multiple millions of rows isn't all that many -- of course this is a generalization and depends upon your specific load window, network performance, ect....

Do you have a way to identify new and/or updated records on the source system? Something like a last updated and create date columns? If so, then you only need to pull records from the source that have changed in the last day.

HTH,
Rob
Post #1386373
Posted Monday, November 19, 2012 8:38 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Saturday, February 01, 2014 2:14 PM
Points: 100, Visits: 435
Thanks....I am using the lookup transformation only...n again I don have to worry about updated records... It is just only the new records...

Thanks:)
Post #1386435
Posted Tuesday, November 20, 2012 6:59 AM


SSCarpal Tunnel

SSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal TunnelSSCarpal Tunnel

Group: General Forum Members
Last Login: Today @ 10:56 AM
Points: 4,832, Visits: 11,200
var05 (11/19/2012)
Hi All,

--

Say if there are 100 new data coming in today into the source,then I wanted to look up the destinationand load only the newly inserted data....Is SSIS look up is the best option to look up the existing records in destination table and load the new ones from the source table???

Any best practices(performance wise) to implement the SSIS package to load data from Source->destination by looking up millions of records and load the new ones???

Thanks in advance.


You need to find a way of identifying newly inserted rows which does not require you to cross check all of them.

Usually, a row will have a date created column and you can compare that (by selecting rows where this date is greater than the max of this column in the destination data).

Alternatively, maybe an autoincrementing primary key on the source table - similar principle to the above.



Help us to help you. For better, quicker and more-focused answers to your questions, consider following the advice in this link.

When you ask a question (and please do ask a question: "My T-SQL does not work" just doesn't cut it), please provide enough information for us to understand its context.
Post #1386890
Posted Tuesday, November 20, 2012 10:10 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: General Forum Members
Last Login: Today @ 5:22 PM
Points: 36,016, Visits: 30,308
Instead of growing your own form of replication, why not just setup replication for that table?

--Jeff Moden
"RBAR is pronounced "ree-bar" and is a "Modenism" for "Row-By-Agonizing-Row".

First step towards the paradigm shift of writing Set Based code:
Stop thinking about what you want to do to a row... think, instead, of what you want to do to a column."

"Change is inevitable. Change for the better is not." -- 04 August 2013
(play on words) "Just because you CAN do something in T-SQL, doesn't mean you SHOULDN'T." --22 Aug 2013

Helpful Links:
How to post code problems
How to post performance problems
Post #1387031
« Prev Topic | Next Topic »

Add to briefcase

Permissions Expand / Collapse