Home Forums Data Warehousing Integration Services Load newly inserted data from millions of records on daily basis-Using SSIS RE: Load newly inserted data from millions of records on daily basis-Using SSIS

  • November 20, 2012 at 6:59 am

    #1560785

    var05 (11/19/2012)

    Hi All,

    --

    Say if there are 100 new data coming in today into the source,then I wanted to look up the destinationand load only the newly inserted data....Is SSIS look up is the best option to look up the existing records in destination table and load the new ones from the source table???

    Any best practices(performance wise) to implement the SSIS package to load data from Source->destination by looking up millions of records and load the new ones???

    Thanks in advance.

    You need to find a way of identifying newly inserted rows which does not require you to cross check all of them.

    Usually, a row will have a date created column and you can compare that (by selecting rows where this date is greater than the max of this column in the destination data).

    Alternatively, maybe an autoincrementing primary key on the source table - similar principle to the above.

    If you haven't even tried to resolve your issue, please don't expect the hard-working volunteers here to waste their time providing links to answers which you could easily have found yourself.