Incemental Load and Full Fresh Load Both

  • Hi,

    I have a requirement where I have to design the SSIS package for both Incremental and Full Load. If I am instructed to run the package as incremental then I run with incremental Load and If I am instructed to load full fresh I have to Load full fresh. I want to make this change(full fresh or incremental Load) dynamic. Here source we use is Eloqua connection manager and cannot do any filtering with source task.

    I added a conditional Split after source which filters data based on date. We have 32 tables and need to add this for all 32 tables. this change would be for incremental load. IF I want Full fresh I have to remove all conditional split from 32 objects,.

    Can any one of you suggest better way of handling this type of scenario?

    regards,

    Sushe

  • Couple questions....

    First, is there some sort of key value that you are able to use in your target system to determine where to pick up in source? In other words, how do you find/define your incremental dataset?

    Second, why not use a different source component? Use c# in a script source if you have to, but you'll need to have a way to pass a parameterized query to the source system.

    John Rowan

    ======================================================
    ======================================================
    Forum Etiquette: How to post data/code on a forum to get the best help[/url] - by Jeff Moden

  • hegdesuchi (10/2/2015)


    Hi,

    I have a requirement where I have to design the SSIS package for both Incremental and Full Load. If I am instructed to run the package as incremental then I run with incremental Load and If I am instructed to load full fresh I have to Load full fresh. I want to make this change(full fresh or incremental Load) dynamic. Here source we use is Eloqua connection manager and cannot do any filtering with source task.

    I added a conditional Split after source which filters data based on date. We have 32 tables and need to add this for all 32 tables. this change would be for incremental load. IF I want Full fresh I have to remove all conditional split from 32 objects,.

    Can any one of you suggest better way of handling this type of scenario?

    regards,

    Sushe

    Create a conditional split everywhere. Make it based on the datetime parameter. Condition is "datetime on 32 tables > @parameter".

    Set the @parameter to the actual latest datetime stamp you need for the incremental load, and set it to '1901-01-01' for the full load.

    Another less preferred way - create two versions of the SSIS packages, one with the condition the other without.

    M.Sc.IT, M.B.A, MCTS BI 2008, MCITP BI 2008, MCTS SQL Dev, CSM, CDVDM

Viewing 3 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic. Login to reply