Post reply

Merge data from two different source until the data is available from both sourc

  • May 13, 2025 at 8:40 pm

    #4590803

    I have two sources , one coming from datalake and another is file. we need to match date to combine both dates. But , the data from source may not be avilable daily or might be delayed.

    So, we have data from datalake coming daily but data from file comes only from Tuesday - Saturday , as we pull today -1 date (using ssis)

    example on thursday , we pull wednesday data.

    BUT On tuesday we get data from saturday , sunday and Monday.we need to wait untill both dates from datalake and file is available and then process.

    How can i process this combined based on date using SSIS  and SQL ? Any suggestion?

     

     

    • This topic was modified 1 month ago by komal145.
    • This topic was modified 4 weeks, 1 day ago by komal145.
    • This topic was modified 4 weeks, 1 day ago by komal145.
    • This topic was modified 4 weeks, 1 day ago by komal145.

  • May 14, 2025 at 9:10 pm

    #4591346

    Thanks for posting your issue and hopefully someone will answer soon.

    This is an automated bump to increase visibility of your question.

  • May 15, 2025 at 7:26 pm

    #4591802

    In your design you can create a batch history table. Then whatever script you run to insert into the tables from the data that only comes in twice weekly, add a statement at the end to insert a record into the batch history table for that batch run.

    In your main processing  , you can check against this table if the data you are expecting indeed loaded (maybe using max(date) in that batch history table).

    ----------------------------------------------------

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply