Incremental Data Loading using Azure Data Factory

  • Comments posted to this topic are about the item Incremental Data Loading using Azure Data Factory

  • Hi Sucharita,

    Thank you for this article!

    I have some comments though:

    • This solution will work if the source data contains a WaterMark Column
    • Step 11: Add Parameters: These parameter values can be modified to load data from different source table to a different sink table.

      • But then the two stored procedures will fail?

    • Step 21: Check Data in Azure SQL Database

      • What if the source data has been changed during the Extraction process?

    • What next?

      • Having source data available in the Landing Zone is the first step
      • Then the source data is copied into another type of Corporate / Enterprise (transaction) data model depending on the implemented Data Warehouse methodology
      • And then the data in the Corporate / Enterprise (transaction) data model is copied into a Corporate / Enterprise information data model (the BUS Matrix of Dimensional Modeling) consisting of Facts and Dimensions

    Best regards,

    René

  • Thank you for your feedback.

    My response on your questions/remarks:

    1. This solution will work if the source data contains a WaterMark Column -- yes.
    2. Step 11: Add Parameters: These parameter values can be modified to load data from different source table to a different sink table.But then the two stored procedures will fail? -- <b>dbo.usp_upsert_Student  should be replaced with the relevant one. Otherwise, most of the code is parameterized including the second SP.</b>
    3. Step 21: Check Data in Azure SQL Database What if the source data has been changed during the Extraction process? -- For any data movement operation, we assume that the source data remains the same before the validation. If data is changed, this validation is not possible in that particular iteration.
    4. What next? Having source data available in the Landing Zone is the first step

      Then the source data is copied into another type of Corporate / Enterprise (transaction) data model depending on the implemented Data Warehouse methodology

      And then the data in the Corporate / Enterprise (transaction) data model is copied into a Corporate / Enterprise information data model (the BUS Matrix of Dimensional Modeling) consisting of Facts and Dimensions -- yes. Once the data is transferred to the destination, many possible activities can be done on the data.

  • Hi Sucharita Das, thanks for the blog I really appreciate this, I have few questions though they are as below:

    1. What if we also want the delete operations to be get replaced in the destinations from the source, what needs to be changed in the existing stored procedure to achive that?
    2. Given that this pipeline will eventually work for a small a single tables, what process we need to do in order to build a for each loop and loop through certain tables and run the same pipeline for each table ?

    Appreciate your respone here, thanks again!

  • Hi Sucharita Das,

    I have a question, how can we do this same thing using an incrementing key instead of timestamp column. Also is it possible to perform delete operation .

    Thanks.

  • Thank you for your feedback.

    You may please refer the article https://www.sqlservercentral.com/articles/incremental-data-loading-through-adf-using-change-tracking.

    Let me know for any more question/query on this.

Viewing 6 posts - 1 through 6 (of 6 total)

You must be logged in to reply to this topic. Login to reply