Incremental Data Loading using Azure Data Factory

  • Sucharita Das

    SSC Enthusiast

    Points: 176

    Comments posted to this topic are about the item Incremental Data Loading using Azure Data Factory

  • Renato10

    Valued Member

    Points: 74

    Hi Sucharita,

    Thank you for this article!

    I have some comments though:

    • This solution will work if the source data contains a WaterMark Column
    • Step 11: Add Parameters: These parameter values can be modified to load data from different source table to a different sink table.

      • But then the two stored procedures will fail?

    • Step 21: Check Data in Azure SQL Database

      • What if the source data has been changed during the Extraction process?

    • What next?

      • Having source data available in the Landing Zone is the first step
      • Then the source data is copied into another type of Corporate / Enterprise (transaction) data model depending on the implemented Data Warehouse methodology
      • And then the data in the Corporate / Enterprise (transaction) data model is copied into a Corporate / Enterprise information data model (the BUS Matrix of Dimensional Modeling) consisting of Facts and Dimensions

    Best regards,

    René

  • Sucharita Das

    SSC Enthusiast

    Points: 176

    Thank you for your feedback.

    My response on your questions/remarks:

    1. This solution will work if the source data contains a WaterMark Column -- yes.
    2. Step 11: Add Parameters: These parameter values can be modified to load data from different source table to a different sink table.But then the two stored procedures will fail? -- <b>dbo.usp_upsert_Student  should be replaced with the relevant one. Otherwise, most of the code is parameterized including the second SP.</b>
    3. Step 21: Check Data in Azure SQL Database What if the source data has been changed during the Extraction process? -- For any data movement operation, we assume that the source data remains the same before the validation. If data is changed, this validation is not possible in that particular iteration.
    4. What next? Having source data available in the Landing Zone is the first step

      Then the source data is copied into another type of Corporate / Enterprise (transaction) data model depending on the implemented Data Warehouse methodology

      And then the data in the Corporate / Enterprise (transaction) data model is copied into a Corporate / Enterprise information data model (the BUS Matrix of Dimensional Modeling) consisting of Facts and Dimensions -- yes. Once the data is transferred to the destination, many possible activities can be done on the data.

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic. Login to reply