ETL Framework In Production

• Rahulmsb5

  SSC Veteran

  Points: 226

  More actions

  October 31, 2025 at 6:17 am

  #4679468

  Hello,

  I am leveraging Python within SQL Server Integration Services (SSIS) packages, primarily through the Execute Process Task.

   

  I am extracting Data from E2 SQL to Azure SQL Server.

  The connection has already been established.

  I am building complete ETL using Function

  Extraction of Data (Full load/Incremental Load)

  Transformation

  Logging

  Audit Table

  Loading

  The entire execution will be drive using Driver table or Configuration table.

  May someone please help me how can we achieve this parameter based ETL framework.

  do I need to create any config table or any metadata table how to pass value like .

  1. input parameter

     2. return value on output parameter

     3. What action function is performing

     Please suggest if any production grade ETL Framework, which helps to achieve this.

  Appreciate your  help.

  Thanks a lot

  • This topic was modified 13 hours, 29 minutes ago by Rahulmsb5.
  • This topic was modified 13 hours, 15 minutes ago by Rahulmsb5.