Low Level design for Error Logging for Data Pipeline

• sqlgigs

  SSC Enthusiast

  Points: 174

  More actions

  July 30, 2025 at 3:28 pm

  #4628666

  Hello,

  May someone please help me how to design Error Logging for the Data pipeline shown below.

  I am trying to design logging in Databricks CI Satellite EDLAP.

  Can I do it in ADLS Gen2 Silver layer or do I need to have any other component.

  Can Someone please help me how we can have folder structure and can we have delta table for logging. what all parameter we can log and how it's value can be capture.

  My understand:

  I can create a separate Delta table like error_logs where i can capture useful details such as: timestamp, table name, pipeline step, error message, source file, and maybe a JSON column to store the problematic row . I may Use try-except blocks in PySpark and append errors into this log table

  Any Implementation link will be very helpful, kindly share.

  

  Thanks a lot

  Attachments:

  You must be logged in to view attached files.

• Site Owners

  SSC Guru

  Points: 80356

  More actions

  July 31, 2025 at 4:10 pm

  #4629037

  Thanks for posting your issue and hopefully someone will answer soon.

  This is an automated bump to increase visibility of your question.