Low Level design for Error Logging for Data Pipeline

  • Hello,

    May someone please help me how to design Error Logging for the Data pipeline shown below.

    I am trying to design logging in Databricks CI Satellite EDLAP.

    Can I do it in ADLS Gen2 Silver layer or do I need to have any other component.

    Can Someone please help me how we can have folder structure and can we have delta table for logging. what all parameter we can log and how it's value can be capture.

    My understand:

    I can create a separate Delta table like error_logs where i can capture useful details such as: timestamp, table name, pipeline step, error message, source file, and maybe a JSON column to store the problematic row . I may Use try-except blocks in PySpark and append errors into this log table

    Any Implementation link will be very helpful, kindly share.

    logging

    Thanks a lot

    Attachments:
    You must be logged in to view attached files.
  • Thanks for posting your issue and hopefully someone will answer soon.

    This is an automated bump to increase visibility of your question.

Viewing 2 posts - 1 through 2 (of 2 total)

You must be logged in to reply to this topic. Login to reply