• Our ETL process ingests data from over a hundred external clients, and what is accepted as "normal" is a constant debate. There is a pre-production load step called AutoCertification that performs record count, cardinality, and standard deviation queries before proceeding to load. If a dataset is outside a configured threshold, then it's held back in stage, flagged, and an alert will show in the ETL monitoring dashboard. If the data analyst clears an alert, then at that point the standard, min and max can be edited, thus defining a "new normal" for that specific data source going forward.

    "Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho