Ed Elliott's Sql Developer Blog

How to test ETL Processes in production

This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...

2019-10-02

6 reads

How do test the upstream data is good in an etl process (etl testing part 3)?

In the first two parts of this series (https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ and https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/), I talked about how to unit test your business logic and integration test your ETL infrastructure code. Having...

2019-09-30

2 reads

Blogs

Notifications in Azure Data Factory: Email, Internal Alerts, Log Analytics Alerts

By

(2019-Dec-15) While working on data integration projects and using Azure Data Factory as your main...

SSMS Tip: Portable User Settings

By

I’m a creature of habit. With the software tools that I use, I like...

Schema evolution solved using Delta Lake & Databricks

By

Don’t know about you, but one of my least favourite data pipeline errors is...

Read the latest Blogs

Forums

The mystery of SQL Memory

By WhiteLotus

Hi All, In server 2012 and 2016 , SQl error logs show this error...

Same DB, geographically diverse?

By SystemlordAnubis

Hi Everyone, I have a database (~50GB in size) with content that is constantly...

Updating Statistics

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Updating Statistics

Visit the forum

Ask SSC

SQL Server Q&A from the SQLServerCentral community

Get answers