Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
1 reads
Ed Elliott's Sql Developer Blog
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
166 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
4 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-30 (first published: 2019-10-16)
405 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-16
3 reads
How to test ETL Processes in production
This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...
2019-10-02
122 reads
How to test ETL Processes in production
This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...
2019-10-02
4 reads
How to connect spark to ms sql server without '[Error] [JvmBridge] java.sql.SQLException: No suitable driver'?
“[Error] [JvmBridge] java.sql.SQLException: No suitable driver” - unable to connect spark to Microsoft SQL Server.
In spark when you want to connect to a database you use Read() passing in...
2019-10-01
422 reads
How to connect spark to ms sql server without '[Error] [JvmBridge] java.sql.SQLException: No suitable driver'?
“[Error] [JvmBridge] java.sql.SQLException: No suitable driver” - unable to connect spark to Microsoft SQL Server.
In spark when you want to connect to a database you use Read() passing in...
2019-10-01
8 reads
How do test the upstream data is good in an etl process (etl testing part 3)?
In the first two parts of this series (https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ and https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/), I talked about how to unit test your business logic and integration test your ETL infrastructure code. Having...
2019-09-30
22 reads
Blogs
Running SQL Server on KubeVirt – Getting Started
With all the changes that have happened with VMware since the Broadcom acquisition I...
Why is shrinking IT budgets a good thing?
Each year around this time, companies enter the familiar ritual of budgeting. For many,...
Reverse Engineering a Physical Model Diagram
By Steve Jones
I recently wrote about a logical diagram with Redgate Data Modeler. That was interesting,...
Forums
Best analysis approach to reduce data type length in multiple tables
Hello SSC, I hope everyone is having a happy and safe holiday season! I...
Do you separate data/log files and TempDB files on different disks using AWS EC2
Hello, I am receiving multiple direction on deploying SQL Servers in AWS EC2. ...
Service account being in local admin in AWS EC2 environment?
Hello, I am working with a company on their database administration. I am trying...
Question of the Day
A New Operator
In SQL Server 2025, what does this code return?
SELECT '1' || '0'See possible answers