Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
3 reads
Ed Elliott's Sql Developer Blog
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
13 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
2 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
169 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
7 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-30 (first published: 2019-10-16)
408 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-16
6 reads
How to test ETL Processes in production
This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...
2019-10-02
122 reads
How to test ETL Processes in production
This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...
2019-10-02
8 reads
How to connect spark to ms sql server without '[Error] [JvmBridge] java.sql.SQLException: No suitable driver'?
“[Error] [JvmBridge] java.sql.SQLException: No suitable driver” - unable to connect spark to Microsoft SQL Server.
In spark when you want to connect to a database you use Read() passing in...
2019-10-01
424 reads
Blogs
T-SQL Tuesday #196 – Two risky career decisions I made
By DataOnWheels
The T-SQL Tuesday topic this month comes James Serra. What career risks have you...
T-SQL Tuesday #192: What career risks have you taken?
This T-SQL Tuesday is hosted by the one and only James Serra – literally...
T-SQL Tuesday #196: Taking Risks
By Steve Jones
This month we have a new host, James Serra. I’ve been trying to find...
Forums
would it be so terrible to install ssms on a few user desktops?
Hi, ssms is free here. I can think of other reasons to do this...
I'm thinking about submitting some articles
I've written some documentation on using different Markdown types of files on GitHub. It's...
Not Just an Upgrade
Comments posted to this topic are about the item Not Just an Upgrade
Question of the Day
Restoring On Top I
I am doing development work on a database and want to keep a backup so I can reset my database. I make some changes and want to restore over top of my changes. When I run this code, what happens?
USE Master BACKUP DATABASE DNRTest TO DISK = 'dnrtest.bak' GO USE DNRTest GO CREATE TABLE MyTest(myid INT) GO USE master RESTORE DATABASE DNRTest FROM DISK = 'dnrtest.bak' WITH REPLACESee possible answers