Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
13 reads
Ed Elliott
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
19 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
3 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-30 (first published: 2019-10-16)
415 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
177 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
8 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-16
12 reads
How to test ETL Processes in production
This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...
2019-10-02
134 reads
How to test ETL Processes in production
This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...
2019-10-02
8 reads
How to connect spark to ms sql server without '[Error] [JvmBridge] java.sql.SQLException: No suitable driver'?
“[Error] [JvmBridge] java.sql.SQLException: No suitable driver” - unable to connect spark to Microsoft SQL Server.
In spark when you want to connect to a database you use Read() passing in...
2019-10-01
428 reads
Blogs
AI Helps Me with My Sloppiness
By Steve Jones
I type fairly well. Well, I type fast, but I do wear out a...
WITH clause gets a 26ai boost!
By way of background, a while back I did video called “My New Favourite...
Automatic Index Compaction
By ReviewMyDB
Index maintenance has always meant nightly jobs and a window you have to defend....
Forums
SQL Art, Part 4: Happy 4th of July — A British DBA's Guide to Celebrating a War We Don't Talk About
Comments posted to this topic are about the item SQL Art, Part 4: Happy...
ALAMAT KONTAK BCA KCU PONTIANAK Telp,Cs:08218154393
WA:08218154393 Jl. Jenderal Ahmad Yani No.1, Benua Melayu Darat, Kec. Pontianak Sel., Kota Pontianak,...
ALAMAT KONTAK BCA KCU BALIKPAPAN Telp,Cs:08218154393
WA:08218154393 Jl. Jenderal Sudirman No.139, Klandasan Ilir, Kec. Balikpapan Kota, Kota Balikpapan, Kalimantan Timur...
Question of the Day
Getting the Average
I have this data in the dbo.Commission table in a SQL Server 2022 database.
salesperson commission Brian 12 Brian 16 Andy 7 Andy 14 Andy 21 Steve 20 Steve NULLAll the data is a varchar, and I decide to run this query to get the totals for each salesperson.
SELECT SalesPerson
, AVG(TRY_PARSE(Commission AS int)) AS TotalCommission
FROM commission
GROUP BY SalesPerson
GO
What average commission is calculated for Steve? See possible answers