Approaches to running Databricks ETL code from Azure ADF
Databricks is fantastic, but there is a small issue with how people use it. The problem is that Databricks is all things to all people. Data scientists and data...
2020-06-03
1 reads
Ed Elliott
The four tenets of ETL testing
Every ETL pipeline is only ever as reliable as the data that the upstream system provides. It is inevitable that assumptions you make about the data you are provided...
2020-03-06
13 reads
The four tenets of ETL testing
Every ETL pipeline is only ever as reliable as the data that the upstream system provides. It is inevitable that assumptions you make about the data you are provided...
2020-03-06
2 reads
2020-02-21
3 reads
Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
48 reads
Passing status messages and results back from Databricks to ADF
When we use ADF to call Databricks we can pass parameters, nice. When we finish running the Databricks notebook we often want to return something back to ADF so...
2020-02-05
2 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
11 reads
Spark Delta Lake, Updates, Deletes and Time Travel
When you use delta lake there are a couple of interesting things to note based around the fact that the data is stored in parquet files which are read-only...
2020-01-20
1 reads
SQLCover 0.5 - Fixes, smaller features and an exciting surprise
It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...
2019-10-30 (first published: 2019-10-16)
403 reads
Validating upstream data quality in ETL processes, SQL edition
It is a non-null constraint, not a non-ish-null constraint You are writing an ETL process, part of this process you need to import a semi-structured file (think CSV, JSON,...
2019-10-28
164 reads
Blogs
CHOOSE’ing a Beer: #SQLNewBlogger
By Steve Jones
We recently published an article on CHOOSE at SQL Server Central. I thought it...
Don’t Treat Your Data Catalog Like a Data Museum
Introduction Treating your data catalog like a “data museum”—a static collection where information quietly...
The trade-offs associated with low-code solutions
Low-code solutions often accelerate development and make tasks accessible to people who can’t or...
Forums
SQL availability group and file server role answering from the same name
Hello, I'm tasked with setting up a 2 node server cluster that will host...
Microsoft Power BI Performance Best Practices: Learn practical techniques for building high-speed Power BI solutions Microsoft Power BI Performance Best Practices
Comments posted to this topic are about the item Microsoft Power BI Performance Best...
DbaTools With Powershell - Omit The Id Column on CSV Export?
I am learning DbaTools in Powershell, and my current project is exporting a CSV...
Question of the Day
Replacing a Null
What is returned from this code in SQL Server 2022?
DECLARE @value INT = NULL; SELECT ISNULL(@value, 100.5) AS Result;See possible answers