Ed Elliott's Sql Developer Blog

SQLCover 0.5 - Fixes, smaller features and an exciting surprise

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

It has been a little while but I have updated SQLCover to include a number of fixes and small features, the majority of which are improvements to the html...

2019-10-16

17 reads

How to test ETL Processes in production

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

This is the final part in the four-part series into testing ETL pipelines, how exciting!
Part 1 - Unit Testing https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ Part 2 - Integration Testing https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/ Part 3...

2019-10-02

1 reads

How to connect spark to ms sql server without '[Error] [JvmBridge] java.sql.SQLException: No suitable driver'?

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

“[Error] [JvmBridge] java.sql.SQLException: No suitable driver” - unable to connect spark to Microsoft SQL Server.
In spark when you want to connect to a database you use Read() passing in...

2019-10-01

How do test the upstream data is good in an etl process (etl testing part 3)?

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

In the first two parts of this series (https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/ and https://the.agilesql.club/2019/08/how-do-we-prove-our-etl-processes-are-correct-how-do-we-make-sure-upstream-changes-dont-break-our-processes-and-break-our-beautiful-data/), I talked about how to unit test your business logic and integration test your ETL infrastructure code. Having...

2019-09-30

Visual Studio tSQLt Test Adapter

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

I finally got around to updating the tSQLt test adapter for visual studio, you can download it from:
https://marketplace.visualstudio.com/items?itemName=vs-publisher-263684.GoEddietSQLt2019
or the search in visual studio extensions thingy finds it as well.
For...

2019-09-02

14 reads

Help! My Apache Spark configuration on windows isn't working (I think)

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

Steps needed Getting Apache Spark running on windows involves:
Installing a JRE 8 (Java 1.8/OpenJDK 8) Downloading and extracting SPARK and setting SPARK_HOME Downloading winutils.exe and setting HADOOP_HOME If...

2019-08-29

11 reads

How do you debug your spark-dotnet app in visual studio?

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

When you run an application using spark-dotnet, to launch the application you need to use spark-submit to start a java virtual machine which starts the spark-dotnet driver which then...

2019-08-20

9 reads

How do we prove our ETL processes are correct? How do we make sure upstream changes don't break our processes and break our beautiful data?

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

ETL Testing Part 2 - Operational Data Testing
This is the second part of a series on ETL testing, the first part explained about unit testing, and in this part,...

2019-08-13

174 reads

spark-dotnet how to manually break a file into rows and columns

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

I found this question on stack overflow that went something like this:
“I have a file that includes line endings in the wrong place and I need to parse the...

2019-08-06

5 reads

java.lang.ClassNotFoundException: org.apache.spark.deploy.DotnetRunner with 0.4.0 of spark-dotnet

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

There was a breaking change with version 0.4.0 that changed the name of the class that is used to load the dotnet driver in Apache Spark.
To fix the issue...

2019-08-02

49 reads