Exploring the Spark Connect gRPC API
All Spark Connect Posts
Code
Goal of this post In the first two posts, we looked at how to run some Spark code, firstly against a local Spark Connect server and...
2024-01-26
313 reads
Ed Elliott's Sql Developer Blog
Using Spark Connect from .NET to run Spark jobs on Databricks
All Spark Connect Posts
Goal of this post This post aims to show how we can create a .NET application, deploy it to Databricks, and then run a Databricks job...
2024-01-24 (first published: 2024-01-23)
71 reads
Using Spark Connect from .NET
Introductory Ramble Spark Connect In July 2022, at the Data and AI summit, Apache Spark announced “Spark Connect,” which was a way of connecting to Apache Spark using the...
2024-01-31 (first published: 2024-01-17)
358 reads
Spark Connect Dotnet First Major Milestone
All Spark Connect Posts
When I wrote the spark-connect-dotnet lib I didn’t envisage that I would implement every function, instead it would be a combination of implementing the most common...
2024-01-17
3 reads
Spark Connect
Spark Connect References https://spark.apache.org/docs/latest/spark-connect-overview.html
https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html
Reference Implementations Rust
GOLang
Source gRPC proto files https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect
Useful Videos If this doesn’t get you excited then nothing will it is, of course, suitable for work, it is...
2024-01-01
18 reads
ADF: Error trying to debug pipeline: BadRequest
I made a mistake recently when I was creating an ADF pipeline, annoyingly I made loads of changes and then clicked the debug button, when I pressed debug the...
2021-03-17 (first published: 2021-03-12)
839 reads
ADF: Error trying to debug pipeline: BadRequest
I made a mistake recently when I was creating an ADF pipeline, annoyingly I made loads of changes and then clicked the debug button, when I pressed debug the...
2021-03-12
11 reads
ADF: Querying JSON documents
In my previous blog post I talked about how to read from an XML Webervice and use xpath to query the XML on the expressions side of things. You...
2021-03-04 (first published: 2021-02-26)
439 reads
ADF: Querying JSON documents
In my previous blog post I talked about how to read from an XML Webervice and use xpath to query the XML on the expressions side of things. You...
2021-02-26
17 reads
ADF, XML objects and XPath in the expression language
When you use ADF, there are two sides to the coin. The first is the data itself that ADF does very well, from moving it from one site to...
2021-02-18 (first published: 2021-02-12)
427 reads
Blogs
Why is shrinking IT budgets a good thing?
Each year around this time, companies enter the familiar ritual of budgeting. For many,...
Reverse Engineering a Physical Model Diagram
By Steve Jones
I recently wrote about a logical diagram with Redgate Data Modeler. That was interesting,...
Prepping for Certification, Part 3 of ?
By Brian Kelley
After I have understood the details for a certification test (part 1), including format...
Forums
Best analysis approach to reduce data type length in multiple tables
Hello SSC, I hope everyone is having a happy and safe holiday season! I...
Do you separate data/log files and TempDB files on different disks using AWS EC2
Hello, I am receiving multiple direction on deploying SQL Servers in AWS EC2. ...
Service account being in local admin in AWS EC2 environment?
Hello, I am working with a company on their database administration. I am trying...
Question of the Day
A New Operator
In SQL Server 2025, what does this code return?
SELECT '1' || '0'See possible answers