Moving towards the DataFrame API using the Spark Connect gRPC API in .NET
All Spark Connect Posts
Code
Goal of this post So there are two goals of this post, the first is to take a look at Apache Arrow and how we can...
2024-02-05
90 reads
Ed Elliott's Sql Developer Blog
Exploring the Spark Connect gRPC API more
All Spark Connect Posts
Code
Goal of this post In this post we will continue looking at the gRPC API and the AnalyzePlan method which takes a plan and analyzes it....
2024-02-01
117 reads
Exploring the Spark Connect gRPC API
All Spark Connect Posts
Code
Goal of this post In the first two posts, we looked at how to run some Spark code, firstly against a local Spark Connect server and...
2024-01-26
331 reads
Using Spark Connect from .NET to run Spark jobs on Databricks
All Spark Connect Posts
Goal of this post This post aims to show how we can create a .NET application, deploy it to Databricks, and then run a Databricks job...
2024-01-24 (first published: 2024-01-23)
103 reads
Using Spark Connect from .NET
Introductory Ramble Spark Connect In July 2022, at the Data and AI summit, Apache Spark announced “Spark Connect,” which was a way of connecting to Apache Spark using the...
2024-01-31 (first published: 2024-01-17)
363 reads
Spark Connect Dotnet First Major Milestone
All Spark Connect Posts
When I wrote the spark-connect-dotnet lib I didn’t envisage that I would implement every function, instead it would be a combination of implementing the most common...
2024-01-17
7 reads
Spark Connect
Spark Connect References https://spark.apache.org/docs/latest/spark-connect-overview.html
https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html
Reference Implementations Rust
GOLang
Source gRPC proto files https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect
Useful Videos If this doesn’t get you excited then nothing will it is, of course, suitable for work, it is...
2024-01-01
22 reads
ADF: Error trying to debug pipeline: BadRequest
I made a mistake recently when I was creating an ADF pipeline, annoyingly I made loads of changes and then clicked the debug button, when I pressed debug the...
2021-03-17 (first published: 2021-03-12)
844 reads
ADF: Error trying to debug pipeline: BadRequest
I made a mistake recently when I was creating an ADF pipeline, annoyingly I made loads of changes and then clicked the debug button, when I pressed debug the...
2021-03-12
14 reads
ADF: Querying JSON documents
In my previous blog post I talked about how to read from an XML Webervice and use xpath to query the XML on the expressions side of things. You...
2021-03-04 (first published: 2021-02-26)
442 reads
Blogs
PowerPoint to HTML with Claude AI
By Steve Jones
I had an idea for an animated view of a sales tool, and started...
Don’t Miss Out – SQL Server Query Tuning Fundamentals Starts Next Monday!
Next Monday, February 9, 2026, my one-day live online training SQL Server Query Tuning...
Monday Monitor Tips: SQL Auditing Preview
By Steve Jones
One of the features we advocates have been advocating for is a better way...
Forums
Delta Lake: The Definitive Guide: Modern Data Lakehouse Architectures with Data Lakes
Comments posted to this topic are about the item Delta Lake: The Definitive Guide:...
Databricks Data Intelligence Platform: Unlocking the GenAI Revolution
Comments posted to this topic are about the item Databricks Data Intelligence Platform: Unlocking...
Storytelling with Data: Let's Practice!
Comments posted to this topic are about the item Storytelling with Data: Let's Practice!
Question of the Day
The Decoded Value
In SQL Server 2025, what is returned from this code:
DECLARE @message VARCHAR(50) = 'Hello SQL Server 2025!'; DECLARE @encoded VARCHAR(MAX); SET @encoded = BASE64_ENCODE(CAST(@message AS VARBINARY(1000))); SELECT BASE64_DECODE(@encoded)See possible answers