Exploring the Spark Connect gRPC API
All Spark Connect Posts
Code
Goal of this post In the first two posts, we looked at how to run some Spark code, firstly against a local Spark Connect server and...
2024-01-26
292 reads
All Spark Connect Posts
Code
Goal of this post In the first two posts, we looked at how to run some Spark code, firstly against a local Spark Connect server and...
2024-01-26
292 reads
All Spark Connect Posts
Goal of this post This post aims to show how we can create a .NET application, deploy it to Databricks, and then run a Databricks job...
2024-01-24 (first published: 2024-01-23)
70 reads
All Spark Connect Posts
When I wrote the spark-connect-dotnet lib I didn’t envisage that I would implement every function, instead it would be a combination of implementing the most common...
2024-01-17
1 reads
Spark Connect References https://spark.apache.org/docs/latest/spark-connect-overview.html
https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html
Reference Implementations Rust
GOLang
Source gRPC proto files https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect
Useful Videos If this doesn’t get you excited then nothing will it is, of course, suitable for work, it is...
2024-01-01
17 reads
I made a mistake recently when I was creating an ADF pipeline, annoyingly I made loads of changes and then clicked the debug button, when I pressed debug the...
2021-03-17 (first published: 2021-03-12)
836 reads
I made a mistake recently when I was creating an ADF pipeline, annoyingly I made loads of changes and then clicked the debug button, when I pressed debug the...
2021-03-12
11 reads
In my previous blog post I talked about how to read from an XML Webervice and use xpath to query the XML on the expressions side of things. You...
2021-03-04 (first published: 2021-02-26)
438 reads
In my previous blog post I talked about how to read from an XML Webervice and use xpath to query the XML on the expressions side of things. You...
2021-02-26
15 reads
When you use ADF, there are two sides to the coin. The first is the data itself that ADF does very well, from moving it from one site to...
2021-02-18 (first published: 2021-02-12)
427 reads
When you use ADF, there are two sides to the coin. The first is the data itself that ADF does very well, from moving it from one site to...
2021-02-12
18 reads
By Steve Jones
This value is something that I still hear today: our best work is done...
By gbargsley
Have you ever received the dreaded error from SQL Server that the TempDB log...
By Chris Yates
Artificial intelligence is no longer a distant concept. It is here, embedded in the...
Hi everyone I am writing an SP where there is logic inside the SP...
Comments posted to this topic are about the item Planning for tomorrow, today -...
We have a BI-application that connects to input tables on a SQL Server 2022...
I try to run this code on SQL Server 2022. All the objects exist in the database.
CREATE OR ALTER VIEW OrderShipping AS SELECT cl.CityNameID, cl.CityName, o.OrderID, o.Customer, o.OrderDate, o.CustomerID, o.cityId FROM dbo.CityList AS cl INNER JOIN dbo.[Order] AS o ON o.cityId = cl.CityNameID GO CREATE OR ALTER FUNCTION GetShipCityForOrder ( @OrderID INT ) RETURNS VARCHAR(50) WITH SCHEMABINDING AS BEGIN DECLARE @city VARCHAR(50); SELECT @city = os.CityName FROM dbo.OrderShipping AS os WHERE os.OrderID = @OrderID; RETURN @city; END; goWhat is the result? See possible answers