Apache Spark from PHP - it is not just a .NET thing
All Spark Connect Posts
Code
Goal of this post I wanted to explore what the Spark Connect API looked like from other languages, I am not a php developer - I...
2024-02-10
10 reads
Ed Elliott
Implementing functions and more fun in Spark Connect using gRPC and .NET
All Spark Connect Posts
Code
Goal of this post The goal of this post is to look at creating a SparkSession and a DataFrame that will wrap the Range relation and...
2024-02-10
12 reads
Moving towards the DataFrame API using the Spark Connect gRPC API in .NET
All Spark Connect Posts
Code
Goal of this post So there are two goals of this post, the first is to take a look at Apache Arrow and how we can...
2024-02-05
27 reads
Exploring the Spark Connect gRPC API more
All Spark Connect Posts
Code
Goal of this post In this post we will continue looking at the gRPC API and the AnalyzePlan method which takes a plan and analyzes it....
2024-02-01
24 reads
Using Spark Connect from .NET
Introductory Ramble Spark Connect In July 2022, at the Data and AI summit, Apache Spark announced “Spark Connect,” which was a way of connecting to Apache Spark using the...
2024-01-31 (first published: 2024-01-17)
316 reads
Exploring the Spark Connect gRPC API
All Spark Connect Posts
Code
Goal of this post In the first two posts, we looked at how to run some Spark code, firstly against a local Spark Connect server and...
2024-01-26
72 reads
Using Spark Connect from .NET to run Spark jobs on Databricks
All Spark Connect Posts
Goal of this post This post aims to show how we can create a .NET application, deploy it to Databricks, and then run a Databricks job...
2024-01-24 (first published: 2024-01-23)
21 reads
Spark Connect
Spark Connect References https://spark.apache.org/docs/latest/spark-connect-overview.html
https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html
Reference Implementations Rust
GOLang
Source gRPC proto files https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect
Useful Videos If this doesn’t get you excited then nothing will it is, of course, suitable for work, it is...
2024-01-01
10 reads
ADF: Error trying to debug pipeline: BadRequest
I made a mistake recently when I was creating an ADF pipeline, annoyingly I made loads of changes and then clicked the debug button, when I pressed debug the...
2021-03-17 (first published: 2021-03-12)
753 reads
ADF: Error trying to debug pipeline: BadRequest
I made a mistake recently when I was creating an ADF pipeline, annoyingly I made loads of changes and then clicked the debug button, when I pressed debug the...
2021-03-12
6 reads
Blogs
Check your regions people
Today I was having a nice discussion with some colleagues about Fabric and pricing/licensing...
Using Git Prune–#SQLNewBlogger
By Steve Jones
As I’ve been working with SQL Saturday and managing changes to events, I’ve accumulated...
Microsoft Purview new data governance features
By James Serra
Starting last week is a rollout of the public preview of a new and...
Forums
Read Only Replica in SQL Server Standard
By Stewart "Arturius" Campbell
Comments posted to this topic are about the item Read Only Replica in SQL...
Identifying Customer Buying Pattern in Power BI - Part 1
Comments posted to this topic are about the item Identifying Customer Buying Pattern in...
Backup of encrypted databases failing
I've had some backups of my encrypted databases failing with the error "BACKUP 'DBName'...
Question of the Day
Read Only Replica in SQL Server Standard
Our environment runs using SQL Server Standard. We are implementing Availability groups. Our database has been experiencing high read volumes, so I want to let the application read the Synchronized Secondary replica, as I read that HADR does this. Can we implement this?
See possible answers