Ed Elliott's Sql Developer Blog

Synapse Analytics and .NET for Apache Spark Example 3 - CTE()

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

The next example is how to do a CTE (Common Table Expression). When creating the CTE I will also rename one of the columns from “dataType” to “x”.
T-SQL WITH...

2021-01-19

5 reads

Synapse Analytics Spark Example 2 - ROW_NUMBER()

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

The next example is how to do a ROW_NUMBER(), my favourite window function.
T-SQL SELECT *, ROW_NUMBER() OVER(ORDER BY dateTime) as RowNumber FROM chicago.safety_data Spark SQL SELECT *, ROW_NUMBER() OVER(ORDER...

2021-01-18

13 reads

Synapse Analytics and .NET for Apache Spark Example 2 - ROW_NUMBER()

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

The next example is how to do a ROW_NUMBER(), my favourite window function.
T-SQL SELECT *, ROW_NUMBER() OVER(ORDER BY dateTime) as RowNumber FROM chicago.safety_data Spark SQL SELECT *, ROW_NUMBER() OVER(ORDER...

2021-01-18

Synapse Analytics Spark Example 1 - Group By

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

I have been playing around with the new Azure Synapse Analytics, and I realised that this is an excellent opportunity for people to move to Apache Spark. Synapse Analytics...

2021-01-18

11 reads

Synapse Analytics and .NET for Apache Spark Example 1 - Group By

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

I have been playing around with the new Azure Synapse Analytics, and I realised that this is an excellent opportunity for people to move to Apache Spark. Synapse Analytics...

2021-01-18

1 reads

How to get data in a DataFrame via .NET for Apache Spark

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

When I first started working with Apache Spark, one of the things I struggled with was that I would have some variable or data in my code that I...

2021-01-17

14 reads

Git in 10 commands to do 99% of gitting and keeping out of trouble

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

Git is hard, probably harder than it needs to be but I have been using it for about 5 years and have a workflow that works for me. I...

2021-01-18 (first published: 2021-01-12)

890 reads

TF-IDF in .NET for Apache Spark Using Spark ML v2

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

Spark ML in .NET for Apache Spark Apache Spark has had a machine learning API for quite some time and this has been partially implemented in .NET for Apache...

2020-12-19

20 reads

Blog .NET for Apache Spark UDFs Missing Shared State

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

The Problem When you use a UDF in .NET for Apache Spark, something like this code:
class Program { static void Main(string args) { var spark = SparkSession.Builder().GetOrCreate(); _logging.AppendLine("Starting Select");...

2020-10-29

22 reads

Spark .NET - Spark ML TF-IDF

• by Ed Elliott

• Ed Elliott's Sql Developer Blog

Spark ML in .NET for Apache Spark Spark is awesome, .NET is awesome, machine learning (ML) is awesome, so what could be better than using .NET to write ML...

2020-07-07

22 reads