KQL Series – ingesting data via Event Grid into Azure Data Explorer

This blog post is about using event grid to ingest data into Azure Data Explorer and was a method I had to use with a client.

It was awesome because it forced me to write some C# code for an Azure function – so be nice and don’t judge the code. Your code is always better than mine….

To ingest data from Event Grid into Azure Data Explorer, we can follow these steps:

1. Create an Event Grid subscription: You can create an Event Grid subscription to subscribe to the events we want to ingest into Azure Data Explorer. When an event is published to the Event Grid topic, Event Grid sends a notification to our subscription.
2. Create an Azure Function: We can create an Azure Function that triggers when an event is published to the Event Grid topic. The function will receive the event data as input.
3. Prepare the event data for ingestion: In the Azure Function, we can prepare the event data for ingestion into Azure Data Explorer. This may include parsing the event data and transforming it into a format that can be ingested by Azure Data Explorer.
4. Ingest the event data into Azure Data Explorer: Using the Azure Data Explorer .NET SDK, we can ingest the event data into Azure Data Explorer. We can use the SDK to create a table and ingest the data into that table.

Here’s an example Azure Function that ingests Event Grid data into Azure Data Explorer:

using System;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.EventGrid.Models;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using Kusto.Data.Common;
using Kusto.Data.Net.Client;
public static void Run(EventGridEvent eventGridEvent, ILogger log)
{
    // Get the data from the event
    var eventData = (dynamic)eventGridEvent.Data;
    // Prepare the data for ingestion into Azure Data Explorer
    var timestamp = DateTime.UtcNow;
    var value = eventData.Value;
    // Ingest the data into Azure Data Explorer
    var kustoConnectionStringBuilder = new KustoConnectionStringBuilder("https://<clustername>.<region>.kusto.windows.net", "<databasename>")
        .WithAadApplicationKeyAuthentication("<applicationId>", "<applicationKey>", "<tenantId>");
    using (var kustoClient = KustoClientFactory.CreateCslQueryProvider(kustoConnectionStringBuilder))
    {
        var command = $"'.create table <tablename> (Timestamp:datetime, Value:real)'n"
            + $"'| .set-or-replace <tablename> <| {timestamp:s} , {value} |>'";
        await kustoClient.ExecuteControlCommandAsync(command);
    }
    log.LogInformation($"Data ingested into Azure Data Explorer: {timestamp} {value}");
}

Replace <clustername>, <region>, <databasename>, <applicationId>, <applicationKey>, <tenantId>, and <tablename> with your own values.

In my example I am using an Azure AD application key for authentication. Alternatively, you could use Azure AD user authentication or a managed identity for authentication.

You can find out more information here:

https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-event-grid?tabs=adx

Yip.