ETL/SSIS/Azure Data Factory

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

  • Script

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

5 (1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

550 reads

Blogs

From Couch-Potato to Triathlete – and What This Means for Your SQL Server

By

Do you know if your SQL Server is really running at its best? To...

Retro Data 2025 – Slidedeck

By

You can find the slides of my session on the €100 DWH in Azure...

The Book of Redgate: We Value Teams

By

This value is something that I still hear today: our best work is done...

Read the latest Blogs

Forums

Is there a way for SP to know who called it?

By water490

Hi everyone I am writing an SP where there is logic inside the SP...

Planning for tomorrow, today - database migrations

By John Martin

Comments posted to this topic are about the item Planning for tomorrow, today -...

Bottlenecks on SQL Server performance

By runarlan

We have a BI-application that connects to input tables on a SQL Server 2022...

Visit the forum

Question of the Day

The Tightly Linked View

I try to run this code on SQL Server 2022. All the objects exist in the database.

CREATE OR ALTER VIEW OrderShipping
AS
SELECT cl.CityNameID,
       cl.CityName,
       o.OrderID,
       o.Customer,
       o.OrderDate,
       o.CustomerID,
       o.cityId
 FROM dbo.CityList AS cl
 INNER JOIN dbo.[Order] AS o ON o.cityId = cl.CityNameID
GO
CREATE OR ALTER FUNCTION GetShipCityForOrder
(
    @OrderID INT
)
RETURNS VARCHAR(50)
WITH SCHEMABINDING
AS
BEGIN
    DECLARE @city VARCHAR(50);
    SELECT @city = os.CityName
    FROM dbo.OrderShipping AS os
    WHERE os.OrderID = @OrderID;
    RETURN @city;
END;
go
What is the result?

See possible answers