Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

744 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

You rated this post out of 5. Change rating

2024-11-11 (first published: )

1,793 reads

Technical Article

While Loop in T-SQL

Often, we encounter situations where we need to loop through a dataset to process or update records iteratively. In such cases, I use WHILE loop like below. The example below demonstrates how a WHILE loop can be used to iterate Sales records. This approach is particularly effective when working with a numerical or date column, […]

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(2)

You rated this post out of 5. Change rating

2024-10-15

630 reads

Blogs

Car Update September 2026

By

Software is hard. While I love our Lucid Gravity, I realize that they are...

Microsoft Tools for Making Data AI-Ready

By

Making Data AI-Ready, Part 2 (This is the second article in a three-part series...

Local Models Using GPUs with Ollama

By

Can you use your GPUs when running a local model under Ollama? You can,...

Read the latest Blogs

Forums

Checking the Error Log II

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Checking the Error Log II

Parameter Sniffing on SQL Server 2025: A Walkthrough with Real Numbers

By deepeshdhake

Comments posted to this topic are about the item Parameter Sniffing on SQL Server...

Designing for Teams

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Designing for Teams

Visit the forum

Question of the Day

Checking the Error Log II

What is the reason I should change my old code that uses xp_readerrorlog?

See possible answers