Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

785 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

You rated this post out of 5. Change rating

2024-11-11 (first published: )

1,907 reads

Blogs

The #MS_DatabaseConnector## Role: #SQLNewBlogger

By

I had seen these roles in various instances, but never dug into the new...

Asking for Help Isn’t a Weakness

By

During COVID, I attended training on software/enterprise architecture given by Mark Richards. One of...

The Proof of the Backup Is in the Pudding

By

For me, the objective is straightforward: have a backup strategy that is reliable, understandable,...

Read the latest Blogs

Forums

Access + ODBC Driver 18 + SQL Server 2025: ~150–250 ms per request

By Bruno De Grauwe

Has anyone encountered recurring request latency when using Microsoft Access with SQL Server 2025...

The ##MS_DatabaseConnector## User

By Steve Jones - SSC Editor

Comments posted to this topic are about the item The ##MS_DatabaseConnector## User

Planning for Change

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Planning for Change

Visit the forum

Question of the Day

The ##MS_DatabaseConnector## User

On SQL Server 2025, I run this code as a sysadmin:

CREATE LOGIN Demo WITH PASSWORD = 'Demo1234567'
GO
ALTER SERVER ROLE ##MS_DatabaseConnector## ADD MEMBER Demo
GO
I then login as Demo and run this:
USE sandbox
go
SELECT CURRENT_USER
What is returned?

See possible answers