Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

785 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

You rated this post out of 5. Change rating

2024-11-11 (first published: )

1,909 reads

Blogs

Wait… Could This Be the End of Index Maintenance?

By

While attending SQLCon and FabCon Europe in Barcelona, there were a lot of announcements...

Copilot on the Power BI Report Canvas vs. Modern Report Builder

By

(This is the second post in a three-part series on how AI is changing...

The #MS_DatabaseConnector## Role: #SQLNewBlogger

By

I had seen these roles in various instances, but never dug into the new...

Read the latest Blogs

Forums

Pay for Performance

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Pay for Performance, which is...

Packing and Decoding Device Status Codes with LEFT_SHIFT() and RIGHT_SHIFT() in SQL Server

By john.martin

Comments posted to this topic are about the item Packing and Decoding Device Status...

SQL Server Audit Failures

By Steve Jones - SSC Editor

Comments posted to this topic are about the item SQL Server Audit Failures

Visit the forum

Question of the Day

SQL Server Audit Failures

If I have a SQL Server Server Audit setup, what are my options for configuring the instance in the event it can't write to the audit log?

See possible answers