Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

788 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

You rated this post out of 5. Change rating

2024-11-11 (first published: )

1,932 reads

Blogs

T-SQL Tuesday #204 – Are blogs still relevant? (Roundup)

By

I did not expect 27 responses to my T-SQL Tuesday #204 invitation, but it...

Build vs Buy AI: When to adopt, partner, build or buy

By

Should you adopt an off-the-shelf AI tool, partner with a specialist, build in-house, or...

What's New in SQL Server for the Developer: SQL Server 2012

By

Part 1 of my What's New in SQL Server for the Developer series. IIF,...

Read the latest Blogs

Forums

Switching Schema II

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Switching Schema II

A Practical Approach to Automating SQL Server Email Notifications

By Kunal Rathi

Comments posted to this topic are about the item A Practical Approach to Automating...

Data Lost in the Cloud

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Data Lost in the Cloud

Visit the forum

Question of the Day

Switching Schema II

I run this query:

SELECT 
 s.name AS schemaname
 , o.name AS objectname
 FROM sys.objects o
 INNER JOIN sys.schemas s ON s.schema_id = o.schema_id
 WHERE o.name = 'Beer'
I get two rows back for the objects, named dbo.beer and etl.beer. Both the tables have identical schemas and are not partitioned. There are 25 rows in dbo.beer and 0 rows in etl.beer. I run this code:
ALTER TABLE dbo.beer switch TO etl.beer
What happens?

See possible answers