Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

787 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

You rated this post out of 5. Change rating

2024-11-11 (first published: )

1,918 reads

Blogs

Are Blogs Still Relevant? I Think They Still Are.

By

As for me, I'll keep writing whenever time permits. I don't expect every post...

Reading Blogs: T-SQL Tuesday #204

By

This is an interesting invitation this month. Jeff Ianucci is a new host, and...

Are Blogs Still Relevant? - T-SQL Tuesday #204

By

T-SQL Tuesday is a monthly blog party hosted by a different community member each...

Read the latest Blogs

Forums

Pay for Performance

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Pay for Performance, which is...

Slow - Linked server query

By krypto69

Hi I have this slow query that is pulling data from a linked server....

Too Many Choices

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Too Many Choices

Visit the forum

Question of the Day

Too Many Choices

I have a table, dbo.MasterBrands, with this data:

BrandID OfficialName
1       Samsung
2       Apple
3       Microsoft
4       Adidas
5       Coca-Cola
I run this query. What is returned for Coca-Cola?
SELECT CHOOSE(BrandID, 'Conglomerate', 'Tech and Design', 'Desktop', 'Apparel')
FROM dbo.MasterBrands;

See possible answers