Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

788 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

You rated this post out of 5. Change rating

2024-11-11 (first published: )

1,934 reads

Blogs

Should You Use CHOOSE(): #SQLNewBlogger

By

I was experimenting with CHOOSE() recently after I was asked a question. In a...

T-SQL Tuesday #204 – Are blogs still relevant? (Roundup)

By

I did not expect 27 responses to my T-SQL Tuesday #204 invitation, but it...

Build vs Buy AI: When to adopt, partner, build or buy

By

Should you adopt an off-the-shelf AI tool, partner with a specialist, build in-house, or...

Read the latest Blogs

Forums

Cs Digital BCA KCP Palmerah Call//Cs:082374977784

By Halo_BCA

Wa:082374977784 Jl. Palmerah Barat No.38 A, RT.2/RW.3, Grogol Utara, Kec. Kebayoran Lama, Kota Jakarta...

Kontak BCA KCP Ratu Plaza Call//Cs:082374977784

By layanan 24jam BCA

Wa:082374977784 Jl. Jenderal Sudirman Lt Dasar, RT.1/RW.3, Gelora, Kecamatan Tanah Abang, Kota Jakarta Pusat,...

Weird problem altering logins with sysadmin account

By Brandie Tarvin

Has anyone ever seen this before? I have sysadmin permission on a server. I...

Visit the forum

Question of the Day

Switching Schema II

I run this query:

SELECT 
 s.name AS schemaname
 , o.name AS objectname
 FROM sys.objects o
 INNER JOIN sys.schemas s ON s.schema_id = o.schema_id
 WHERE o.name = 'Beer'
I get two rows back for the objects, named dbo.beer and etl.beer. Both the tables have identical schemas and are not partitioned. There are 25 rows in dbo.beer and 0 rows in etl.beer. I run this code:
ALTER TABLE dbo.beer switch TO etl.beer
What happens?

See possible answers