Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

746 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

You rated this post out of 5. Change rating

2024-11-11 (first published: )

1,808 reads

Blogs

Prep data for AI vs. Fabric Data Agent Instructions: What Goes Where?

By

Making Data AI-Ready, Part 3 (This is the final article in a three-part series...

Keeping Track of my ConEmu CLIs

By

I use ConEmu for my terminal interface. I’m still on Windows 10 at home,...

10 Decisions to Make Before You Create A Fabric Workspace

By

Creating a Fabric workspace takes about 30 seconds. Restructuring workspaces after people have built...

Read the latest Blogs

Forums

Alamat BCA KCP Mall Grand Indonesia Telp:08218520153

By layanan_Bca88

WhatsApp Cs 628218520153 Gedung Grand Indonesia Mall A, Lower Ground No. 20-21, Jl. M.H....

Alamat Kantor BCA KCP Senen Telp:08218520153

By R4nt4u

WhatsApp Cs 628218520153 Pusat Grosir Senen Jaya Blok A8-9, Lantai 2, RW.3, Senen, Kec....

BCA KCU PANGERAN JAYAKARTA Tlp:08218520153

By m4rt1n4

WhatsApp Cs 628218520153 Jl. P Jayakarta Komp. No.127, RT.7/RW.7, Mangga Dua Sel., Kecamatan Sawah...

Visit the forum

Question of the Day

Adding new column with DEFAULT II

Which number did the two COUNT(*) return:

DROP TABLE IF EXISTS #tmp
CREATE TABLE #tmp (id INT NOT NULL)

INSERT INTO #tmp (id) 
SELECT gs.value
  FROM GENERATE_SERIES(1, 5) AS gs

ALTER TABLE #tmp ADD my_value INT NOT NULL CONSTRAINT df_tmp_my_value DEFAULT 1

SELECT COUNT(*)  FROM #tmp AS t WHERE my_value = 1

ALTER TABLE #tmp DROP CONSTRAINT df_tmp_my_value

ALTER TABLE #tmp ADD CONSTRAINT df_tmp_my_value DEFAULT 2 FOR my_value

SELECT COUNT(*)  FROM #tmp AS t WHERE my_value = 1

See possible answers