Scripts

Technical Article

Real-Time SQL Server to BigQuery Streaming ETL using CDC

CDC Changes: The script queries the CDC tables in SQL Server to retrieve the changes (inserts, updates, deletes) since the last sync. Each change is processed with a mapped operation type (INSERT, UPDATE, DELETE).
Real-Time Streaming to BigQuery: The captured changes are streamed directly to BigQuery using its real-time insert_rows_json method, avoiding the need for batch uploads via Google Cloud Storage.
Tracking Last Sync Time: The script tracks the last synchronization time and updates it after every successful sync, ensuring no data is missed.
Low Latency: By continuously querying the CDC tables and streaming the changes, the script achieves near real-time data synchronization.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

(1)

You rated this post out of 5. Change rating

2024-11-13 (first published: )

787 reads

Technical Article

Concatenating Multiple Row Values into a Single Comma-Separated List

In scenarios where you need to consolidate multiple rows into a single, comma-separated value, you can achieve this using FOR XML PATH. This script demonstrates how to retrieve volunteer data and display the days they have selected for participation.

★ ★ ★ ★ ★ ★ ★ ★ ★ ★

You rated this post out of 5. Change rating

2024-11-11 (first published: )

1,928 reads

Blogs

A New Word: Semaphorism

By

semaphorism – n. a conversational hint that you have something personal to say on...

SQL Server Linux Container Images Got Smaller: Image Size and Startup Time Across Cumulative Updates

By

This week I was at PASS Data Community Summit in Seattle, and I attended...

Using AI to Build Semantic Models, Ontologies, and Fabric Data Agents

By

(This is the third post in a three-part series on how AI is changing...

Read the latest Blogs

Forums

Pay for Performance

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Pay for Performance, which is...

Helping an English Student with R

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Helping an English Student with...

Adding Bits I

By Steve Jones - SSC Editor

Comments posted to this topic are about the item Adding Bits I

Visit the forum

Question of the Day

Adding Bits I

I have a table, dbo.bittest, where I am testing an application response. This has this structure:

CREATE TABLE [dbo].[BitTest]
(
[insertdate] [datetime] NULL CONSTRAINT [DF__BitTest__insertd__300F11AC] DEFAULT (getdate()),
[a] [bit] NULL
) ON [PRIMARY]
GO
My application runs this code:
INSERT dbo.BitTest
(
    insertdate,
    a
)
VALUES
(DEFAULT, 'true')
GO

See possible answers