Think LSNs Are Unique? Think Again - Preventing Data Loss in CDC ETL

When using only an LSN to track which changes you have processed, you can lose data. This article shows how to avoid this problem.

A gentle introduction to dbt

Over the past years, the tool dbt – short for data build tool – has become quite popular in the data engineering world for handling such an ELT process. dbt takes on the role of the “T”, meaning it’s responsible for transforming the data in a certain data store.

Time-Series Data Analysis with Apache Druid

Learn the basics of Apache Druid and how it can be used to ingest data into a data lake.

What I Wish Everyone Knew About ETL Processes - Traditional to Modern Cloud Solutions

Learn about the evolutionary journey of ETL (Extract, Transform, Load) from traditional processes to modern cloud solutions.

Unpivot Data Using Python

This article looks at manipulating data using Python to unpivot data received in a cross tab format.

Advantages of Kappa architecture in the Modern Data Stack

A comparison of the lambda and kappa architectures for real time ingestion and analysis of big data.

Python 3 Script for Generating SQL Insert Statements from CSV Data

This Python 3 script is designed to take CSV file data pasted into the csv_data variable and generate SQL insert statements that can be used to insert the data into a MySQL database. The script is easy to use and can save you a lot of time when working with large amounts of data.

The Art of Data Ingestion - Part 1

Data movement is a fundamental piece of a data engineer’s duties, and recently I’ve been thinking about the art of data movement. What are some of the most important pieces that a data engineer needs to think about when confronted with data ingestion? There is of course data exporting as well, and in that case, […]

Parquet

Chopping Off Data

An ETL issue with Excel causes issues, because the users didn't realize a limitation.