What I Wish Everyone Knew About ETL Processes - Traditional to Modern Cloud Solutions
Learn about the evolutionary journey of ETL (Extract, Transform, Load) from traditional processes to modern cloud solutions.
2023-10-18
ETL/SSIS/Azure Data Factory
GIT Configuration and Automated Release for Azure Data Factory
Learn a step-by-step method to save your ADF pipelines in Git and automatically release new changes to test and production.
2023-10-16
1,863 reads
Sending Multiple Attachments in Email from Azure Data Factory Pipeline
This article shows how you can have your Azure Data Factory (ADF) pipeline send an email using a Logic App.
2023-09-29
2,055 reads
How Effective Is Your Data Load Monitoring in SQL and Python?
Learn how you can use monitor your data warehouse load process with Python code and a structured process.
2023-10-06 (first published: 2023-09-20)
1,729 reads
Unpivot Data Using Python
This article looks at manipulating data using Python to unpivot data received in a cross tab format.
2023-06-30
2,751 reads
Advantages of Kappa architecture in the Modern Data Stack
A comparison of the lambda and kappa architectures for real time ingestion and analysis of big data.
2023-06-19
4,443 reads
Python 3 Script for Generating SQL Insert Statements from CSV Data
This Python 3 script is designed to take CSV file data pasted into the csv_data variable and generate SQL insert statements that can be used to insert the data into a MySQL database. The script is easy to use and can save you a lot of time when working with large amounts of data.
2023-05-08 (first published: 2023-05-05)
3,499 reads
How to Implement CI/CD in Azure Data Factory (ADF)
Learn how you can use CI/CD with your ADF Pipelines and Azure DevOps using ARM templates.
2023-03-10
10,535 reads
Ingesting Data from Google Drive to Azure Data Lake Gen2 using Azure Data Factory
Learn how to setup, configure and ingest data from Google Drive using Azure Data Factory in this step-by-step article.
2023-02-01
The Art of Data Ingestion - Part 1
Data movement is a fundamental piece of a data engineer’s duties, and recently I’ve been thinking about the art of data movement. What are some of the most important pieces that a data engineer needs to think about when confronted with data ingestion? There is of course data exporting as well, and in that case, […]
2022-11-11
5,651 reads
Blogs
Dynamically Unpivot columns in SQL
By DataOnWheels
Picture this, your data ingestion team has created a table that has the sales...
Friday Basics: RPO and RTO
By Brian Kelley
I did a post last month titled RTO and RPO are myths unless you've...
A New Word: ioia
By Steve Jones
ioia – n.the wish that you could see statistics overlaid on every person you...
Forums
SQL Server Encrypt data into a file, send it and then decrypt
First off, my apologies for what could potentially be a bad title! I am...
Table partitioning best practice
I've inherited a couple of rather large databases from my ex-colleague when I join...
Identifying Customer Buying Pattern in Power BI - Part 2
Comments posted to this topic are about the item Identifying Customer Buying Pattern in...
Question of the Day
Finding Marks
I have marked a few transactions in my code. How can I find out which marks were stored in a transaction log?
See possible answers