Using Fivetran (HVR) to Replicate Data From PostgreSQL to an Azure Databricks Warehouse: Data Engineering with Azure

Learn how to use Fivetran HVR software to ETL data into your One Lake Storage from a PostgreSQL database.

Archive to the Lake

Microsoft Fabric changes the paradigm for how you might store data in your warehouse.

Database Deployment with Terraform - Modules

Deploy resources in Azure using reusable code with Terraform modules.

Azure Data Lake Storage Gen 2

Implementing ETL Logging on Lakehouse using Delta Lake's Time Travel capability

In this article I demonstrate how to use Delta Lake's time travel functionality for ETL logging along with steps to take to implement.

Dynamically Add a Timestamp To Files in Azure Data Factory

This article will describe how to add your local timestamp at the end of the each file in Azure Data Factory (ADF). In general, ADF gets a UTC timestamp, so we need to convert the timestamp from UTC to EST, since our local time zone is EST. For example, if the input Source file name […]

Processing Data Using Azure Data Lake, Azure Data Analytics, and Microsoft’s Big Data Language U-SQL

Data analytics has become one of the powerful domains in the world of data science. An enormous amount of data is being generated by each organization in every sector. Computer science has found solutions to store and process this data in a smart way through a distributed file system. One such example is Azure Data Lake. It uses the Hadoop Distributed File System, and to perform analytics on this data, Azure Data Lake storage is integrated with Azure Data Analytics Service and HDInsight. In this article, Suhas Pande will explain how to store data using Azure Data Lake and how to perform data analysis on it using U-SQL, a big data SQL and C# language.

Processing Data Using Azure Data Lake, Azure Data Analytics, and Microsoft’s Big Data Language U-SQL

Data analytics has become one of the powerful domains in the world of data science. An enormous amount of data is being generated by each organization in every sector....

Guide to Quickly Creating an Azure Data Lake Storage

Read a step by step guide to setting up an Azure Data Lake Storage account.

Azure Data Lake Analytics using U-SQL Queries

This article will help with gaining confidence and familiarity with Microsoft Azure's Data Lake Analytics offering to process large datasets quickly while demonstrating the potential and capabilities of U-SQL to aggregate and process big data files.