Managing Files and Folders with Python – Data Engineering with Fabric
In this article, learn how you can manage files and folders for both full and incremental loading situations.
2024-03-27
1,427 reads
In this article, learn how you can manage files and folders for both full and incremental loading situations.
2024-03-27
1,427 reads
Learn how to get started with Microsoft Fabric along with the differences between managed and unmanaged tables.
2024-03-20
2,111 reads
Generative Al tools like Gemini and GPT promise to automate and augment knowledge-based work. Data professionals must adapt to this transformation by acquiring new skills and playing a central role in their organization's AI-driven future. Data preparation, curation, ethical sourcing and labeling, and collecting user feedback become crucial as high-quality data is essential for effective LLM based application.
2024-03-04
2,267 reads
Get ready to be blown away! The highly anticipated Microsoft Build in May 2023 has finally unveiled its latest and greatest creation: the incredible Microsoft Fabric - an unparalleled Data Intelligence platform that is guaranteed to revolutionize the tech world! fig 1: OneLake for all Data One of the most exciting things in Fabric I […]
2023-07-26
4,365 reads
This article examines how one can structure a pipeline for processing real-time data using Kafka and Informatica.
2023-04-26
4,228 reads
Whether you work as a Data Engineer or a Data Scientist, a Jupyter Notebook is a helpful tool. One of the projects I was working required a comparison of two parquet files. This is mainly a schema comparison, not a data comparison. Though the two .parquet were created from two different sources, the outcome should […]
2021-05-17
4,935 reads
This article will describe how to add your local timestamp at the end of the each file in Azure Data Factory (ADF). In general, ADF gets a UTC timestamp, so we need to convert the timestamp from UTC to EST, since our local time zone is EST. For example, if the input Source file name […]
2021-04-22
26,972 reads
Introduction I recently passed the Google Cloud Professional Data Engineer certification exam, Professional Data Engineer Certification. It took me about five month to prepare for this, and I would like to share my thoughts of why I decided to take it on and how I prepared for it. At the moment, Google cloud (GCP) is […]
2021-03-15
9,547 reads
By James Serra
(Shameless plug: The price of my book “Deciphering Data Architectures: Choosing Between a Modern...
By Steve Jones
I was working with a customer and discussing how to do error handling. This...
By DataOnWheels
The 14th annual Ability Summit is a global event that I attended a few...
Hello, I was given this SQL question and was hoping someone could please help...
In recent days past, there was a Performance gain by placing data files, log...
Ok, I've come up with another problem in my project. I am happily pulling...
How does the Resource database in SQL Server 2022 get backed up?
See possible answers