How to Implement CI/CD in Azure Data Factory (ADF)
Learn how you can use CI/CD with your ADF Pipelines and Azure DevOps using ARM templates.
2023-03-10
4,499 reads
diponkar.paul
- Interests: Data, Travel, Swimming, Cycling, cross-country skate
- Blog: http://allaboutdata.ca/
- Jobs: Senior Data Engineer
- Skills: Big data, Business Intelligence, Database, ETL, ELT, Data Warehouse
- Twitter: @@paulswengrr
- Website: http://allaboutdata.ca/
- LinkedIn: https://www.linkedin.com/in/diponkarpaul/
How do you secure sensitive data in a modern Data Warehouse?
In 2019 Canadian Broadcasting Corporation (CBC) news reported a massive data breach at the Desjardins Group, which is a Canadian financial service cooperative and the largest federation of credit unions in North America. The report indicated, a "malicious" employee copied sensitive personal information collected by Desjardins from their data warehouse. The data breach compromised the […]
2022-03-30
2,598 reads
How to Integrate Azure DevOps with Azure Synapse Studio
Learn how to connect your Synapse workspace to a Git repository.
2022-01-26
6,081 reads
How to Recover a Corrupted Azure Data Factory Integration Runtime
I would like to share my recent experience with Azure Data Factory (ADF) where AutoResolveIntegrationRuntime become corrupted and how did I recover it. I still don't know how the Integration Runtime (IR) was corrupted. However, if it happens, then this article will help you to solve the issue. Problem In general, the ADF AutoResolveIntegrationRuntime should […]
2021-11-05
1,860 reads
How to Flatten JSON in Azure Data Factory?
When you work with ETL and the source file is JSON, many documents may get nested attributes in the JSON file. Your requirements will often dictate that you flatten those nested attributes. There are many ways you can flatten the JSON hierarchy, however; I am going to share my experiences with Azure Data Factory (ADF) […]
2021-09-17
13,254 reads
Why Power Query as an Activity in Azure Data Factory and SSIS?
There is a Power Query activity in SSIS and Azure Data Factory, which can be more useful than other tasks in some situations.
2021-07-26
3,770 reads
Handling Row-level Errors in ADF Data Flows
If you are working with ADF (Azure Data Factory) data flows, then you may have noticed there was a new feature released in November 2020, which is useful to capture any error while inserting/updating the records in a SQL database. This article will describe how to setup the error row handling feature and why it's […]
2021-06-16 (first published: 2021-05-02)
3,680 reads
Step by Step Guide to Install Jupyter Notebook
Whether you work as a Data Engineer or a Data Scientist, a Jupyter Notebook is a helpful tool. One of the projects I was working required a comparison of two parquet files. This is mainly a schema comparison, not a data comparison. Though the two .parquet were created from two different sources, the outcome should […]
2021-05-17
4,168 reads
Dynamically Add a Timestamp To Files in Azure Data Factory
This article will describe how to add your local timestamp at the end of the each file in Azure Data Factory (ADF). In general, ADF gets a UTC timestamp, so we need to convert the timestamp from UTC to EST, since our local time zone is EST. For example, if the input Source file name […]
2021-04-22
20,370 reads
Reading Stored Procedure Output Parameters in Azure Data Factory
This article will help you to work with Store Procedure with output parameters in Azure data factory.
2021-04-06
21,759 reads
Blogs
Why i use a smaller docker image part I
What is Docker? Docker is a famous platform for building, shipping, and running container...
New Job -> Week 2 -> Creating a Checklist
This is part of a series of new job blog posts. You can find...
SQL Server 2022 Administration Inside Out
By Will Assaf
SQL Server 2022 Administration Inside Out is now available for pre-order from Microsoft Press. This...
Forums
Managed Instance Link and SQL Server 2022
Comments posted to this topic are about the item Managed Instance Link and SQL...
Passing parameters to query without TVFs
Comments posted to this topic are about the item Passing parameters to query without...
Capturing Data Changes
Comments posted to this topic are about the item Capturing Data Changes
Ask SSC