When you run a CI/CD pipeline, you often need confidential values like passwords, authentication tokens, service principal secrets etc. when you want to deploy a certain artefact. You don’t want to store those secrets directly in your pipelines as this might pose a considerable security leak. Instead, you either store them as secret variables, or as secrets in an Azure Key Vault. The advantage of using Azure Key Vault is that you have a centralized location for your secrets. If you need a secret somewhere else, for example in Azure Data Factory, you have to store it only once. Since service principal secrets have a limited lifetime, it’s easier if you need to update it in only one location.

The case used in this article to illustrate the concept of fetching Azure Key Vault secrets in an Azure Devops pipeline is a deployment pipeline for Power BI semantic models. When you want to deploy a Power BI semantic model using the Fabric APIs, you can use the FabricPS-PBIP PowerShell module. The authentication to the Fabric service (Power BI is now part of Microsoft Fabric) is done using a service principal.

There are two options to get the secret and object ID of this service principal:

• you run an Azure CLI task with PowerShell. In this task, you use a pre-defined service connection and then retrieve the service principal’s secret and object ID using task variables. The YAML code looks like this:

- task: AzureCLI@2
displayName: "Get key and id"
inputs:
azureSubscription: myserviceconnection"
scriptType: "ps"
scriptLocation: "inlineScript"
addSpnToEnvironment: true
inlineScript: |
Write-Host "##vso[task.setvariable variable=servicePrincipalId;issecret=true]$($env:servicePrincipalId)"
Write-Host "##vso[task.setvariable variable=servicePrincipalKey;issecret=true]$($env:servicePrincipalKey)"

• you get the service principal secret and object ID from Azure Key Vault, which is the focus of this article.

If the service connection in Azure Devops is defined with workload identity federation, there’s no service principal with a secret that you can use, which means we need the Azure Key Vault option to fetch the secret and object ID (also make sure you’re using the correct Object ID).

Accessing Azure Key Vault Secrets in an Azure Devops Pipeline

First, you need an Azure Key Vault obviously. If you don’t have one, you can create a new Key Vault in the Azure Portal. A Key Vault has a very low cost, just make sure to choose Standard pricing tier when you create one:

An Azure Key Vault has two options to give access to secrets:

• via access policies, which is considered legacy.
• via Azure Role-based Access Controls (RBAC), which is now the preferred method.

Once the vault is created, you can enter your secrets. In Azure Devops, you need a service connection to the resource group that holds the Key Vault. Behind every service connection is a service principal, which you can find by clicking on “manage app registration” in the connection details:

This service principal should be assigned the Key Vault Secrets User role in the Key Vault if you use RBAC:

We can now add a task to our Devops pipeline that fetches the secrets from the Key Vault with the following YAML:

- task: AzureKeyVault@2
displayName: Fetch Azure Key Vault Secrets
inputs:
azureSubscription: "myserviceconnection"
KeyVaultName: "mykeyvault"
SecretsFilter: 'sp-clientid,sp-secret'
RunAsPreJob: false

Here the secrets are named “sp-clientid” and “sp-secret”. After this task has run successfully, we can use the key vault secrets in a PowerShell task. For example, in the following script the secrets are used to fetch a Fabric authentication token using Set-FabricAuthToken:

$servicePrincipalId = "$(sp-clientid)"
$servicePrincipalKey = "$(sp-secret)"
Set-FabricAuthToken `
-servicePrincipalId $servicePrincipalId `
-servicePrincipalSecret $servicePrincipalKey `
-tenantId $tenant_id `
-reset

What if Azure Key Vault has Public Access Disabled?

As with many Azure services, you can configure the network access policy for Azure Key Vault. There are three options available:

• allow public access from all networks.
• disable public access. You can only access the Vault through a private endpoint connection.
• a hybrid option, where public access from only specify virtual networks or IP-addresses is allowed.

If the first option is enabled, you can skip this section as Azure Devops can reach your Key Vault anytime and you don’t need any special configuration.

When access is limited, there’s an option to allow trusted Microsoft services to bypass this firewall:

However, Azure Devops is not a trusted service (as you can run custom code and 3^rd party tools in this service), which means that selecting this option will not give your Azure Devops pipelines access to the Key Vault. If public access is completely disabled, an option is to use self-hosted agents on a virtual machine that is in a virtual network with a private endpoint to the Vault. The configuration of this set-up is out-of-scope for this article, but you can find a walkthrough in the documentation.

Another option is whitelisting the IP addresses of the Microsoft-hosted agents. The following screenshot shows the agent configuration of a classic pipeline using Microsoft-hosted agents:

Granting access to an entire IP range (which might change) doesn’t seem very robust. Alternatively, we can add a task in the pipeline that adds the IP address of the agent to the network settings of the Key Vault, fetch the needed secrets in the Key Vault as described in the previous section and then remove the IP address again.

To do this, we first need to fetch the current IP address (this will most likely change with every run of the pipeline), which we can do using curl in a bash task. With this command we fetch the IP address using the website ipinfo.io.

- task: Bash@3
displayName: Get IP Address
inputs:
targetType: 'inline'
script: |
ipaddr=$(curl -s http://ipinfo.io/ip)
echo $ipaddr
echo "##vso[task.setvariable variable=address;]$ipaddr"

The IP address is saved to a variable for later use (##vso stands for Visual Studio Online, a remnant of one of the previous brand names of Azure Devops). Using a PowerShell task, we can add a new network rule to the Key Vault, using the IP address of the bash task:

- task: AzureCLI@2
displayName: Add Azure Key Vault Network Rule
inputs:
azureSubscription: "myserviceconnection"
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# Add IP-address into Key Vault access restrictions
az keyvault network-rule add --name "mykeyvault" --ip-address "$(address)/32"

To make this work, the service principal behind the service connection needs contributor access to the Key Vault. Typically, this is already the case as the service connection is created with contributor rights on the resource group, and the Key Vault will inherit this permission.

Once the IP address is whitelisted, the Key Vault task of your pipeline will be able to access the Key Vault and fetch the secret. Similarly, the network rule can be removed once the secrets are fetched:

- task: AzureCLI@2
displayName: Remove Azure Key Vault Network Rule
inputs:
azureSubscription: "myserviceconnection"
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# Add IP-address into Key Vault access restrictions
az keyvault network-rule remove --name "mykeyvault" --ip-address "$(address)/32"

Conclusion

In this article, we’ve shown you how you can fetch secrets from an Azure Key Vault from within an Azure Devops Pipeline. If network access to the Key Vault is limited, a work-around is proposed where the IP address of the Devops agent is added to the firewall configuration to enable access from the Devops pipeline. All code samples are provided in YAML, but the same scripts (PowerShell, Azure CLI and Bash) can be used in a classic pipeline as well.