Secret Environment Variables in a YAML pipeline

Environment variables and pre-deployment approvals can’t be created in a YAML pipeline, but obviously they still can be used in Azure DevOps.

Environment variables can be defined in the environment specific parameter files, but what if you want to use secret values? To use secret values, you can use variable groups. I created the variable groups int-shared-dev, int-shared-tst, int-shared-acc and int-shared-prod in DevOps via Pipelines/Library. In these variable groups, there’s only one variable named SBConnectionString. This variable can be marked as a secret value using the Lock icon. As an alternative, we could also store the SBConnection string in KeyVault and reference the KeyVault secret in the variable group. More complicated and not necessary.

Next we need to link the variable groups to the environment. This is done in the root pipeline named azure-pipeline.yml where you define the different build and release stages. As an example:

– stage: Dev
– group: int-shared-dev
dependsOn: Build
condition: succeeded()

The next adjustment is made in files cd-tasks.yml and ci-tasks.yml. Here we typically define an AzureResourceGroupDeployment task. At the end of the task we specify cmsParametersFile and overrideParameters.

csmParametersFile: ${{ format(‘{0}/{1}/{2}’, ‘$(Pipeline.Workspace)’, parameters.artifactName, parameters.parametersFile ) }}
overrideParameters: ‘
-storageAccountCICD {
“url”: “$(storageAccountURI)${{parameters.BlobPrefix}}”,
“sasToken”: “$(storageAccountSASToken)”
-SBConnectionString “$(SBConnectionString)”

StorageAccountCICD is an output parameter from the previous AzureFileCopy task. Next we see SBConnectionString being overwritten by the variable from the variable group, in this case int-shared-dev. We can actually remove SBConnectionString from the parameter files. It will be added to the parameters file when not specified. Note that the overridden parameters are not separated by a comma. Also note that storageAccoutCICD is an object and SBConnectionString a string.

Now, when we use SBConnectionString in an orchestrator or an individual ARM template, the value will be replaced with the environment variable in the specific variable group. Parameter SBConnectionString doesn’t have to be defined in the parameter files, but it should be defined in the template files.

An alternative to using variable groups (either or not referencing KeyVault), you can add a so-called KeyVault task to cd-tasks.yml:

– task: AzureKeyVault@1
azureSubscription: [serviceconnection]keyVaultName: kv-…
secretsFilter: ‘*’
runAsPreJob: false # Azure DevOps Services only

If you set the secretsFilter to *, all secrets will be retrieved. You can specify a comma-separated list of secrets to only retrieve specific secrets. For example, if there is a secret named connectionString, a task variable connectionString is created with the latest value of the respective secret fetched from Azure key vault. This variable is then available in subsequent tasks.

When it comes to choosing between a Variable Group and an Azure Key Vault Task, I found a valuable take away in the following post.  The main takeaway is:

  • Use Variable Groups for configuration re-used across multiple pipelines
  • Use the Azure Key Vault Task for single-purpose access to a vault from a specific pipeline

But then again. If you use an Azure Key Vault task in cd-tasks.yml, how can you specify a different KeyVault per environment. In my view, you could then define a variable KeyVault in your variable group and use that variable in your Azure Key Vault task. I tried it out, but unfortunately that doesn’t work. For the record, here is the specification that does not work.

– task: AzureKeyVault@1
  azureSubscription: “$(ServiceConnection)”
  keyVaultName: “$(KeyVault)”
  secretsFilter: ‘*’
  runAsPreJob: false # Azure DevOps Services only