JSON Variable Substitution in Multi-Stage YAML Azure Pipeline

If you made it past that long and technical article title then you are here for a purpose. Maybe you’re just getting started with the relatively new YAML experience in Azure Pipelines.  Or maybe you’ve been doing this awhile and found the variables guidance to be less than clear. In any case, in this article I’m going to show how to reference pipeline variables in an Azure Pipelines CI/CD multi-stage YAML file. See also the followup post to this one in which variables are moved into an Azure Key Vault.

The Azure DevOps project is publicly visible here.  My code  sample is a .NET Core 3.1 API app called TimeApi. It has one controller named “Time” which has a single endpoint that returns the current Unix timestamp. You can exercise the API and examine the OpenApi (Swagger) spec here:

https://timeapi-sbx.azurewebsites.net/swagger/index.html

My YAML file is organized into three stages: Build, Development, and Sandbox. The build stage has all the usual tasks for restore, build, and publish as well as a copy files task for the ARM template that will create the resource group, plan, and Azure App Service. Here’s the basic structure of the YAML file with a build stage and two stages for deployment jobs:

# -----------------------------------------
# Basic Multi-Stage YAML Structure
# -----------------------------------------

  trigger:
    - master

  pool:
    vmImage: 'ubuntu-latest'

  stages:
    - stage: Build

      jobs:
      - job: Build

        steps:
      
          # TASKS GO HERE

    - stage: Development

      jobs:
        - deployment: Development
          displayName: Development
          environment: Development

          strategy:
            runOnce:
                deploy:
                    steps:

                      # TASKS GO HERE

                            
    - stage: Sandbox
      dependsOn: Development

      jobs:
         - deployment: Sandbox
           displayName: Sandbox
           environment: Sandbox

           strategy:
             runOnce:
                deploy:
                    steps:

                      # TASKS GO HERE

These three stages are visible in my portal’s pipelines view:

And if I click on the latest build I see the visualization of the three stages and their successful run to completion:

It’s very common to have different values and secrets for each stage. For example, the database connection string in my development environment is probably different from the one in the sandbox environment. And there might be other secrets that I don’t dare hard-code in my app. My appsettings.json file is open for the whole world to see. It looks like this:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*",
  "Settings": {
    "Uri":  "https://my-backendservice-dev.domain.com"
  },
  "ConnectionStrings": {
    "DefaultConnection": "Data Source=localhost.db"
  },
  "SecretOne": "configured in pipeline",
  "SecretTwo": "configured in pipeline"
}

In a production app I would insert the boilerplate “configured in pipeline” for all of these values and use Secret Manager (with a secrets.json) for development on my local machine. But I want to leave a couple in the clear so we can see JSON variable substitution at work.

In my pipeline I have set these variables:

The first two connection strings are encrypted at rest and can no longer be seen in the clear. The second two secrets I left in plaintext. Notice that the Settings Uri value is missing altogether from this list. That’s intentional. Let’s suppose it’s a value that I’m comfortable being hard-coded in my YAML file.

Now we need to reference these variables in the CI/CD pipeline. In order for JSON variable substitution to work the variable name in my YAML file must match exactly the variable name in my appsettings.json file. Nested elements must be dot-delimited. So in my appsettings.json file above SecretOne is a top-level standalone element and my YAML variable can just be SecretOne. But my connection string variable has to be named ConnectionStrings.DefaultConnection. I’m going to assume you’re comfortable with this format. See the documentation on this here for a full explanation.

I’m going to make SecretOne and SecretTwo global variables because they are the same for all stages. So I can add them to my YAML file at the top-level like so:

# -----------------------------------------
# TimeApi
# -----------------------------------------

  trigger:
    - master

  pool:
    vmImage: 'ubuntu-latest'

  variables:
    SecretOne: $(SecretOneVariable)
    SecretTwo: $(SecretTwoVariable)

  stages:
    - stage: Build

      jobs:
      - job: Build

My Settings.Uri and ConnectionStrings.DefaultConnection JSON values are stage-specific and thus will change between Development and Sandbox. So I want to reference them in the stages in which they are used:

# YAML Snippet Development

    - stage: Development

      jobs:
        - deployment: Development
          displayName: Development
          environment: Development

          variables:
            Settings.Uri: 'https://my-backendservice-dev.domain.com' 
            ConnectionStrings.DefaultConnection: $(ConnectionStringDevelopmentVariable)

          strategy:
            runOnce:
                deploy:
                    steps:

And in the Sandbox stage:

# YAML Snippet Sandbox

- stage: Sandbox
   dependsOn: Development

   jobs:
      - deployment: Sandbox
        displayName: Sandbox
        environment: Sandbox

        variables:
         Settings.Uri: 'https://my-backendservice-sbx.domain.com'
         ConnectionStrings.DefaultConnection: $(ConnectionStringSandboxVariable)

        strategy:
          runOnce:
             deploy:
                 steps:

I’ve hard-coded the Settings.Uri value and that makes it easy. The connection string is available to the pipeline at runtime and I just have to use the $() syntax to tell Azure DevOps to retrieve and decrypt it.

Finally, after the pipeline run I can inspect my appsettings.json file for both app services on the Azure portal.

Development

Sandbox

Looks good!