CI/CD for Azure Data Factory with Azure DevOps

A data factory is authored in a browser, which makes it tempting to treat production the same way: open it, change it, publish. That works until two people do it in the same week. CI/CD for Azure Data Factory replaces the browser with a path every change must follow, from a branch through review to a release that is the only thing allowed to touch production.

The build definition below is adapted from Microsoft's sample and has not been run in a DevOps organisation here.

How deployment works

Two facts explain the whole process (Microsoft Learn: continuous integration and delivery).

A factory is JSON. Every pipeline, dataset, linked service and trigger is a JSON document. With Git configured, those documents are files in a repository.

Deployment is an ARM template. To move a factory to another environment, Data Factory generates an Azure Resource Manager template describing all its entities, and the template is deployed to the target factory with different parameter values.

So CI/CD here means: get the right JSON into the main branch, turn it into a template, and deploy the template with the right parameters.

The environments

Microsoft is explicit that only the development factory is associated with a repository, and that test and production should be updated only through a deployment pipeline or a template.

Branches and review

On the development factory:

1. A developer creates a feature branch and works there. Saving commits to the branch. Debug runs use the branch's version.
2. A pull request merges the branch into the collaboration branch, usually main.
3. The merged main branch is what gets built and released.

Microsoft's recommendations for permissions make this meaningful (Microsoft Learn: source control): do not allow direct check-ins to the collaboration branch, and let only a select set of people publish.

The build: two ways to get a template

The original flow. After a merge, someone opens the development factory and selects Publish. Data Factory validates everything, publishes to the live development factory, and writes the template to a branch named adf_publish. A release picks it up from there. The weakness is the manual step.

Automated publishing. Microsoft packaged the same two actions, validate all and export ARM template, as an npm package, so a build pipeline can run them on every merge (Microsoft Learn: automated publishing). No button, and the template becomes a build artifact. Both flows are supported.

The automated flow needs a package.json in the repository:

{
  "scripts": {
    "build": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index"
  },
  "dependencies": {
    "@microsoft/azure-data-factory-utilities": "^1.0.0"
  }
}

And a build definition, here trimmed from Microsoft's sample:

trigger:
- main

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: UseNode@1
  inputs:
    version: '20.x'
  displayName: 'Install Node.js'

- task: Npm@1
  inputs:
    command: 'install'
    workingDir: '$(Build.Repository.LocalPath)/build'
  displayName: 'Install npm package'

- task: Npm@1
  inputs:
    command: 'custom'
    workingDir: '$(Build.Repository.LocalPath)/build'
    customCommand: 'run build export $(Build.Repository.LocalPath) /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/data-dev-rg/providers/Microsoft.DataFactory/factories/contoso-adf-dev "ArmTemplate"'
  displayName: 'Validate and generate ARM template'

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.Repository.LocalPath)/build/ArmTemplate'
    artifact: 'ArmTemplates'
    publishLocation: 'pipeline'

The export command validates before it generates, so a pull request that breaks a pipeline fails the build instead of reaching a factory. Microsoft notes that Node.js 20.x is needed, and that the exported template is not published anywhere by this step; deployment is the release's job.

The release

A release takes the template and deploys it to each environment in turn (Microsoft Learn: Azure Pipelines releases). Each stage has three parts.

1. Stop the triggers. Deployment can fail if it tries to update an active trigger. Microsoft supplies a PowerShell script for before and after the deployment, and the automated flow's artifact already includes it (Microsoft Learn: pre- and post-deployment scripts). A second version of the script stops only the triggers that changed.

2. Deploy the template with the ARM template deployment task, overriding the template's parameters with this environment's values.

3. Restart the triggers and clean up. The same script, run after deployment, restarts triggers and deletes resources that were removed from the repository. Without that, a pipeline deleted in development lives on in production.

One setting deserves a warning of its own. The deployment mode must be Incremental. In Complete mode, Resource Manager deletes everything in the resource group that is not in the template, which may include your storage account.

Between test and production, add an approval. The release history then records who approved what, which is part of the audit trail described in ADF logging and auditing.

Values that differ between environments

The template is the same everywhere. The differences are parameters.

• What is parameterized by default. Secure strings, such as Key Vault secret references, connection strings, keys and tokens, are template parameters automatically.
• What you add. A file named arm-template-parameters-definition.json, in the root of the branch, tells Data Factory which other properties to expose: a storage URL, a trigger's schedule (Microsoft Learn: custom parameters).
• The ceiling. A template can have at most 256 parameters. A large factory with default parameterization can exceed it; the same file is how you reduce the count.

Three design choices keep the parameter list short:

1. A key vault per environment with identical secret names. Only the vault's URL changes. See Azure Key Vault with Data Factory.
2. Managed identities, which have no secret to parameterize. See managed identities.
3. Global parameters for values used across many pipelines, such as an environment name. See global parameters and variables.

Integration runtimes need attention too. Microsoft's guidance is that a runtime must have the same name and type in every environment.

Large factories

A Resource Manager template has limits on the number of resources. For factories that exceed them, Data Factory also generates linked templates, a master template with child templates, in the same output (Microsoft Learn: linked templates). Deploying them requires uploading the files to a storage account that Resource Manager can read during deployment.

Hotfixes

Production has a bug. Main already contains half-finished work that cannot be released. Microsoft's documented procedure (Microsoft Learn: hotfix):

1. Find the commit that was last deployed to production.
2. Create a hotfix branch from that commit.
3. Fix and test in the development factory, on that branch.
4. Export the template from the hotfix branch and release it to test, then production.
5. Merge the fix into the development branch, so the next release does not bring the bug back.

Step 5 is the one that gets forgotten.

What it will not do

Worth knowing before designing a process around it:

• No selective publishing. A deployment contains everything in the collaboration branch. You cannot release one pipeline and hold back another. For work that must merge and not yet run in production, Microsoft suggests feature flags built from global parameters and an If Condition.
• No publishing from private branches.
• No Bitbucket. Azure Repos and GitHub are the supported hosts.
• Linked services with inline secrets bypass Git. They are published immediately. This is another reason to use Key Vault references.
• Changes by PowerShell or SDK are not in Git. They go straight to the live factory.

When a release fails

A checklist

• Git on the development factory only; no direct commits to main.
• A build that validates and exports on every merge.
• A release with stop triggers, incremental deploy, start triggers and clean-up, per stage.
• An approval before production.
• Key vault per environment, same secret names; managed identities where possible.
• A written hotfix procedure, practised once before it is needed.
• Nobody with rights to publish to production by hand.

For the wider set of production practices, see ADF pipeline best practices.

Azure Data Factory, with Docker and CI/CD for data pipelines, is covered in the Data Engineering course.

Sources

• Continuous integration and delivery in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Automated publishing for continuous integration and delivery (Microsoft Learn) — Microsoft Learn
• Automate continuous integration using Azure Pipelines releases (Microsoft Learn) — Microsoft Learn
• Custom parameters in a Resource Manager template (Microsoft Learn) — Microsoft Learn
• Continuous integration and delivery pre- and post-deployment scripts (Microsoft Learn) — Microsoft Learn
• Using a hotfix production environment (Microsoft Learn) — Microsoft Learn
• Using linked Resource Manager templates (Microsoft Learn) — Microsoft Learn
• Source control in Azure Data Factory (Microsoft Learn) — Microsoft Learn