Azure Key Vault Integration with Data Factory

A managed identity is the right way for Azure Data Factory to authenticate, and it is not always available. An on-premises SQL Server wants a password. A partner's API wants a key. A service principal has a client secret. For everything that still needs a secret, Azure Key Vault is where it belongs, and this guide covers connecting the two properly.

The definitions below are written from Microsoft's documentation and have not been deployed.

What changes when a secret is in Key Vault

Without a vault, the secret is part of the linked service. Data Factory encrypts it, and it is still held by the factory, deployed with the factory, and set by whoever edits the linked service.

With a vault, the linked service holds only a pointer: the name of a secret. Data Factory retrieves the value when an activity that uses the connection runs (Microsoft Learn: store credentials in Key Vault). That gives you four things:

• The secret is managed in one place, by people who need not be pipeline developers.
• Rotation does not touch the factory. A new secret version is picked up on the next run.
• The factory's definition contains nothing sensitive, so it can live in a repository.
• Access to the secret is logged by the vault, separately from the factory.

Microsoft notes one exception: every activity type supports Key Vault references except the custom activity.

Setting it up

There are four steps.

1. Find the factory's managed identity. Its object ID is shown in the factory's properties, and in the dialog for creating a Key Vault linked service.

2. Grant it access to secrets. A vault uses one of two permission models:

Microsoft describes role-based access control as the recommended authorization system for Key Vault, and the access policy model as legacy; from API version 2026-02-01 it is also the default for new vaults (Microsoft Learn: Azure RBAC vs. access policies). The Secrets User role reads secret values and nothing else. Do not give the factory a role that can write or delete secrets.

3. Create a linked service for the vault.

{
  "name": "CorpKeyVault",
  "properties": {
    "type": "AzureKeyVault",
    "typeProperties": {
      "baseUrl": "https://contoso-adf-prod-kv.vault.azure.net"
    }
  }
}

It has no credential of its own. It works because the factory's identity has access.

4. Reference a secret from the data store's linked service.

{
  "name": "OnPremSalesDb",
  "properties": {
    "type": "SqlServer",
    "typeProperties": {
      "server": "sqlprod01.corp.example.com",
      "database": "Sales",
      "encrypt": "mandatory",
      "trustServerCertificate": false,
      "authenticationType": "SQL",
      "userName": "adf_reader",
      "password": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "CorpKeyVault", "type": "LinkedServiceReference" },
        "secretName": "sales-db-reader-password"
      }
    },
    "connectVia": { "referenceName": "CorpSelfHostedIR", "type": "IntegrationRuntimeReference" }
  }
}

The reference has three parts: its type, the vault linked service in store, and the secretName. For connectors that use a connection string, you can keep either the password alone or the whole connection string in the vault. Linked services in Azure Data Factory covers the object itself.

Versions and rotation

A fourth, optional property is secretVersion. Microsoft's documentation states the behaviour exactly: if it is not specified, the latest version of the secret is always used; if it is specified, the linked service sticks to that version.

Leave it out. Then rotation is two steps that do not involve Data Factory: change the password at the source system, and add a new version of the secret. The next activity run uses it. Pinning a version turns every rotation into a factory change.

One detail about expiry is easy to misread. A secret in Key Vault can carry an expiration date, and that attribute is informational: the documentation states that a get operation works for expired secrets (Microsoft Learn: about Key Vault secrets). An expiry date in the vault will not stop a pipeline. What stops it is the credential expiring at the system that issued it, such as a service principal's client secret reaching its end date. Track those dates, and treat the vault's attribute as a reminder, not a control.

Secrets inside a pipeline

Sometimes an activity needs a secret that is not part of a linked service: a token for an API called from a Web activity, for example. The documented pattern is a Web activity that reads the secret from the vault's REST interface, using the factory's identity (Microsoft Learn: Key Vault secrets in pipeline activities):

{
  "name": "GetApiKey",
  "type": "WebActivity",
  "policy": { "secureOutput": true },
  "typeProperties": {
    "method": "GET",
    "url": "https://contoso-adf-prod-kv.vault.azure.net/secrets/partner-api-key?api-version=7.5",
    "authentication": { "type": "MSI", "resource": "https://vault.azure.net" }
  }
}

Three points are essential:

• Secure output must be on. Without it the secret's value is written to the activity's output in the run history, in plain text, readable by anyone who can view the run.
• Secure input on whatever consumes it. The next activity reads the value as @activity('GetApiKey').output.value, and its input would otherwise be logged too.
• No version in the URL, so the latest is used. The api-version parameter is required.

Never assign the value to a pipeline variable or pass it as a pipeline parameter. Both are recorded with the run.

One vault per environment

Microsoft's CI/CD guidance gives a simple arrangement (Microsoft Learn: continuous integration and delivery):

• A separate key vault for each environment: development, test, production.
• The same secret names in every vault.
• Different permissions on each. Developers may read development secrets and have no access to production's.

The benefit shows at deployment. Every linked service refers to a secret by name, and the names are identical everywhere, so the only value that changes between environments is the vault's URL. That is one template parameter, not one per connection. Key Vault's own guidance points the same way, recommending a vault per application per environment with roles assigned at the vault (Microsoft Learn: Key Vault RBAC guide). CI/CD for Azure Data Factory covers the deployment.

There is a second reason. Data Factory does not store secrets in Git, so a linked service with a secret typed into it behaves differently from everything else: it is published to the live service immediately, outside the branch and pull request flow (Microsoft Learn: source control). Linked services that use Key Vault references have no such exception. They are ordinary files.

Network and the self-hosted runtime

• A vault behind a firewall can admit trusted Azure services, which lets the factory's identity through without opening the vault to public networks (Microsoft Learn: data access strategies).
• A self-hosted integration runtime needs an outbound route to the vault's address on port 443 when linked services it uses keep credentials there. A new firewall rule on that path is a common cause of a connection that fails only from the self-hosted runtime. See copying data from on-prem SQL.

Protecting the vault itself

Moving secrets into a vault concentrates them. The vault now deserves the care:

• Soft delete and purge protection. A deleted secret stays recoverable for a period of 7 to 90 days, 90 by default (Microsoft Learn: Key Vault soft-delete). An accidental deletion is then an inconvenience, not an outage.
• Few people with write access. The factory reads. A small group sets and rotates.
• Diagnostic logs on the vault, so every read of a secret is recorded. ADF logging and auditing covers the factory's side.

When it fails

Managed identity first

Key Vault protects a secret. A managed identity removes it. Where the data store supports Microsoft Entra authentication, Azure storage, Azure SQL, Synapse, use the identity and store nothing. Managed identities in Azure Data Factory covers that. Key Vault is for what is left.

Azure Data Factory and cloud data security are covered in the Data Engineering course.

Sources

• Store credentials in Azure Key Vault (Microsoft Learn) — Microsoft Learn
• Use Azure Key Vault secrets in pipeline activities (Microsoft Learn) — Microsoft Learn
• Azure role-based access control vs. access policies (Microsoft Learn) — Microsoft Learn
• Provide access to Key Vault keys, certificates, and secrets with Azure role-based access control (Microsoft Learn) — Microsoft Learn
• About Azure Key Vault secrets (Microsoft Learn) — Microsoft Learn
• Continuous integration and delivery in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Source control in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Data access strategies (Microsoft Learn) — Microsoft Learn
• Azure Key Vault soft-delete overview (Microsoft Learn) — Microsoft Learn