Securing Azure Data Factory: Managed Identities

Every connection in Azure Data Factory has to prove who it is. The traditional proof is a secret: an account key, a password, a connection string. Secrets get copied into notebooks, pasted into chat, left in old branches and forgotten until they expire on a Sunday. Managed identities replace the secret with something the factory is, not something it knows.

The definitions below are written from Microsoft's documentation and have not been deployed.

What a managed identity is

A managed identity is an identity in Microsoft Entra ID that belongs to an Azure resource, here the data factory. When a pipeline needs to reach a data store, the service uses that identity to obtain a token, and the data store checks what the identity is allowed to do. Nothing secret is stored in the factory. In Microsoft's words, managed identities eliminate the need to manage credentials (Microsoft Learn: managed identity for Data Factory).

There are two kinds:

A factory created in the portal or with PowerShell gets a system-assigned identity automatically. It cannot be modified, and it disappears with the factory.

Granting access

The identity does nothing until a data store gives it a role. This is where security is decided, and where least privilege is applied.

Storage. Microsoft's connector documentation gives the minimums: Storage Blob Data Reader as a source, Storage Blob Data Contributor as a sink (Microsoft Learn: Data Lake Storage Gen2 connector). For finer control, access control lists can grant Execute on the folders above the data and Read or Write on the data itself. Assign the role at the container, not the whole account, where that is enough.

Azure SQL Database. Three steps, from the connector documentation (Microsoft Learn: Azure SQL Database connector): the server needs a Microsoft Entra administrator; a contained user is created in the database for the identity; and that user is given roles like any other.

CREATE USER [contoso-adf-prod] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER [contoso-adf-prod];

The name in brackets is the name of the data factory. Grant the narrowest role that works. A pipeline that reads does not need to write.

Key Vault. Secrets that cannot be avoided live in a vault, and the factory's identity is what reads them (Microsoft Learn: store credentials in Key Vault). Azure Key Vault with Data Factory covers that side.

The linked services

With identity-based authentication, a linked service shrinks to an address:

{
  "name": "LakeProd",
  "properties": {
    "type": "AzureBlobFS",
    "typeProperties": {
      "url": "https://contosolakeprod.dfs.core.windows.net"
    }
  }
}

No key, no secret reference. For Azure SQL Database the authentication type is named explicitly:

{
  "name": "SalesDbProd",
  "properties": {
    "type": "AzureSqlDatabase",
    "typeProperties": {
      "server": "contoso-sql-prod.database.windows.net",
      "database": "Sales",
      "encrypt": "mandatory",
      "trustServerCertificate": false,
      "authenticationType": "SystemAssignedManagedIdentity"
    }
  }
}

Both definitions can be committed to a repository and read by anyone, because there is nothing in them to steal. Linked services in Azure Data Factory covers the object in general.

User-assigned identities

A user-assigned identity is created on its own, associated with the factory, and then wrapped in a credential object that linked services select (Microsoft Learn: credentials). Configuring one requires the Managed Identity Operator role on the identity.

When it is the better choice:

• Access can be prepared in advance. The identity exists before the factory does, so role assignments can be made, reviewed and approved first.
• Rebuilding a factory does not lose access. A system-assigned identity dies with its factory, and every role assignment has to be recreated for the new one.
• Several identities per factory. One for finance data, one for marketing data, each with its own permissions, instead of one identity that can reach everything.

Its documented limitation: firewall bypass for trusted services works only with the system-assigned identity.

The warning to take seriously

Microsoft's documentation carries a note that deserves more attention than its formatting suggests. Users with the Data Factory Contributor role can create and run pipelines that use the system-assigned identity and any user-assigned identities attached to the factory, and those identities carry all permissions already granted to them on external resources.

Read that from an attacker's side. The identity can read the finance lake. A developer cannot. But the developer can author a pipeline in that factory, and the pipeline runs as the identity. Whoever can edit pipelines can copy anything the identity can reach to anywhere the identity can write.

The identity is therefore not a security boundary between people who share a factory. The practical consequences:

• Give the identity only what its pipelines need. Not Contributor on a subscription. Not account-wide storage roles for one folder's worth of work.
• Be careful who has the Contributor roles on the factory. Microsoft's advice is to assign them only to trusted principals and at the narrowest scope.
• Separate factories for separate trust levels. If two teams must not see each other's data, they should not share a factory whose identity can see both.
• In production, nobody authors. Changes arrive by deployment pipeline, which removes the easiest route.
• Review role assignments on the identity regularly, and watch the activity log. ADF logging and auditing covers the records.

Identity and the network

An identity answers "who is calling". Firewalls answer "from where". The two combine in two documented ways (Microsoft Learn: data access strategies):

• Trusted services. Azure Storage and Azure Key Vault can keep their firewalls closed to public networks and still admit trusted Azure services. This path enforces managed identity authentication, so only a factory whose identity has been granted access gets in. It does not apply to self-hosted or SSIS integration runtimes.
• Private endpoints. An Azure integration runtime in a managed virtual network connects to data stores through private endpoints, and traffic stays on Microsoft's network.

Integration runtimes in ADF explains the runtime types; for sources inside your own network, see copying data from on-prem SQL.

Where a managed identity cannot be used

• Systems outside Microsoft Entra ID. An on-premises SQL Server with SQL logins, a third-party API with a key, another cloud's storage. These need a secret, and the secret belongs in Key Vault.
• Connectors that do not offer it. Each connector's documentation lists its authentication types. Microsoft names Blob storage, Data Lake Storage, Azure SQL Database, SQL Managed Instance, Azure Synapse Analytics, REST and the Web activity among those that support it.

A short checklist

1. Managed identity on every linked service that supports it.
2. Roles granted at the smallest scope: container, database, vault.
3. Read roles for sources, write roles only for sinks.
4. Key Vault for every remaining secret, read by the identity.
5. Contributor roles on the factory limited to people trusted with everything its identity can reach.
6. Separate factories, or separate user-assigned identities, for separate data domains.
7. Role assignments reviewed on a schedule.

Azure Data Factory and data governance are covered in the Data Engineering course.

Sources

• Managed identity for Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Using credentials (Microsoft Learn) — Microsoft Learn
• Copy and transform data in Azure Data Lake Storage Gen2 (Microsoft Learn) — Microsoft Learn
• Copy and transform data in Azure SQL Database (Microsoft Learn) — Microsoft Learn
• Store credentials in Azure Key Vault (Microsoft Learn) — Microsoft Learn
• Data access strategies (Microsoft Learn) — Microsoft Learn