Linked Services in Azure Data Factory Explained

Every byte Azure Data Factory moves passes through a linked service. It is the object that knows how to reach a storage account, a database, an API or a compute service, and so it is where security, environments and a good deal of troubleshooting meet. Microsoft compares a linked service to a connection string: it defines the connection information the service needs to reach external resources (Microsoft Learn: linked services).

Creating one takes a minute. Creating them well, so that no secret is exposed and the same pipelines run unchanged in development and production, takes a little design.

Two kinds

• Data store linked services connect to places data lives: Azure Blob Storage, Azure Data Lake Storage, SQL databases, on-premises file systems, SaaS applications, REST endpoints.
• Compute linked services connect to places work runs: an Azure Databricks workspace, an Azure Function, a batch service.

There is also one that is neither: the Azure Key Vault linked service, which other linked services use to fetch their secrets.

Anatomy

A linked service definition has four parts:

The last is how private sources are reached. A linked service to an on-premises SQL Server sets connectVia to a self-hosted integration runtime installed inside that network; see integration runtimes in ADF.

Authentication: from weakest to best

Most connectors offer several ways to authenticate. Blob Storage, for example, supports account key, shared access signature, service principal and managed identity (Microsoft Learn: Blob Storage connector). They are not equivalent.

Managed identity is the one to prefer. Microsoft's description is that managed identities eliminate the need to manage credentials: the factory has an identity in Microsoft Entra, and you grant that identity a role on the target (Microsoft Learn: managed identity). There are two types:

• System-assigned: created with the factory and deleted with it. Only that factory can use it.
• User-assigned: a standalone resource that can be assigned to one or more factories.

For Blob Storage the grant is a role assignment: at least Storage Blob Data Reader to read, Storage Blob Data Contributor to write. Nothing secret exists to leak, rotate or expire.

{
  "name": "LakeStorage",
  "properties": {
    "type": "AzureBlobStorage",
    "typeProperties": {
      "serviceEndpoint": "https://examplelake.blob.core.windows.net/"
    }
  }
}

That is the whole definition: an endpoint and no credential. The factory's managed identity does the rest. This and the next example are written from the documentation and have not been deployed.

Secrets belong in Key Vault

Where a secret is unavoidable, a SQL login for a database that does not accept Entra authentication, a third-party API key, do not type it into the linked service. Reference it. A secret field can be given as an object of type AzureKeyVaultSecret, naming the secret and the Key Vault linked service; if no version is specified, the latest version is always used (Microsoft Learn: credentials in Key Vault).

Three benefits follow. The secret is rotated in one place and the factory picks up the new version. The factory's JSON, which sits in Git, contains no secret. And who can read the secret is controlled in Key Vault, separately from who can edit pipelines. The factory's managed identity is what authenticates to Key Vault. Azure Key Vault integration with Data Factory covers the setup.

One linked service, many databases

A server with twenty databases does not need twenty linked services. Linked services take parameters, and Microsoft gives exactly this example: parameterise the database name to avoid creating a linked service for each database on a logical SQL server (Microsoft Learn: parameterize linked services).

{
  "name": "SqlServerAnyDatabase",
  "properties": {
    "type": "AzureSqlDatabase",
    "parameters": {
      "databaseName": { "type": "String" }
    },
    "typeProperties": {
      "server": "example-sql.database.windows.net",
      "database": "@{linkedService().databaseName}",
      "authenticationType": "SQL",
      "userName": "etl_loader",
      "password": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "CompanyKeyVault", "type": "LinkedServiceReference" },
        "secretName": "sql-etl-loader-password"
      }
    }
  }
}

The expression @{linkedService().databaseName} is filled in at run time from the dataset or activity that uses the linked service.

Microsoft adds a caution worth repeating: do not parameterise passwords or secrets. Store them in Key Vault and, if they must vary, parameterise the secret name.

Development, test and production

The same pipeline must point at different servers in each environment. There are two sound ways to arrange that, and one unsound.

• Sound: keep names identical and vary the values at deployment. A linked service called LakeStorage exists in every environment's factory. Its endpoint differs. The deployment process substitutes the value per environment.
• Sound: reference Key Vault, and have a vault per environment. The linked service always asks for the secret sql-connection; each environment's vault returns its own.
• Unsound: linked services named for environments, such as LakeStorage_Dev and LakeStorage_Prod, with datasets that must be repointed. Every promotion then edits datasets, and sooner or later a development pipeline writes to production.

CI/CD for Azure Data Factory shows how values are overridden at deployment.

Testing and troubleshooting

The Test connection button checks the connection from the integration runtime the linked service uses, which is the point: a connection that works from your laptop proves nothing about the runtime.

When a connection fails, the cause is nearly always one of four:

1. Network. The runtime cannot reach the store: a firewall rule, a private endpoint, or the wrong integration runtime.
2. Identity. The factory's managed identity or the service principal has no role on the target.
3. Secret. The Key Vault reference points at a missing secret, or the factory's identity cannot read the vault.
4. Wrong runtime. A cloud linked service accidentally set to a self-hosted runtime, or the reverse.

Good practice in brief

• One linked service per store and authentication method, shared by all datasets.
• Managed identity first; Key Vault for whatever is left.
• No secrets in definitions, ever.
• Least privilege: reader where the factory only reads, and separate identities for separate trust levels where needed.
• Name for the system, not the environment or the project.
• Parameterise instead of multiplying.

For how linked services relate to datasets and pipelines, see datasets vs linked services vs pipelines; for identity in depth, securing Azure Data Factory with managed identities.

Azure Data Factory and Azure data services are covered in the Data Engineering course.

Sources

• Linked services in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Store credentials in Azure Key Vault (Microsoft Learn) — Microsoft Learn
• Managed identity for Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Parameterize linked services in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Copy and transform data in Azure Blob Storage (Microsoft Learn) — Microsoft Learn