Datasets vs Linked Services vs Pipelines in ADF

New users of Azure Data Factory are asked to create three kinds of object before anything moves: a linked service, a dataset and a pipeline. The names do not explain themselves, the creation screens look alike, and it is common to finish a first tutorial without being able to say why there are three. The distinction is simple once each is tied to a question.

Microsoft's one-line version is the one to memorise: the dataset represents the structure of the data within the linked data stores, and the linked service defines the connection to the data source (Microsoft Learn: datasets).

Linked service: the connection

A linked service is, in Microsoft's words, much like a connection string. It holds what ADF needs to reach an external resource: an address, credentials or an identity, and optionally which integration runtime to connect through.

There are two families (Microsoft Learn: linked services):

• Data store linked services: a storage account, a SQL database, an on-premises file share, a SaaS application.
• Compute linked services: somewhere an activity runs, such as an Azure Databricks workspace.

A linked service knows nothing about tables or files. One linked service to a database serves every table in it.

Dataset: the data

A dataset is a named reference to specific data inside a linked store. It has a type, such as DelimitedText or AzureSqlTable, a reference to its linked service, an optional schema, and type-specific properties: for a file, the container, folder, file name and format details; for a table, the schema and table name.

A dataset holds no data and no credentials. Delete it and nothing happens to the file it described. A linked service must exist before a dataset can be created, because the dataset has to point at one.

Pipeline: the work

A pipeline is a logical grouping of activities. An activity is where datasets are used: Microsoft describes an activity as taking zero or more input datasets and producing one or more output datasets (Microsoft Learn: pipelines and activities). A Copy activity names one dataset as its source and one as its sink.

A pipeline does not mention servers or passwords. It refers to datasets, and the datasets carry the rest.

All three, for one copy

The JSON below is written from the documentation and not deployed. It shows one of each, linked in a chain.

{
  "name": "LakeStorage",
  "properties": {
    "type": "AzureBlobStorage",
    "typeProperties": {
      "connectionString": "DefaultEndpointsProtocol=https;AccountName=examplelake;",
      "accountKey": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "CompanyKeyVault", "type": "LinkedServiceReference" },
        "secretName": "lake-storage-key"
      }
    }
  }
}

{
  "name": "ZonesCsv",
  "properties": {
    "type": "DelimitedText",
    "linkedServiceName": { "referenceName": "LakeStorage", "type": "LinkedServiceReference" },
    "typeProperties": {
      "location": {
        "type": "AzureBlobStorageLocation",
        "container": "raw",
        "folderPath": "reference/zones",
        "fileName": "taxi_zone_lookup.csv"
      },
      "columnDelimiter": ",",
      "firstRowAsHeader": true
    }
  }
}

{
  "name": "LoadZones",
  "properties": {
    "activities": [
      {
        "name": "CopyZones",
        "type": "Copy",
        "inputs": [ { "referenceName": "ZonesCsv", "type": "DatasetReference" } ],
        "outputs": [ { "referenceName": "ZonesTable", "type": "DatasetReference" } ],
        "typeProperties": {
          "source": { "type": "DelimitedTextSource" },
          "sink": { "type": "AzureSqlSink" }
        }
      }
    ]
  }
}

Read the references from the bottom up. The pipeline's activity names the dataset ZonesCsv. The dataset names the linked service LakeStorage. The linked service holds the connection, with the account key given as a reference to a secret in Azure Key Vault and not as a literal value. Nothing sensitive appears in the dataset or the pipeline, which is why those two can be stored in Git and promoted between environments without change.

The sink side has the same shape: a ZonesTable dataset of type AzureSqlTable, pointing at a database linked service.

The fourth object behind them

A linked service can name an integration runtime with connectVia. If it does not, the default Azure integration runtime is used. The integration runtime is the compute that carries out the work; Microsoft calls it the bridge between an activity and linked services (Microsoft Learn: introduction). For a source inside a private network, the linked service points at a self-hosted integration runtime installed in that network. Integration runtimes in ADF covers the choice.

So the full chain is: pipeline, activity, dataset, linked service, integration runtime, data store.

How many of each?

This is where designs go right or wrong.

Linked services: one per store per authentication method. One for the storage account, one for the database. Not one per table, and not one per pipeline.

Datasets: far fewer than you think. The beginner's instinct is a dataset for every table and file. A hundred tables then means a hundred datasets that differ by one string. Datasets can take parameters, so one generic SqlTable dataset with schema and table parameters, and one generic delimited-text dataset with folder and file parameters, can serve the whole factory.

Pipelines: one per unit of work that should succeed or fail together. A pipeline per source system or per subject area, with a parent pipeline calling them, is easier to rerun and monitor than one giant pipeline or hundreds of tiny ones.

Parameterising pipelines shows how the right-hand column is built.

Mistakes that come from confusing them

• Credentials in the wrong place. Secrets belong in the linked service, by reference to Key Vault, and nowhere else.
• A dataset per file. Then every new file is a deployment.
• Environment names baked into datasets, such as a development container name. Keep environment differences in linked services and parameters.
• Changing a shared dataset for one pipeline's convenience, breaking the others that use it.
• Expecting a dataset to validate data. Its schema is optional and descriptive. The Copy activity's mapping and the sink decide what happens to mismatched columns.

For the hands-on sequence of creating all three, see ADF pipelines explained; for connection options in depth, linked services in Azure Data Factory; and for the learning order, the Azure Data Factory roadmap.

Azure Data Factory and data pipeline design are covered in the Data Engineering course.

Sources

• Datasets in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Linked services in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Pipelines and activities in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• What is Azure Data Factory? (Microsoft Learn) — Microsoft Learn
• Delimited text format in Azure Data Factory (Microsoft Learn) — Microsoft Learn