ADF Pipelines Explained: Building Your First Pipeline

In Azure Data Factory, the pipeline is the thing you run. Everything else, connections, datasets, triggers, exists to serve it. Microsoft defines a pipeline as a logical grouping of activities that together perform a task, deployed and scheduled as a unit instead of activity by activity (Microsoft Learn: pipelines and activities).

This guide explains what is inside a pipeline and walks through building the simplest useful one: copy a CSV file from blob storage into a SQL table. The steps follow Microsoft's documentation; they were not carried out in an Azure subscription for this article.

Pipelines and activities

An activity is one step. ADF groups activities into three kinds:

Activities are connected by dependencies: this one runs after that one succeeds, or fails, or completes either way. Activities with no dependency between them run in parallel. A pipeline has a default soft limit of 120 activities, a number worth knowing only because approaching it means the pipeline should be split.

What you need before the pipeline

A pipeline that copies data needs four supporting objects. Create them in this order:

1. A linked service for the source. The connection to the storage account.
2. A linked service for the sink. The connection to the database.
3. A source dataset. Which container, folder and file, and that it is delimited text with a header row.
4. A sink dataset. Which table.

Datasets vs linked services vs pipelines explains why these are separate. The short version: a linked service says where and how to connect; a dataset says what data, there.

Building it

1. Create the data factory in Azure Data Factory Studio. Microsoft notes the UI is supported in Microsoft Edge and Google Chrome.
2. Create the two linked services on the Manage tab. Test each connection.
3. Create the two datasets on the Author tab. Preview the source data to confirm ADF can read it.
4. Create a pipeline and drag a Copy data activity onto the canvas.
5. Set the Source tab to the CSV dataset and the Sink tab to the table dataset.
6. Check the Mapping tab. Import schemas to see which source column goes to which sink column, and fix any that do not line up.
7. Select Debug.

What the canvas produces

Every pipeline is a JSON document. The canvas is an editor for it, and the code view shows it. The pipeline above, written from the documentation and not deployed:

{
  "name": "CopyZonesToSql",
  "properties": {
    "description": "Load the zone lookup file into the staging table",
    "activities": [
      {
        "name": "CopyZonesCsvToSql",
        "type": "Copy",
        "inputs": [
          { "referenceName": "ZonesCsv", "type": "DatasetReference" }
        ],
        "outputs": [
          { "referenceName": "ZonesTable", "type": "DatasetReference" }
        ],
        "typeProperties": {
          "source": { "type": "DelimitedTextSource" },
          "sink": { "type": "AzureSqlSink" }
        },
        "policy": {
          "timeout": "00:30:00",
          "retry": 2,
          "retryIntervalInSeconds": 60
        }
      }
    ]
  }
}

Three things to notice.

The activity names datasets, not connections. inputs and outputs reference datasets; the datasets reference linked services. The Copy activity takes exactly one input and one output (Microsoft Learn: Copy activity).

Source and sink have types. DelimitedTextSource and AzureSqlSink tell the activity how to read and how to write. Each connector has its own options here.

The policy is where reliability starts. By default an activity's timeout is 12 hours and its retry count is 0. A copy that hangs will hang for half a day, and a transient network error fails the run. Setting a realistic timeout and a couple of retries, as above, is the first production habit to form.

Reading the JSON is worth the effort early. It is what gets stored in Git, what a reviewer compares between versions, and what deployment moves between environments.

Debug moves real data

Debug runs the pipeline from the canvas without publishing it. It is easy to assume this is a dry run. It is not. The documentation is explicit: selecting Debug actually runs the pipeline, and if it contains a copy activity, the test run copies data from source to destination (Microsoft Learn: iterative development and debugging).

Two consequences:

• Point debug runs at test tables and folders, as Microsoft recommends, until the pipeline is right.
• Running Debug twice loads the data twice if the sink simply inserts. That is your first encounter with the question every pipeline must answer: what happens on a rerun?

Publish and trigger

Changes on the canvas are drafts until you Publish. Publishing saves them to the service, and only published pipelines can be run by triggers. With Git integration enabled, you save to a branch and publish from the collaboration branch instead.

A published pipeline runs in one of two ways:

• Trigger now: a manual run of the published version.
• A trigger: a schedule, a tumbling window, or an event such as a file arriving. Pipeline dependencies and scheduling covers these.

Reading the result

After a run, select the activity's output. For a Copy activity it reports, among other things (Microsoft Learn: monitor copy activity):

Compare rowsRead with rowsCopied, and both with what you expected. A run marked Succeeded that copied zero rows is a common and silent failure, usually a wrong path or an empty source.

What a first pipeline leaves unsolved

The pipeline works, and it is not yet a good pipeline:

• It is hard-coded to one file and one table. Parameterising pipelines fixes that.
• It duplicates on rerun. Truncate-and-load, upsert, or loading only what changed are the remedies; see the Copy activity guide.
• It tells nobody when it fails.
• It runs only when you click.

Each of those is a stage in the Azure Data Factory roadmap. The first pipeline's job is to make the objects and their relationships concrete, so the rest has something to attach to.

Azure Data Factory and ETL pipeline design are covered in the Data Engineering course.

Sources

• Pipelines and activities in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Copy activity in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Iterative development and debugging with Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Monitor copy activity (Microsoft Learn) — Microsoft Learn
• Quickstart: create a data factory (Microsoft Learn) — Microsoft Learn