Pipeline Dependencies and Scheduling in ADF

"What runs when?" has three answers in Azure Data Factory, because ordering is decided at three levels. Inside a pipeline, dependencies between activities set the order of steps. Between pipelines, one pipeline calls another. And at the top, a trigger decides when a pipeline starts at all. Confusion usually comes from trying to solve a problem at the wrong level.

The JSON below is written from Microsoft's documentation and has not been deployed.

Level 1: dependencies between activities

Activities with no dependency between them start together. To make one wait for another, you draw an arrow, which in JSON is a dependsOn entry with one or more dependency conditions. There are four (Microsoft Learn: pipelines and activities):

[
  {
    "name": "LoadStaging",
    "type": "Copy"
  },
  {
    "name": "MergeToTarget",
    "type": "SqlServerStoredProcedure",
    "dependsOn": [ { "activity": "LoadStaging", "dependencyConditions": [ "Succeeded" ] } ]
  },
  {
    "name": "LogFailure",
    "type": "SqlServerStoredProcedure",
    "dependsOn": [ { "activity": "LoadStaging", "dependencyConditions": [ "Failed" ] } ]
  },
  {
    "name": "DropTempFiles",
    "type": "Delete",
    "dependsOn": [ { "activity": "MergeToTarget", "dependencyConditions": [ "Completed" ] } ]
  }
]

Two rules catch people out.

Several arrows into one activity mean AND. If C depends on both A and B succeeding, C runs only when both have. There is no built-in OR between different upstream activities.

Skipped propagates. If A fails, an activity that depends on A succeeding is skipped, and so is everything that depends on that one succeeding. A clean-up step at the end of a chain will not run after an early failure unless it is attached with a condition that allows for it. Handling errors and retries covers how these conditions also decide whether the pipeline as a whole is marked failed.

Level 2: dependencies between pipelines

Large workloads are split into pipelines, and a parent runs them in order with the Execute Pipeline activity. Its key setting is waitOnCompletion, true by default: the activity waits for the child to finish and takes on its result (Microsoft Learn: Execute Pipeline activity).

• Wait, the default, when later steps depend on the child's work. The parent's arrows then express "load dimensions, then load facts".
• Do not wait to launch independent work and move on. The parent cannot then react to the child's failure.

The parent-and-children shape, often called a master or orchestrator pipeline, is the standard way to express "these ten loads in this order". It keeps each pipeline small, lets a child be rerun alone, and gives one run to monitor.

Level 3: triggers

A pipeline starts either manually or from a trigger. There are three trigger types (Microsoft Learn: pipeline execution and triggers):

The difference in the fourth row matters for monitoring. Microsoft describes the schedule trigger's behaviour as "fire and forget": the trigger run is successful as long as a pipeline run started. A green trigger run says nothing about whether the pipeline succeeded. Monitor pipeline runs, not trigger runs.

Choosing between them:

• Schedule, for work tied to the clock and not to a slice of data: a nightly full reload, a weekly report, housekeeping.
• Tumbling window, when each run processes one period of data and may need a backfill or rerun. See tumbling window triggers.
• Event-based, when work should start because data arrived, not because time passed. See event-based triggers.

A trigger must be started after it is created or published. A trigger in the stopped state does nothing, which is a frequent cause of "the pipeline did not run last night" after a deployment.

Schedule triggers in detail

A schedule trigger has a recurrence: a frequency, an interval, and optionally a schedule that narrows it to particular hours, minutes, week days or month days (Microsoft Learn: schedule trigger).

{
  "name": "WeekdaysAt0630",
  "properties": {
    "type": "ScheduleTrigger",
    "typeProperties": {
      "recurrence": {
        "frequency": "Week",
        "interval": 1,
        "startTime": "2026-10-05T00:00:00",
        "timeZone": "India Standard Time",
        "schedule": {
          "weekDays": [ "Monday", "Tuesday", "Wednesday", "Thursday", "Friday" ],
          "hours": [ 6 ],
          "minutes": [ 30 ]
        }
      }
    },
    "pipelines": [
      {
        "pipelineReference": { "type": "PipelineReference", "referenceName": "DailySalesLoad" },
        "parameters": {
          "runDate": { "type": "Expression", "value": "@trigger().scheduledTime" }
        }
      }
    ]
  }
}

Points to note:

• Time zone. The timeZone setting applies to the start time, end time and schedule. For UTC, times carry a Z suffix; for other zones they do not.
• Daylight saving. For zones that observe it, the trigger time adjusts automatically when the recurrence is in days or longer. Hourly and minute recurrences keep firing at regular intervals.
• Pass the scheduled time, not the current time. @trigger().scheduledTime is the time the run was meant for. Passing it as a parameter gives the pipeline a stable date to work with, where reading the clock inside the pipeline gives a different answer on a retry.
• The parameters property is required for each pipeline in the trigger, even if empty.
• One trigger, several pipelines, and several triggers per pipeline, are both allowed.

Patterns for ordering work

Sequential loads with a hard dependency. One orchestrator pipeline, Execute Pipeline activities chained on Succeeded, one trigger. Simple, and the order is guaranteed.

Independent loads at the same time. One trigger starting several pipelines, or an orchestrator with unchained Execute Pipeline activities.

Dependency across schedules. A daily aggregate that needs all of the day's hourly loads. Do not solve this with timing, by scheduling the aggregate "late enough". Use tumbling window triggers with a dependency, so the aggregate waits for the windows it needs.

Start when the file lands, but not before a cut-off. An event trigger for the arrival, with the pipeline checking that all expected files are present before proceeding.

Avoiding overlap. If a run can take longer than the gap between runs, two runs will overlap. A pipeline's concurrency setting limits how many runs execute at once; a tumbling window trigger with a self-dependency makes each window wait for the previous one.

Mistakes to avoid

• Scheduling by guesswork. Pipeline B at 03:00 "because A usually finishes by 02:30" fails the first time A is slow. Express the dependency.
• Watching trigger runs instead of pipeline runs.
• Forgetting to start the trigger after deploying.
• Clean-up attached with Succeeded, so it is skipped exactly when it is needed.
• Assuming UTC, or assuming local time. State the time zone in the trigger and in the documentation.
• Reading the clock in the pipeline instead of passing the trigger's time.

For loops and branches inside a pipeline, see control flow in ADF.

Azure Data Factory and workflow orchestration are covered in the Data Engineering course.

Sources

• Pipelines and activities in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Pipeline execution and triggers in Azure Data Factory or Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Create a trigger that runs a pipeline on a schedule (Microsoft Learn) — Microsoft Learn
• Execute Pipeline activity in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Create a trigger that runs a pipeline in response to a storage event (Microsoft Learn) — Microsoft Learn