ADF Global Parameters and Variables Explained

Azure Data Factory gives you three places to keep a value, and they are easy to mix up because all three appear in expressions that look alike. Choosing the wrong one produces errors that are puzzling until the rules are clear: a value that refuses to change, a counter that will not increment, a loop whose results overwrite each other.

The rules are few.

The three, side by side

In one line each: a parameter is an input, a variable is working memory, a global parameter is a constant.

Pipeline parameters: inputs

A parameter is fixed for the life of a run. Microsoft's wording is that parameters are defined at the pipeline level and cannot be modified during a pipeline run (Microsoft Learn: parameters and variables). Whatever started the run supplies the value, or the default applies.

That immutability is a feature. A run's parameters are recorded with it, so you can see exactly what a run was asked to do, and rerun it with the same inputs. Parameterising pipelines covers their use.

Variables: working memory

A variable holds a value that an activity sets while the pipeline is running. Typical uses:

• Storing something computed once and used several times, such as a formatted date or a folder path.
• A flag recording whether an earlier step found anything.
• A counter or cursor in an Until loop.
• A list built up over a ForEach.

Two activities change variables. Set Variable assigns a value. Append Variable adds an item to an array variable.

Three limits matter.

Only three types. String, Bool and Array. There is no integer type, so a counter is held as a string and converted with int() when used.

Pipeline scope. A variable belongs to one pipeline. A child pipeline called with Execute Pipeline cannot see the parent's variables; pass values as parameters. In the other direction, Set Variable can set a pipeline return value, which the calling pipeline can read from the Execute Pipeline activity's output.

No self-reference. This is the one that surprises everyone. In a Set Variable activity you cannot reference the variable being set in its own value (Microsoft Learn: Set Variable activity). counter = counter + 1 is not allowed.

Incrementing a counter

The documented workaround is a temporary variable and two activities: compute the new value into the temporary, then copy it back. Written from the documentation and not run:

[
  {
    "name": "ComputeNext",
    "type": "SetVariable",
    "typeProperties": {
      "variableName": "temp",
      "value": { "value": "@string(add(int(variables('counter')), 1))", "type": "Expression" }
    }
  },
  {
    "name": "StoreNext",
    "type": "SetVariable",
    "dependsOn": [ { "activity": "ComputeNext", "dependencyConditions": [ "Succeeded" ] } ],
    "typeProperties": {
      "variableName": "counter",
      "value": { "value": "@variables('temp')", "type": "Expression" }
    }
  }
]

Both variables are strings: int() converts for the arithmetic and string() converts back.

Collecting values, and the parallel trap

Append Variable builds an array, for example the names of files that failed validation:

{
  "name": "RecordFailure",
  "type": "AppendVariable",
  "typeProperties": {
    "variableName": "failedFiles",
    "value": { "value": "@item().name", "type": "Expression" }
  }
}

Appending inside a ForEach is the usual way to gather results; in a parallel loop the items arrive in no guaranteed order.

Set Variable inside a parallel loop is the trap. The documentation warns that variables are scoped at the pipeline level, so they are not thread safe and can cause unexpected behaviour when accessed within a parallel iteration such as a ForEach, especially when the value is also modified there. All iterations share one variable. Iteration A sets it, iteration B overwrites it, and A then reads B's value.

The remedies:

• Do not use a variable for per-iteration state. Use item() and activity outputs directly inside the loop.
• Move the loop body into a child pipeline called with Execute Pipeline. Each call gets its own variables.
• Make the ForEach sequential if order and shared state truly matter, accepting that it is slower.

Control flow in ADF covers ForEach settings.

Global parameters: constants for the factory

Some values are the same for every pipeline and differ only by environment: the name of the storage account, an alert email address, a flag for whether this is production. Repeating them as a parameter on fifty pipelines invites inconsistency.

Global parameters are constants across a data factory that pipelines can use in any expression (Microsoft Learn: global parameters). They are created on the Manage tab and read with @pipeline().globalParameters.name.

Two points on using them:

• They reach datasets and data flows by being passed. A dataset cannot read a global parameter itself; the pipeline passes the value into the dataset's own parameter.
• They are the hook for environments. Microsoft's guidance is that they can be overridden in each environment during continuous integration and deployment, and recommends including them in the ARM template. Development's factory holds one storage account name, production's another, and no pipeline changes. CI/CD for Azure Data Factory covers the deployment side.

Do not put secrets in global parameters. They are configuration, visible to anyone who can view the factory. Secrets go in Key Vault.

Which to use

Mistakes to avoid

• Trying to change a parameter mid-run. Copy it into a variable first if it must change.
• Incrementing in one activity, as in counter = counter + 1.
• Per-iteration state in a variable inside a parallel ForEach.
• Using a variable where an activity output would do. activity('Lookup1').output is already available; copying it into a variable adds an activity and a place to go wrong.
• Dozens of global parameters. Keep them for true constants; pipeline-specific settings belong on the pipeline.
• Expecting variables to persist between runs. They are reset every run. State that must survive, such as a watermark, belongs in a table or a file.

For expression syntax, see ADF expressions and functions.

Azure Data Factory and pipeline design are covered in the Data Engineering course.

Sources

• Pipeline parameters and variables in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Global parameters in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Set Variable activity in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Append Variable activity in Azure Data Factory and Synapse Analytics (Microsoft Learn) — Microsoft Learn