ADF Pipeline Best Practices for Production

Most pipelines that fail in production worked perfectly on the day they were built. They were run once, by their author, on good data, and watched. Production is the opposite: unattended, repeated, on whatever data turns up. These ADF pipeline best practices are the ones that address that difference, grouped by the question a reviewer should ask.

Is the pipeline small enough to understand?

One pipeline, one job. Ingest one source, or load one subject area. A parent pipeline calls the children in order with Execute Pipeline. Small pipelines can be rerun alone and read in a minute.

Stay well inside the limits. They are documented (Microsoft Learn: Azure service limits):

A pipeline approaching 120 activities was too large long before it got there.

Parameterize instead of copying. Two pipelines that differ by a table name should be one pipeline with a parameter. Forty should be a control table. See parameterizing pipelines.

Name things so the name is enough. A consistent pattern such as a prefix for the object type, then source, then subject, makes the Monitor list readable. Two rules come from the service itself: names must start with a letter, and dashes are not allowed in the names of linked services, datasets and data flows (Microsoft Learn: naming rules). Microsoft's deployment guidance adds a third: avoid spaces, which cause problems in Resource Manager templates, and use underscores.

Use folders and annotations. Folders organise the authoring view; annotations let you filter runs in Monitor by system or owner.

Can it be run twice?

This is the most important question, because every pipeline is eventually rerun: after a failure, by a retry, by a backfill, by mistake.

• Make each load replace or merge, never blindly append. A pre-copy script that clears the target slice, an upsert on a key, or overwriting a dated folder. Idempotency in data pipelines explains the principle.
• Pass the period in. Use the trigger's scheduled or window time as a parameter. A pipeline that reads the clock produces different results on a rerun.
• Move watermarks last, after the load has succeeded.

What happens when something fails?

An activity's policy has defaults worth knowing: a time-out of 12 hours, zero retries, and a 30-second retry interval (Microsoft Learn: pipelines and activities).

• Set time-outs deliberately. With the default, a copy that hangs holds its place for twelve hours before anyone is told. Set each activity's time-out to a small multiple of its normal duration.
• Retry what is transient, and only that. Two or three retries with a pause suit network and throttling errors. A retry will not fix a missing file or a schema change, and on a load that is not idempotent it does harm.
• Build a failure path that logs the error and notifies someone, and make sure the pipeline still ends as failed. Handling errors and retries covers how dependency conditions decide a pipeline's final status.
• Limit concurrency where overlap would hurt. By default a pipeline has no maximum number of concurrent runs. Set it to 1 for a load that must not overlap with itself; further runs queue.

Where are the credentials?

• Managed identity first. A linked service that authenticates as the factory has no secret to store, rotate or leak.
• Key Vault for the rest. No password typed into a linked service. See Azure Key Vault with Data Factory.
• No secrets in parameters. Parameters are recorded with the run. For activities that handle sensitive values, the secure input and secure output options keep them out of the monitoring logs.
• Least privilege at the data stores. The factory's identity should read only what it copies and write only where it loads.
• Enforce with policy. Azure Policy has built-in definitions requiring Key Vault for linked service secrets and managed identity where supported (Microsoft Learn: built-in policy definitions).

How does a change reach production?

Microsoft's CI/CD guidance is specific, and worth following closely (Microsoft Learn: continuous integration and delivery):

1. Git on the development factory only. Test and production have no repository attached and are updated only by a deployment pipeline or a Resource Manager template.
2. Changes go through a pull request into the collaboration branch.
3. A release pipeline promotes the same template to test, then to production, with per-environment values supplied as template parameters.
4. Triggers are stopped before deployment and restarted after, by the pre- and post-deployment script. A trigger left stopped is a common cause of a pipeline that simply does not run the next morning.
5. A key vault per environment, with the same secret names in each. Then only the vault's name changes between environments.
6. The same integration runtime names in every environment.

Two constraints to plan around: publishing is all or nothing, since Data Factory does not support publishing a selected subset of resources, and environment-specific logic is handled with global parameters. CI/CD for Azure Data Factory walks through the set-up.

The rule that matters most is the simplest. Nobody edits production by hand.

What does it cost, and why?

Data Factory bills by consumption on separate meters (Microsoft Learn: plan to manage costs): activity runs for orchestration, DIU-hours for copies on the Azure integration runtime, and vCore-hours for data flows, including debug sessions.

• Find the bottleneck before raising DIUs. More units do nothing when the source, sink or network is the limit. See ADF performance tuning.
• Turn off data flow debug. A session is a running cluster. The service caps a session at 8 hours; do not rely on the cap.
• Group data flows sensibly. Sequential flows share a warm cluster with a time to live; a data flow inside a ForEach starts a cluster per iteration.
• Watch the activity count in loops. A ForEach over ten thousand items with five activities each is fifty thousand activity runs per execution.
• Use the consumption report on a pipeline run to see which meter a pipeline is spending on.

Will anyone know when it breaks?

• Diagnostic settings from day one. Run history is kept for 45 days unless you route it to Log Analytics or storage (Microsoft Learn: monitor Azure Data Factory).
• An alert on failed pipeline runs, reaching a person.
• An alert on runs that did not happen. A stopped trigger fails nothing.
• Row counts and freshness, not only status. A copy of zero rows succeeds.

Monitoring and alerts covers these in depth.

Was it tried with bad data?

Debug runs prove the happy path. Before release, run the pipeline against:

• An empty source.
• A source with a duplicate key, a null in a required column, and a row that is too long.
• The same period twice.
• A deliberately failed middle step, to see what the rerun does.

Those four runs find most of what production would have found for you.

A note on new projects

Microsoft now describes Data Factory in Microsoft Fabric as the next generation of Azure Data Factory and suggests that those new to data integration start there. The practices above are not specific to one product: small units, safe reruns, managed credentials, automated deployment and monitoring apply to pipelines in either.

The review checklist

Azure Data Factory and production data pipelines are covered in the Data Engineering course.

Sources

• Azure subscription and service limits, quotas, and constraints (Microsoft Learn) — Microsoft Learn
• Pipelines and activities in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Continuous integration and delivery in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Azure Data Factory naming rules (Microsoft Learn) — Microsoft Learn
• Plan to manage costs for Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Monitor Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Azure Policy built-in policy definitions for Data Factory (Microsoft Learn) — Microsoft Learn