Monitoring and Alerts in Azure Data Factory

A pipeline that fails silently is worse than one that never ran, because people keep trusting the data. Out of the box, Azure Data Factory records every run in detail and notifies no one. ADF monitoring is therefore two jobs: knowing where to look, and arranging to be told.

The queries in this guide are written from Microsoft's documentation and have not been run against a workspace.

The Monitor hub

The Monitor section of Data Factory Studio is the first stop (Microsoft Learn: visually monitor).

Pipeline runs lists each run with its start, end, duration, what triggered it, status, parameters and error. You can filter by time range, status, pipeline name or annotation.

Activity runs open when you select a pipeline run. Each activity has icons for its input, its output and, for some activity types, a detailed view. The error icon shows the full message of a failed activity. This is where diagnosis happens.

Features worth knowing:

• Rerun. A run can be rerun from the start, from a chosen activity, or from the failed activity, optionally with new parameters.
• Gantt view shows runs as bars over time, grouped by pipeline or annotation. It is the quickest way to see overlap and drift in duration.
• Consumption shows the resources a run used, which can be put into the pricing calculator for an estimate.
• User properties. Up to five properties of an activity, such as a Copy activity's source and destination, can be promoted so they appear as columns.
• Debug runs are on their own tab, separate from triggered runs.

Two details catch people. The list does not refresh by itself; you select Refresh. And times are shown in the browser's local time zone unless you change it.

There is also a view for trigger runs. Remember what a successful trigger run means for a schedule trigger: only that a pipeline run was started. Pipeline dependencies and scheduling explains why pipeline runs are the thing to watch.

The 45-day limit

Data Factory stores pipeline run data for only 45 days (Microsoft Learn: monitor Azure Data Factory). After that the history is gone from the hub. If you need to compare this month-end with the last one, investigate an incident reported late, or show an auditor a year of runs, the built-in history is not enough.

The fix is a diagnostic setting, which routes the factory's logs elsewhere:

Set this up when the factory is created. Logs are not collected retroactively.

Log Analytics

In the factory's Diagnostic settings, add a setting, choose the log categories, and send them to a workspace. Microsoft recommends resource-specific mode, in which each category gets its own table (Microsoft Learn: configure diagnostic settings):

Other tables cover SSIS, which you can leave unselected if you do not use it. Expect a delay: up to 15 minutes can pass between an event and its appearance in the workspace.

With the data there, questions become queries. The columns below are from Microsoft's table reference (Microsoft Learn: ADFPipelineRun reference).

Pipelines that failed in the last day, with their errors:

ADFPipelineRun
| where TimeGenerated >= ago(24h)
| where Status == "Failed"
| project TimeGenerated, PipelineName, RunId, FailureType, ErrorMessage
| order by TimeGenerated desc

Which pipelines fail most over a month:

ADFPipelineRun
| where TimeGenerated >= ago(30d)
| where Status in ("Succeeded", "Failed")
| summarize runs = count(), failures = countif(Status == "Failed") by PipelineName
| extend failureRate = round(100.0 * failures / runs, 1)
| order by failures desc

How long each pipeline takes, to catch one that is slowly getting worse:

ADFPipelineRun
| where TimeGenerated >= ago(30d)
| where Status == "Succeeded"
| extend minutes = datetime_diff("minute", End, Start)
| summarize avgMinutes = avg(minutes), maxMinutes = max(minutes) by PipelineName, bin(TimeGenerated, 1d)

The activities behind the failures:

ADFActivityRun
| where TimeGenerated >= ago(7d)
| where Status == "Failed"
| summarize failures = count() by PipelineName, ActivityName, ActivityType, ErrorCode
| order by failures desc

Microsoft's own example queries filter on a column named FailureType to separate user errors from system errors. That distinction is useful in practice: a user error is usually your data or configuration, a system error usually is not.

Metrics

Metrics are collected automatically, with no configuration, and are lighter than logs. The ones that matter (Microsoft Learn: monitoring data reference):

Alerts

An alert rule watches a metric or a log query and notifies an action group, which holds the recipients: email, SMS, push or voice, and can call a webhook. Rules can be created from the factory's own Alerts and metrics page or from Azure Monitor.

A starting set:

1. Failed pipeline runs greater than 0. This is the first rule in Microsoft's recommended list, and the one every factory should have. It can be scoped to particular pipelines or failure types.
2. Failed trigger runs greater than 0.
3. Self-hosted runtime queue length or CPU above a threshold, if you run one.
4. No succeeded runs in an expected window. A missing run raises no failure. A pipeline whose trigger was left stopped after a deployment fails nothing and loads nothing. Alert on the absence, with a log query that counts successful runs of a named pipeline since midnight.
5. Duration above normal, from a log query, for the loads with a deadline.

Which type of rule to use:

Avoid one mistake above all: an alert that fires so often it is ignored. Send failures of critical pipelines to people; send everything else to a channel that is reviewed daily.

Notifications from inside a pipeline

Alert rules tell you a run failed. Sometimes you want a message with business context: which file, how many rows, what to do. For that, the pipeline sends its own. Microsoft's documented method is a Logic Apps workflow with an HTTP request trigger and a send-email action, called from a Web activity on the failure path, with the body built from system variables and the failed activity's error (Microsoft Learn: how to send email).

Use both. In-pipeline notifications are informative and can themselves fail, or never run if the pipeline is not started. A metric alert is blunt and independent of the pipeline. Handling errors and retries covers building the failure path.

What built-in monitoring does not tell you

A green run means the activities succeeded. It does not mean the data is right. A copy of zero rows succeeds. So does a copy of half the expected rows.

• Record row counts per load and compare with history. Building a metadata framework shows a load log that does this.
• Check freshness at the destination: the latest timestamp in the target table, not the status of the pipeline.
• Validate the data, as covered in data quality testing and validation.

A monitoring checklist

• Diagnostic setting to Log Analytics, resource-specific mode, on every factory.
• Workspace retention set to what audit and analysis need.
• Metric alert on failed pipeline runs, with an action group that reaches a person.
• Log alerts for missing runs and long runs on critical pipelines.
• Annotations on pipelines, so runs can be filtered by system or owner.
• A weekly look at failure counts and duration trends.
• Row counts and freshness checked, not only run status.

For the long-term record and who-changed-what, see ADF logging and auditing. The equivalent on AWS is covered in CloudWatch for monitoring data pipelines.

Azure Data Factory, with pipeline monitoring and alerting, is covered in the Data Engineering course.

Sources

• Visually monitor Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Monitor Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Configure diagnostic settings and a workspace (Microsoft Learn) — Microsoft Learn
• Monitoring data reference for Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Azure Monitor Logs reference: ADFPipelineRun (Microsoft Learn) — Microsoft Learn
• How to send email (Microsoft Learn) — Microsoft Learn