Event-Based Triggers with Azure Event Grid and ADF

A nightly schedule is a guess about when data will be ready. Sometimes the guess is good. When files arrive at unpredictable times, or the business wants them processed within minutes, the better design is to let the arrival start the pipeline. Azure Data Factory supports that with event-based triggers.

The definitions below are written from Microsoft's documentation and have not been deployed.

Two kinds of event trigger

• Storage event trigger. Runs a pipeline when a blob is created or deleted in a storage account.
• Custom event trigger. Runs a pipeline when an application publishes an event to an Azure Event Grid custom topic.

Both rely on Azure Event Grid, the Azure service that routes events from publishers to subscribers. Data Factory integrates with it natively: creating the trigger creates a subscription, and Event Grid pushes matching events to the factory (Microsoft Learn: storage event trigger). You do not build or operate anything in between.

Storage event triggers

A storage event trigger has a small set of properties:

At least one of the two path filters is required.

{
  "name": "OnOrdersCsvArrival",
  "properties": {
    "type": "BlobEventsTrigger",
    "typeProperties": {
      "scope": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/data-rg/providers/Microsoft.Storage/storageAccounts/examplelake",
      "events": [ "Microsoft.Storage.BlobCreated" ],
      "blobPathBeginsWith": "/landing/blobs/orders/",
      "blobPathEndsWith": ".csv",
      "ignoreEmptyBlobs": true
    },
    "pipelines": [
      {
        "pipelineReference": { "type": "PipelineReference", "referenceName": "IngestOneOrdersFile" },
        "parameters": {
          "folderPath": "@triggerBody().folderPath",
          "fileName": "@triggerBody().fileName"
        }
      }
    ]
  }
}

Three details in that definition repay attention.

The path format. blobPathBeginsWith here means "container landing, folder orders". The /blobs/ segment between the container and the folder is required whenever you specify a container with a folder or a file; the authoring UI inserts it for you, hand-written JSON must include it.

Only prefix and suffix matching. The documentation states that begins-with and ends-with are the only pattern matching allowed; other wildcards are not supported. Design folder and file names so a prefix and a suffix are enough.

The file is passed to the pipeline. The trigger captures the blob's location in @triggerBody().folderPath and @triggerBody().fileName. They must be mapped to pipeline parameters, as above, and are then read inside the pipeline as @pipeline().parameters.fileName. A pipeline started this way processes one specific file, which is exactly what makes it reliable.

What the documentation says about limits

• Supported accounts. Azure Data Lake Storage Gen2 and general-purpose v2 storage accounts.
• 500 triggers per storage account, because of an Event Grid limitation.
• Permissions to create the trigger. The account publishing the trigger needs appropriate role-based access on the storage account; the factory's own identity needs no special permission on the storage account or on Event Grid. The subscription must also be registered for the Event Grid resource provider.
• Private storage. If the account blocks public network access, network rules must allow Blob Storage to communicate with Event Grid.
• Filters and cost. The trigger's data preview shows existing blobs that match. Microsoft warns that filters that are too broad can match a large number of files and significantly affect cost, since each match is a pipeline run.

Custom event triggers

Not every event is a file. An order system finishing its end-of-day close, an upstream job completing, a user requesting a refresh: these are business events. A custom event trigger handles them. Your application publishes an event to an Event Grid custom topic; Data Factory subscribes and runs the pipeline for matching events (Microsoft Learn: custom event trigger).

Events must follow the Event Grid event schema, with fields including subject and eventType, and the trigger filters on those. Values from the event's data payload can be passed to pipeline parameters through @triggerBody().

Use a custom event trigger when an upstream system can announce "I am done", which is more precise than watching for a file and far more precise than a schedule.

Designing for how events really behave

An event-triggered pipeline is simple to configure and has four characteristic failure modes. Plan for each.

One run per file. A thousand files arriving at once start a thousand pipeline runs. That may be exactly right, or it may overwhelm the sink. Set the pipeline's concurrency limit, keep each run small, and for bulk drops consider one pipeline that processes the folder instead of one run per file.

Duplicates. Event delivery systems can deliver an event more than once, and a file can be overwritten, which is a new blob-created event. The pipeline must be idempotent: record each processed file and its content hash or last-modified time in a control table, and skip or replace on a repeat. Lookup and Get Metadata covers the two activities that read that state.

Partial files. A large file being written in stages can raise an event before the content is complete, depending on how the writer uploads. Two defences work regardless of the upload method: have producers write to a temporary name or folder and rename or move on completion, so the trigger's filter only ever sees finished files; or have them drop a small marker file, such as _SUCCESS, last, and trigger on that.

Missed events. If a trigger is stopped during a deployment, files that arrive in that window raise no pipeline run. Event triggers have no backfill. A scheduled sweep that looks for unprocessed files, daily or hourly, closes the gap. Event-driven with a scheduled safety net is the dependable combination.

Event or schedule?

Event triggers and time-based triggers combine well. A file's arrival starts ingestion to the raw layer; a tumbling window trigger later builds the day's aggregates from whatever has landed. Pipeline dependencies and scheduling compares the trigger types.

Checklist

• Narrow filters: a specific folder prefix and a file extension.
• Map folderPath and fileName to pipeline parameters.
• Never write output to a path the trigger watches. A pipeline that writes a .csv into its own landing folder triggers itself.
• A processed-files control table, checked first.
• A concurrency limit on the pipeline.
• A failure alert, and a scheduled sweep for anything missed; see handling errors and retries.
• Start the trigger after publishing, and confirm it is running after each deployment.

Azure Data Factory and event-driven pipeline design are covered in the Data Engineering course.

Sources

• Create a trigger that runs a pipeline in response to a storage event (Microsoft Learn) — Microsoft Learn
• Create a custom event trigger to run a pipeline in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Pipeline execution and triggers in Azure Data Factory or Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• System variables supported by Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn