Azure Databricks with ADF: Orchestration Patterns

The division of labour is settled in most Azure data platforms: Data Factory brings the data in and keeps the schedule, Databricks does the heavy transformation. What is less settled is the seam between them. How does a pipeline start a notebook, tell it what to process, learn what happened, and avoid paying for ten clusters where one would do? This guide covers running Azure Databricks with ADF in practice.

The definitions below are written from Microsoft's documentation and have not been deployed.

Four activities

Data Factory has four activities for Databricks:

The first three run a single piece of code on a cluster the linked service describes. The fourth is different in kind: it starts a job that is defined, with all its tasks, inside Databricks (Microsoft Learn: Databricks Job activity).

The linked service decides the cluster

An Azure Databricks linked service holds two decisions: how to authenticate, and where the code runs (Microsoft Learn: compute environments).

Authentication is by access token or by the factory's managed identity. With the identity there is no token to store or rotate. If a token is used, keep it in Key Vault; see Azure Key Vault with Data Factory and managed identities.

Compute has three options, plus a fourth for the Job activity:

Microsoft's page carries a pointed remark on the second option: Databricks suggests running jobs on new clusters for greater reliability, and an interactive cluster may need a manual restart if it stops responding.

{
  "name": "DatabricksJobCluster",
  "properties": {
    "type": "AzureDatabricks",
    "typeProperties": {
      "domain": "https://adb-0000000000000000.0.azuredatabricks.net",
      "accessToken": {
        "type": "AzureKeyVaultSecret",
        "store": { "referenceName": "CorpKeyVault", "type": "LinkedServiceReference" },
        "secretName": "databricks-access-token"
      },
      "newClusterVersion": "15.4.x-scala2.12",
      "newClusterNodeType": "Standard_D4ds_v5",
      "newClusterNumOfWorker": "2:8"
    }
  }
}

The worker setting accepts a fixed number or a range: "2:8" means autoscale between two and eight workers. The example keeps its token in Key Vault; with managed identity authentication, which the same page documents, no token is needed at all.

Why the cluster choice dominates

A transformation step is rarely one notebook. Suppose a pipeline runs ten notebooks of four minutes each. The script below works out elapsed time and cluster-minutes for four arrangements. It is a model: the five-minute start-up is an assumption, and real start-up times vary with the cluster and the day.

# A model of cluster strategies for a pipeline that runs 10 notebooks of 4 minutes each.
# START is an ASSUMED time for a new job cluster to become ready; change it and run again.
NOTEBOOKS, RUN, START = 10, 4, 5          # count, minutes of work each, minutes to start a cluster

strategies = {
    # name: (elapsed minutes, cluster-minutes paid for)
    "10 activities in sequence, new job cluster each": (NOTEBOOKS * (START + RUN), NOTEBOOKS * (START + RUN)),
    "10 activities in parallel, new job cluster each": (START + RUN,               NOTEBOOKS * (START + RUN)),
    "1 job with 10 tasks on one job cluster":          (START + NOTEBOOKS * RUN,   START + NOTEBOOKS * RUN),
    "10 activities in sequence on a running cluster":  (NOTEBOOKS * RUN,           NOTEBOOKS * RUN),
}
print(f"{NOTEBOOKS} notebooks x {RUN} min of work, {START} min assumed cluster start\n")
print(f"{'strategy':50} {'elapsed':>8} {'cluster-min':>12} {'of which start-up':>18}")
for name, (elapsed, cluster_minutes) in strategies.items():
    startup = cluster_minutes - NOTEBOOKS * RUN
    print(f"{name:50} {elapsed:8} {cluster_minutes:12} {startup / cluster_minutes:18.0%}")

10 notebooks x 4 min of work, 5 min assumed cluster start

strategy                                            elapsed  cluster-min  of which start-up
10 activities in sequence, new job cluster each          90           90                56%
10 activities in parallel, new job cluster each           9           90                56%
1 job with 10 tasks on one job cluster                   45           45                11%
10 activities in sequence on a running cluster           40           40                 0%

Read the rows as choices:

• Ten activities in sequence, each on a new job cluster, take 90 minutes for 40 minutes of work. More than half of what you pay for, 56%, is clusters starting.
• The same ten in parallel finish in 9 minutes and cost the same 90 cluster-minutes. Parallelism buys time, not money.
• One job with ten tasks on one cluster takes 45 minutes and 45 cluster-minutes. Start-up is paid once.
• A cluster that is already running has no start-up at all. The figure flatters it, though: the model does not count the hours such a cluster runs between pipelines.

The conclusion is not that one row is right. It is that the unit of work matters. Ten tiny notebooks each demanding a fresh cluster is the most expensive way to run forty minutes of Spark.

Parameters in

A pipeline tells a notebook what to do through base parameters (Microsoft Learn: Databricks Notebook activity):

{
  "name": "CleanSales",
  "type": "DatabricksNotebook",
  "linkedServiceName": { "referenceName": "DatabricksJobCluster", "type": "LinkedServiceReference" },
  "typeProperties": {
    "notebookPath": "/Repos/data-platform/pipelines/clean_sales",
    "baseParameters": {
      "run_date": { "value": "@pipeline().parameters.runDate", "type": "Expression" },
      "source_path": { "value": "@concat('raw/sales/', pipeline().parameters.runDate)", "type": "Expression" }
    }
  }
}

If the notebook defines a parameter that the activity does not pass, the notebook's own default is used. That is convenient in development and dangerous in production: a misspelt parameter name does not fail, it silently runs with the default. Give production notebooks no usable defaults for anything that matters, so a missing value stops the run.

Pass the run date from the trigger, never let the notebook read the clock. A rerun for last Tuesday must process last Tuesday.

Results out

A notebook hands a value back by ending with dbutils.notebook.exit. The pipeline reads it as the activity's runOutput. The documented limit on the returned value is 2 MB.

The notebook side:

import json

run_date = dbutils.widgets.get("run_date")
source_path = dbutils.widgets.get("source_path")

# ... read, clean and write the data ...

dbutils.notebook.exit(json.dumps({"rows_written": rows_written, "rows_rejected": rows_rejected}))

The pipeline side, an If Condition that depends on the notebook:

{
  "name": "CheckRejects",
  "type": "IfCondition",
  "dependsOn": [ { "activity": "CleanSales", "dependencyConditions": [ "Succeeded" ] } ],
  "typeProperties": {
    "expression": {
      "value": "@greater(int(activity('CleanSales').output.runOutput.rows_rejected), 0)",
      "type": "Expression"
    }
  }
}

When the returned value is a JSON object, its properties can be read by name, as runOutput.rows_rejected is here.

Return facts, not data. Row counts, a status, the path that was written. The data itself goes to the lake, where the next step reads it.

Failure

A notebook that raises an unhandled error fails the activity, and the pipeline's dependency conditions take over. Two habits make that work:

• Let errors fail. A notebook that catches every exception and exits normally reports success for a failed load. If you catch an error in order to log it, raise it again.
• Make the notebook safe to rerun. Data Factory's retry setting will run it again. Writing with overwrite of the run date's partition, or a merge on a key, means the second attempt replaces the first. Handling errors and retries covers the pipeline side.

Patterns that hold up

Ingest, then transform. Copy activities land raw files; one Notebook activity per layer of refinement follows. Data Factory owns the order and the schedule. This is the common shape, and it matches the layers described in the medallion architecture.

One coarse step, not many fine ones. Where several notebooks always run together, group them in Databricks, either with a driver notebook or as tasks of a Databricks job, and call that once. The model above shows why.

Hand over the workflow with the Job activity. Define the multi-task workflow as a Databricks job, with its own task dependencies, and let Data Factory start it and wait. Each tool then orchestrates what it knows: Data Factory the cross-system steps, Databricks the steps inside it (Microsoft Learn: Lakeflow Jobs).

Fan out with ForEach, with care. A ForEach over a list of tables, running the same parameterized notebook for each, is the metadata-driven pattern applied to Databricks. Set the batch count deliberately. Twenty parallel iterations on new job clusters are twenty clusters; on a shared cluster they compete for its cores.

One generic notebook, many parameters, in preference to many near-identical notebooks.

What to keep where

Pass @pipeline().RunId to the notebook as a parameter and write it into whatever the notebook logs. One identifier then ties the Data Factory run, the Databricks run and the rows in the Delta tables together.

Mistakes to avoid

• A new cluster per small notebook, in a long sequence.
• Production jobs on an interactive cluster that analysts also use.
• Two schedules: a trigger in Data Factory and a schedule on the Databricks job, both active.
• Returning data through the notebook's exit value.
• Swallowed exceptions.
• Notebook defaults that mask missing parameters.
• Access tokens typed into the linked service.

If you are still deciding how the two tools divide the work, start with ADF vs Databricks.

Azure Data Factory, Azure Databricks and PySpark are covered in the Data Engineering course.

Sources

• Transform data by running a Databricks notebook (Microsoft Learn) — Microsoft Learn
• Transform data by running a Databricks job (Microsoft Learn) — Microsoft Learn
• Compute environments supported by Azure Data Factory and Synapse pipelines (Microsoft Learn) — Microsoft Learn
• Transform data by running a Python activity in Azure Databricks (Microsoft Learn) — Microsoft Learn
• Transform data by running a Jar activity in Azure Databricks (Microsoft Learn) — Microsoft Learn
• Lakeflow Jobs (Microsoft Learn) — Microsoft Learn