ADF vs Databricks: When to Use Each

Teams new to Azure often frame ADF vs Databricks as a choice between two ETL tools. That framing leads to trouble, because the two were built for different jobs and most platforms that have one have both. The overlap is real, though, and it has grown. This guide separates the roles, looks squarely at the overlap, and gives a way to decide.

What each one is

Azure Data Factory is a managed service for data integration. In Microsoft's description it creates data-driven workflows that orchestrate data movement and transform data at scale (Microsoft Learn: introduction to Azure Data Factory). Its strengths are connectors, reaching into private networks, and scheduling.

Azure Databricks is an analytics platform built on Apache Spark and the lakehouse. Microsoft describes it as a unified, open analytics platform for building data, analytics and AI solutions at scale (Microsoft Learn: what is Azure Databricks). You write code in notebooks, in Python, SQL, Scala or R, and it runs on clusters the platform manages. Around that core sit Delta Lake tables, SQL warehouses, machine learning tooling and Unity Catalog for governance.

The shortest honest summary: Data Factory is about getting data from place to place and making things happen in order. Databricks is about doing heavy work on data once it has arrived.

Where they overlap

Three areas, and it helps to name them.

Transformation. Data Factory has mapping data flows, visual transformations that run on Spark clusters the service manages. Databricks has notebooks and jobs running Spark code. Both can join, aggregate and reshape large data.

Orchestration. Data Factory has pipelines and triggers. Databricks has Lakeflow Jobs: jobs made of tasks with dependencies, branching and loops, started by time-based or event-based triggers, with notifications on failure (Microsoft Learn: Lakeflow Jobs). A job can hold up to 1000 tasks.

Ingestion. Data Factory has the Copy activity. Databricks has its own tools for loading files from cloud storage incrementally.

Outside the overlap each has territory the other does not enter. Data Factory has no notebooks, no tables of its own, no machine learning. Databricks is not where you go for a self-hosted runtime reaching an on-premises SQL Server, or for running SSIS packages.

The same transformation, two ways

The overlap in transformation is the one people argue about, so here is a concrete case. Mapping data flows in ADF walks through a flow on the Chinook sample database (Chinook on GitHub): filter invoices to 2013, look up each customer, derive a customer type, aggregate by country and type, and sort.

Below is that same logic as PySpark, the code you would write in a Databricks notebook. It was run on open-source PySpark on a local machine, not on Databricks.

import csv, sqlite3
from pyspark.sql import SparkSession, functions as F

# export two tables of the Chinook sample database as files, the way a Copy activity would land them
db = sqlite3.connect("chinook.sqlite")
for name, sql in {"invoices": "SELECT InvoiceId, CustomerId, InvoiceDate, Total FROM Invoice",
                  "customers": "SELECT CustomerId, Country, Company FROM Customer"}.items():
    rows = db.execute(sql)
    with open(f"{name}.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.writer(f)
        writer.writerow([column[0] for column in rows.description])
        writer.writerows(rows)

spark = SparkSession.builder.master("local[2]").appName("sales-by-country").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
invoices = spark.read.csv("invoices.csv", header=True, inferSchema=True)
customers = spark.read.csv("customers.csv", header=True, inferSchema=True)

summary = (
    invoices
    .filter(F.col("InvoiceDate") >= "2013-01-01")                                     # Filter
    .join(customers, "CustomerId", "left")                                            # Lookup
    .withColumn("CustomerType",                                                       # Derived column
                F.when(F.col("Company").isNotNull(), "Business").otherwise("Individual"))
    .groupBy("Country", "CustomerType")                                               # Aggregate
    .agg(F.count("InvoiceId").alias("Invoices"), F.round(F.sum("Total"), 2).alias("Revenue"))
    .orderBy(F.desc("Revenue"))                                                       # Sort
    .limit(5)
)
summary.show()
spark.stop()

+--------------+------------+--------+-------+
|       Country|CustomerType|Invoices|Revenue|
+--------------+------------+--------+-------+
|           USA|  Individual|      14|  75.24|
|        Canada|  Individual|      12|  56.43|
|        France|  Individual|       6|  40.59|
|        Brazil|    Business|       7|  37.62|
|United Kingdom|  Individual|       6|  28.71|
+--------------+------------+--------+-------+

The result is identical to the data flow version, row for row. That is expected: a data flow is translated into Spark, and this is Spark. What differs is everything around the result.

Neither column is the winner. A team of SQL-minded analysts will be productive on the canvas in a day. A team of engineers will find the code version easier to test, review and refactor, and will reach for libraries the canvas does not have.

A fuller comparison

When to use which

The last two rows are the orchestration decision, and the rule is about scope. When every step is a Databricks task, the platform's own jobs keep the definition, the code and the monitoring together. When the workflow crosses systems, a copy from SQL Server, then a notebook, then a stored procedure in a warehouse, then an email, Data Factory is the more natural conductor. Orchestration: Airflow vs Dagster vs Prefect covers the same question for open-source tools.

Using both

The common arrangement uses each for what it does well, and Microsoft's documentation supports it directly: Data Factory lists Databricks among the compute services it can drive, with activities for notebooks, Jar files and Python files (Microsoft Learn: transform data).

1. Data Factory ingests. Copy activities bring data from source systems into the raw zone of the lake.
2. Data Factory calls Databricks. A Notebook activity runs the transformation, passing the run date and paths as parameters (Microsoft Learn: Databricks notebook activity).
3. Databricks transforms. Spark cleans, joins and models the data into Delta tables.
4. Data Factory continues. It loads a warehouse, refreshes a report, or notifies, depending on the notebook's result.

Azure Databricks with ADF: orchestration patterns covers how to wire this up, including clusters, parameters and return values.

Can one replace the other?

Databricks without Data Factory is workable when sources are files in cloud storage or systems Databricks can reach directly, and the whole pipeline lives in notebooks and jobs. The gap is usually connectivity to on-premises and business systems.

Data Factory without Databricks is workable when transformations are modest and data flows or SQL in the destination database are enough. The gap appears with complex logic, streaming, machine learning, or a team that wants to work in code.

Most organisations beyond a certain size have requirements on both sides, which is why the pairing is so common.

Cost, in principle

The two bill differently, and specific prices change, so only the shape is given here. Data Factory charges per activity run, per unit of data movement and per core-hour of data flow. Databricks charges for the compute its clusters and warehouses use while running. In both, the expensive mistake is the same: compute left running, or started far more often than needed. A data flow in a loop starts a cluster per iteration; an interactive cluster left on over a weekend bills for the weekend.

Mistakes to avoid

• Choosing one "for everything" on principle. Each is weak where the other is strong.
• Rebuilding ingestion in notebooks when a Copy activity and a connector would do it.
• Forcing complex logic into data flows because the team wants to avoid code. Past a point, the canvas is harder to maintain than the code would be.
• Two schedulers for one workflow. A Data Factory trigger and a Databricks job schedule, each assuming the other has finished. Pick one conductor.
• Ignoring skills. The right tool for a team that cannot use it is the wrong tool.

For the platforms themselves, see the Databricks tutorial and the PySpark tutorial.

Azure Data Factory, Azure Databricks and PySpark are covered in the Data Engineering course.

Sources

• Introduction to Azure Data Factory (Microsoft Learn) — Microsoft Learn
• What is Azure Databricks? (Microsoft Learn) — Microsoft Learn
• Lakeflow Jobs (Microsoft Learn) — Microsoft Learn
• Transform data in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Transform data by running a Databricks notebook (Microsoft Learn) — Microsoft Learn
• Chinook sample database (GitHub) — GitHub