Mapping Data Flows in ADF: Transformations Without Code

A Copy activity moves data. It does not reshape it: no joins, no aggregations, no conditional logic. For that, Azure Data Factory offers mapping data flows, where you draw the transformation as a graph and the service runs it on Apache Spark for you. This guide explains how mapping data flows in Azure Data Factory work, what building one looks like, and when they are the right choice.

One note first. Microsoft now describes Data Factory in Microsoft Fabric as the next generation of the product, where the equivalent feature is Dataflow Gen2. Data flows remain fully documented for Azure Data Factory and Azure Synapse, and that is what this article covers.

What a data flow is

Microsoft's definition has three parts (Microsoft Learn: data flow overview):

• Visually designed. You build the logic on a canvas by adding transformations, not by writing code.
• Run on Spark. The service translates the graph into Spark jobs and runs them on clusters it creates and manages. You never see or administer the cluster.
• Executed from a pipeline. A data flow does nothing on its own. A pipeline runs it through the Data Flow activity, so scheduling, parameters and monitoring are the ones you already use.

That last point is the mental model to keep: the pipeline orchestrates, the data flow transforms.

The canvas

The authoring screen has three areas:

• The graph shows the streams, from one or more sources on the left to one or more sinks on the right. A plus sign after any transformation adds the next one.
• The configuration panel shows the settings of whichever transformation is selected.
• The top bar holds actions for the whole flow: validation, the debug switch, and a button to see the script behind the graph.

Every transformation has at least four tabs:

Inspect works without a cluster. Data preview needs debug mode.

One flow, stage by stage

A typical flow reads two sources, filters one, looks up columns from the other, derives a new column, aggregates and writes the result. The script below reproduces those stages in pandas on the Invoice and Customer tables of the Chinook sample database (Chinook on GitHub) and prints the shape after each stage, which is what Inspect and Data preview would show you. It is a local reproduction of the logic, not a run of Data Factory.

import sqlite3
import pandas as pd

db = sqlite3.connect("chinook.sqlite")


def show(step, df):
    # what the Inspect and Data preview tabs tell you at each transformation: shape and columns
    print(f"{step:34} {len(df):4} rows  {df.shape[1]:2} columns")


# Source x2
invoices = pd.read_sql("SELECT InvoiceId, CustomerId, InvoiceDate, Total FROM Invoice", db)
customers = pd.read_sql("SELECT CustomerId, Country, Company FROM Customer", db)
show("Source: invoices", invoices)
show("Source: customers", customers)

# Filter
recent = invoices[invoices["InvoiceDate"] >= "2013-01-01"]
show("Filter: 2013 only", recent)

# Lookup (a left outer join that appends columns)
joined = recent.merge(customers, on="CustomerId", how="left")
show("Lookup: add customer columns", joined)

# Derived column
joined["CustomerType"] = joined["Company"].notna().map({True: "Business", False: "Individual"})
show("Derived column: CustomerType", joined)

# Aggregate: only group-by and aggregate columns survive
summary = joined.groupby(["Country", "CustomerType"], as_index=False).agg(
    Invoices=("InvoiceId", "count"), Revenue=("Total", "sum"))
show("Aggregate: country and type", summary)

# Sort, then Sink
top = summary.sort_values("Revenue", ascending=False).head(5).round({"Revenue": 2})
print()
print(top.to_string(index=False))

Source: invoices                    412 rows   4 columns
Source: customers                    59 rows   3 columns
Filter: 2013 only                    80 rows   4 columns
Lookup: add customer columns         80 rows   6 columns
Derived column: CustomerType         80 rows   7 columns
Aggregate: country and type          24 rows   4 columns

       Country CustomerType  Invoices  Revenue
           USA   Individual        14    75.24
        Canada   Individual        12    56.43
        France   Individual         6    40.59
        Brazil     Business         7    37.62
United Kingdom   Individual         6    28.71

Reading the shapes down the page is exactly how you check a flow on the canvas:

• Filter changes the row count and nothing else: 412 rows become 80.
• Lookup keeps every row of the main stream, still 80, and adds columns, 4 to 6. If the count had grown, the lookup key would not be unique on the other side.
• Derived column adds one column and leaves the rows alone.
• Aggregate changes everything: 24 rows, and only the grouping columns and the aggregates survive. Columns you did not group by or aggregate are gone, which surprises people who expect them to pass through.

The transformations available

Microsoft groups the transformations by what they do to the stream (Microsoft Learn: transformation overview):

The transformation reference covers each one.

Behind the graph: data flow script

The graph is stored as text called data flow script, which the Script button shows. Each line has the same form: the incoming stream, the transformation with its properties, then ~> and the name of the new stream (Microsoft Learn: data flow script). A minimal flow in that form:

source(output(
        InvoiceId as integer,
        InvoiceDate as timestamp,
        Total as decimal(10,2)
    ),
    allowSchemaDrift: true,
    validateSchema: false) ~> Invoices
Invoices filter(year(InvoiceDate) == 2013) ~> Only2013
Only2013 derive(TotalBand = iif(Total >= 10, 'large', 'small')) ~> AddBand
AddBand sink(allowSchemaDrift: true,
    validateSchema: false) ~> SalesOut

You rarely write this by hand. It matters because it makes a flow readable in a code review, and because an error message that names a stream is easier to trace in the script than on a large canvas.

How a data flow runs, and what that costs

Three facts about execution shape every decision about data flows.

A cluster has to start. Each data flow job uses its own just-in-time Spark cluster, and Microsoft puts the start-up at generally 3 to 5 minutes (Microsoft Learn: performance guide). A flow that transforms a thousand rows still waits for that. A time to live setting on the integration runtime keeps a cluster warm for the next job in a sequence.

You pay for cores over time. Data flows are billed in vCore-hours of the cluster, a different meter from the Copy activity. A larger cluster costs more per minute and may finish sooner.

Debug mode is a running cluster. Switching on debug starts a cluster so that Data preview works, and Microsoft's documentation is explicit that you are charged for every hour the session runs, including its time to live (Microsoft Learn: debug mode). Preview does not write to the sink; to exercise the sink, run the flow from a pipeline. Debug and performance goes further.

Two limits also matter in practice. Data flows run on the Azure integration runtime, not on a self-hosted one, so on-premises data has to be landed in Azure first; see integration runtimes. And data flows read and write a shorter list of stores than the Copy activity. Microsoft's guidance for the others is to use the Copy activity to stage the data in a supported store.

When to use a data flow

The comparison with hand-written Spark is the one people ask about most. A data flow is Spark underneath, so the concepts in Apache Spark explained, partitions, shuffles, joins, all apply. What you give up is control over the code; what you gain is that nobody has to write or maintain it.

Mistakes to avoid

• Leaving debug on. The session bills until it times out or is switched off.
• Running a data flow inside a ForEach over files. Each iteration starts its own cluster. One flow reading a folder or a wildcard path does the same work on one.
• Expecting columns to survive an aggregate.
• Checking joins on sampled data. Preview reads a limited number of rows, so matching keys may not be in the sample. Use a small known file when checking join logic.
• Using a data flow where a Copy would do.

Azure Data Factory, with Spark-based transformation, is covered in the Data Engineering course.

Sources

• Mapping data flows in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Mapping data flow transformation overview (Microsoft Learn) — Microsoft Learn
• Mapping data flow Debug Mode (Microsoft Learn) — Microsoft Learn
• Mapping data flows performance and tuning guide (Microsoft Learn) — Microsoft Learn
• Mapping data flow script (Microsoft Learn) — Microsoft Learn
• Chinook sample database (GitHub) — GitHub