Integration Runtimes in ADF: Azure, Self-Hosted and SSIS

A pipeline is a set of instructions. Something has to carry them out: open the connection, read the rows, write them somewhere else. In Azure Data Factory that something is the integration runtime, or IR. Microsoft defines it as the compute infrastructure that provides ADF's data integration capabilities across different network environments (Microsoft Learn: integration runtime).

Beginners can ignore the IR for a while, because a default one exists and works for cloud-to-cloud copies. The first time a source sits behind a firewall, it becomes the most important object in the factory.

What an integration runtime does

The documentation lists four capabilities:

• Data flow: run mapping data flows on managed Spark compute.
• Data movement: copy data between stores, with connectors, format conversion and column mapping.
• Activity dispatch: send transformation activities to other compute, such as Azure Databricks, a SQL database running a stored procedure, or HDInsight, and monitor them.
• SSIS package execution: run SQL Server Integration Services packages.

A linked service says which IR to use through its connectVia property. If it says nothing, the default Azure IR is used (Microsoft Learn: linked services).

The three types

One limitation to note: Synapse pipelines support only the Azure and self-hosted types.

Azure integration runtime

The Azure IR is the default. It is fully managed and serverless: no provisioning, patching or capacity planning, and you pay for what you use. It copies between stores with publicly reachable endpoints, runs data flows, and dispatches activities.

Location. The factory has a region, where its metadata is stored. The IR's location is where the work actually happens, and it can differ. By default the Azure IR autoresolves: for a copy, ADF tries to detect the sink's region and runs the copy there, or in the closest region in the same geography; if it cannot detect the region, it uses the factory's region.

That default is sensible for cost and speed, and wrong for some compliance rules. If data must not leave a geography, Microsoft's advice is to create an Azure IR pinned to a region and point the linked service at it with connectVia.

Private networks. With a managed virtual network enabled, the Azure IR can reach data stores through private endpoints without a machine of your own. For Azure-hosted stores locked to private access, that is often simpler than a self-hosted IR.

Self-hosted integration runtime

When a data store is not reachable from Azure's public network, an on-premises SQL Server, a file share, a database on a private virtual machine, the compute has to sit inside the network. The self-hosted IR is software you install on a machine there. It reads from the source locally and sends data out to the destination, so no inbound connection from the internet is needed.

Requirements and rules from the documentation (Microsoft Learn: self-hosted integration runtime):

• A 64-bit Windows machine. Installation on a domain controller is not supported.
• One instance per machine.
• The machine must stay awake. If the host hibernates, the IR stops responding. Configure the power plan before installing.
• Put it near the source, but not on it. Microsoft recommends a machine different from the one hosting the data source, so the two do not compete for resources.
• One IR can serve many sources, and can be shared with other data factories in the same Microsoft Entra tenant.
• Use it for IaaS too. A data store on an Azure virtual machine is treated like an on-premises source.

High availability. A self-hosted IR can have up to four nodes. More than one node removes the single point of failure and adds capacity for concurrent work. A production pipeline that depends on one laptop-class machine under someone's desk is a familiar and avoidable outage.

The rule that surprises people. If a self-hosted IR is used for either the source or the sink of a Copy activity, both ends must be reachable from the machine hosting it. The copy runs on that machine. Copying from on-premises SQL Server to a storage account therefore needs the IR machine to reach the storage account as well, and the machine's CPU, memory and network become the limit on throughput.

Copying data from on-premises SQL to Azure works through that scenario.

Azure-SSIS integration runtime

Organisations with years of SQL Server Integration Services packages can move them to the cloud without rewriting them. The Azure-SSIS IR is a fully managed cluster of Azure virtual machines dedicated to running SSIS packages. You choose the node size and the number of nodes, and you can stop it when it is not needed to control cost.

It is a migration tool. It exists to lift and shift, not as the way to build new pipelines; ADF vs SSIS covers the decision.

Which runtime does an activity use?

Each linked service can name an IR, and a Copy activity has two linked services. When they differ, precedence applies:

1. Self-hosted wins. If either side uses a self-hosted IR, the copy runs there.
2. Otherwise an Azure IR in a managed virtual network takes precedence over the global Azure IR.
3. Otherwise the Azure IR is used, in the pinned region or autoresolved.

This is why a copy between two cloud stores can unexpectedly run on an on-premises machine: one of the linked services was created with the self-hosted IR selected.

Choosing

Practical points

• Name runtimes for what they reach, such as the network or site, not for a project.
• Monitor self-hosted nodes: CPU, memory, available concurrent jobs, and whether each node is online. A degraded node is a common cause of slow or queued copies.
• Keep the self-hosted software updated. Auto-update exists; make sure it is not blocked.
• Stage data when transforming on-premises data. Data flows cannot read through a self-hosted IR, so copy to cloud storage first, then transform.
• Stop what bills while idle: the Azure-SSIS IR, and data flow debug sessions.

For how linked services select a runtime, see linked services in Azure Data Factory; for how runtime choice affects copy throughput, ADF performance tuning.

Azure Data Factory and cloud data integration are covered in the Data Engineering course.

Sources

• Integration runtime in Azure Data Factory (Microsoft Learn) — Microsoft Learn
• Create and configure a self-hosted integration runtime (Microsoft Learn) — Microsoft Learn
• Copy activity in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn
• Linked services in Azure Data Factory and Azure Synapse Analytics (Microsoft Learn) — Microsoft Learn