AWS Step Functions: Orchestrating Serverless Workflows

A data pipeline is rarely one job. It is a job, then a check, then a load, with a notification if anything breaks, and a rule about what to retry. Written as code inside a Lambda function, that logic becomes a tangle of waits and try blocks, and nobody can see where a run stopped.

The service takes the coordination out of your code. You describe the workflow as a state machine; Step Functions runs it, waits for each step, retries failures, and records what happened. AWS lists creating data and machine learning pipelines among its main uses (Step Functions docs: what is Step Functions).

The vocabulary

• State machine, or workflow: the definition, a series of steps.
• State: one step. Each has a type and names the state that follows it.
• Execution: one run of the workflow, with its own input, history and outcome.
• Amazon States Language (ASL): the JSON format in which workflows are written.

Workflows can be drawn in the console's Workflow Studio or written as JSON. Either way the result is an ASL document, which belongs in version control.

The states

A Task state does work: it calls another AWS service. The other seven are flow states that direct the workflow (Step Functions docs: workflow states):

Map deserves a note for data work. It runs the same steps for every item in a dataset, in parallel, and the input can be a JSON array, a list of S3 objects or a CSV file (Step Functions docs: Map state). "Process each of these 200 files" is one Map state.

How a task talks to a service

More than two hundred AWS services can be called from a task. What matters is the integration pattern, which decides when the workflow moves on (Step Functions docs: integration patterns):

The second pattern is why the service suits pipelines. Starting a Glue job returns immediately, long before the job finishes. With .sync, the workflow waits for the real outcome and proceeds, or fails, accordingly. You write no polling loop.

A pipeline in ASL

The definition below runs a Glue job, then an Athena query against its output, and sends a notification if either fails. It is written from the documentation and has not been deployed.

{
  "Comment": "Nightly trips pipeline",
  "QueryLanguage": "JSONata",
  "StartAt": "TransformTrips",
  "States": {
    "TransformTrips": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Arguments": { "JobName": "trips-to-curated" },
      "Retry": [
        { "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 60, "MaxAttempts": 2, "BackoffRate": 2.0 }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "Next": "NotifyFailure" }
      ],
      "Next": "CheckRowCount"
    },
    "CheckRowCount": {
      "Type": "Task",
      "Resource": "arn:aws:states:::athena:startQueryExecution.sync",
      "Arguments": {
        "QueryString": "SELECT count(*) FROM lake.daily_borough WHERE pickup_date = current_date - interval '1' day",
        "WorkGroup": "pipelines"
      },
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "Next": "NotifyFailure" }
      ],
      "End": true
    },
    "NotifyFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Arguments": {
        "TopicArn": "arn:aws:sns:us-east-1:123456789012:pipeline-alerts",
        "Message": "Nightly trips pipeline failed"
      },
      "Next": "Failed"
    },
    "Failed": { "Type": "Fail", "Error": "PipelineFailed" }
  }
}

Reading it top to bottom:

• The first task starts the Glue job and waits for it. If the task fails, it is retried twice, after 60 and then 120 seconds.
• Its catch block sends any error that survives the retries to NotifyFailure instead of ending the execution at that point.
• The second task runs an Athena query and waits for it, a simple check on the job's output.
• The failure branch publishes to an SNS topic, then moves to a Fail state so the execution is recorded as failed.

Without the catch block, the behaviour is the default the guide describes: when a state reports an error, the entire execution fails.

Retries and catches

Error handling is the main reason to use an orchestrator, so the fields are worth knowing (Step Functions docs: error handling):

Two cautions. Retrying is only safe when the task is idempotent: a Glue job that appends rows will append them again. And retry transient errors, not everything; a job that fails because of bad data will fail again on every retry and delay the alert. Idempotency in data pipelines explains how to make a step safe to repeat.

Standard or Express

You choose a type when creating the workflow, and it cannot be changed afterwards (Step Functions docs: choosing workflow type).

Data pipelines are almost always Standard: jobs run longer than five minutes, need the .sync pattern, and benefit from the stored execution history.

What it costs

Standard workflows are charged by state transition, counted each time a step is executed, retries included, with 4,000 free transitions a month (AWS: Step Functions pricing). A nightly pipeline of a dozen steps is a few hundred transitions a month. The pattern to watch is a loop: a workflow that polls every few seconds, or a Map over millions of items in a Standard workflow, generates transitions quickly. The services the workflow calls are billed separately.

Where it fits in a data platform

This is a general orchestrator for AWS services. It is strong when a pipeline spans several of them, Lambda, Glue, Athena, EMR, ECS, and when event-driven starts and per-step retries matter. It is less convenient for hundreds of interdependent tables with backfills and data-aware scheduling, where a tool built for data pipelines fits better; the Airflow tutorial shows that model, and Glue workflows vs Step Functions compares it with Glue's built-in option.

A common arrangement: an S3 event or a schedule in EventBridge starts the state machine; small checks run as Lambda functions; heavy work runs as Glue or EMR tasks. Event-driven pipelines with Lambda and S3 covers the trigger.

Good practice

• Keep business logic out of the state machine. It coordinates; jobs and functions compute.
• Pass references, not data. Hand S3 paths between states, not file contents.
• Give every task a timeout, so a hung job does not hold an execution open.
• Catch at the task, notify, then fail, so failures are both reported and recorded.
• Name states for what they do. The names are what you read in the execution history at 3 am.

Workflow orchestration and AWS data services are covered in the Data Engineering course.

Sources

• What is Step Functions? (Step Functions Developer Guide) — AWS Documentation
• Discovering workflow states to use in Step Functions (Step Functions Developer Guide) — AWS Documentation
• Discover service integration patterns in Step Functions (Step Functions Developer Guide) — AWS Documentation
• Handling errors in Step Functions workflows (Step Functions Developer Guide) — AWS Documentation
• Choosing workflow type in Step Functions (Step Functions Developer Guide) — AWS Documentation
• Map workflow state (Step Functions Developer Guide) — AWS Documentation
• AWS Step Functions pricing (Amazon Web Services) — Amazon Web Services