Building a Serverless Analytics Stack on AWS

Ten years ago an analytics platform began with a cluster: machines to size, software to install, and a bill that arrived whether or not anyone ran a query. On AWS today a complete platform can be assembled from services that have no servers for you to manage and that charge mainly when used. For a small team, or for a workload that is busy for an hour a day and idle for the rest, that changes what is affordable.

"Serverless" is used loosely, so here is the working definition for this article: no infrastructure for you to provision or patch, capacity that scales with the work, and billing tied to usage. The services below meet it. The design is a reference, not a prescription. It is the default to start from and adjust.

The stack at a glance

Storage: everything rests on S3

All data lives in S3, organised into a raw zone, files exactly as received, and a curated zone, cleaned Parquet partitioned for queries. Every other service in the stack reads from or writes to these prefixes, which is what lets compute appear and disappear without data moving. Building a data lake on AWS with S3 and Glue covers the layout.

Ingest: files and streams

Batch sources drop files into the raw zone, directly or through a transfer tool. Streaming sources go through Amazon Data Firehose, a fully managed delivery service that buffers events and writes them to S3 with no application to run (Firehose docs: what is Amazon Data Firehose). Either way the result is the same: objects in a raw prefix, with their arrival announced by an S3 event.

Catalog: the shared index

The Glue Data Catalog holds a definition for each table: schema, format, location, partitions. It is the piece that makes the stack coherent, because Athena, Glue jobs and BI tools all resolve table names through it. Populate it with crawlers for discovery and with explicit definitions for curated tables; the Glue Data Catalog and crawlers explains both.

Transform: three engines, by size

AWS Glue is a serverless data integration service with Spark-based ETL (Glue docs: what is AWS Glue), and it is the default engine. It is not the only one, and matching the engine to the job keeps costs down:

• Lambda for small, per-file work: validate a file on arrival, convert a small CSV. Invocations are limited to 15 minutes (Lambda docs: what is AWS Lambda).
• Athena SQL for set-based transformation: CREATE TABLE AS and INSERT INTO statements that write new Parquet tables from existing ones. For a team strong in SQL this can cover most modelling.
• Glue Spark jobs for large or complex transformation.

Glue vs EMR vs Lambda goes through the choice, and building an ETL pipeline on AWS shows the structure a job should have whichever engine runs it.

Orchestrate: schedules, events, workflows

Three mechanisms start work:

• Time. EventBridge Scheduler is a serverless scheduler that invokes AWS services on a schedule (EventBridge Scheduler docs).
• Arrival. An S3 event triggers a Lambda function when a file lands; see event-driven pipelines with Lambda and S3.
• Dependency. Step Functions runs the multi-step pipeline: transform, check, load, notify, with retries around each step (Step Functions docs: what is Step Functions).

A typical flow: the schedule or the event starts a state machine; the state machine runs a Glue job, then an Athena query that validates the output, and sends an alert on failure.

Query: Athena

Athena is a serverless interactive query service for analysing data in S3 with standard SQL, with payment only for the queries run (Athena docs: what is Athena). Analysts query the curated zone directly. Amazon Athena explained covers how it works and what drives its cost.

Visualise

A BI tool connects to Athena and presents dashboards. AWS's own is Quick Sight, the visualisation and business intelligence feature of Amazon Quick (Quick docs: what is Amazon Quick); other tools connect through Athena's JDBC and ODBC drivers. Dashboards issue queries, and queries cost money, so dashboards that refresh often should read small, pre-aggregated tables, not raw detail.

Govern

• IAM roles per pipeline and per team, with least privilege on buckets and catalog resources.
• Lake Formation when access must be controlled by table, column or row.
• Encryption at rest in S3 by default, with KMS keys where key control is required.
• Athena workgroups to separate teams and enforce settings.

What you pay for

The appeal of the design is that most meters stop when work stops.

Two meters never reach zero: storage, which grows unless lifecycle rules are in place, and whatever your BI tool charges. One meter is unpredictable: Athena's, because it depends on what people ask. Guard it from the first day with a per-query data limit on each workgroup, which cancels any query that scans more than the set amount (Athena docs: data usage controls), and by storing curated data as partitioned Parquet.

Where the design stops fitting

Serverless is a starting point, and it has edges:

• High-concurrency dashboards. Many users refreshing many charts all day is a warehouse workload. Per-query billing becomes expensive and response times vary. This is the point to add Redshift, whose serverless option keeps the operational model.
• Constant heavy processing. A job that runs around the clock gains nothing from pay-per-use; reserved or provisioned capacity is usually cheaper.
• Low-latency serving. Athena answers in seconds, not milliseconds. Applications that need instant lookups need a database.
• Fine control of Spark. Specific versions, custom libraries on nodes or other frameworks point to EMR.
• Frequent row-level updates. Plain Parquet in S3 is append and replace. Use a table format such as Iceberg, or a warehouse.

None of these is a reason to avoid the design at the start. They are the signs that tell you when to add a component.

A build order

1. Buckets, zones and IAM roles.
2. One dataset landed raw, catalogued, and queryable in Athena.
3. One transformation to curated Parquet, run by hand.
4. A schedule or event to run it, then Step Functions around it with a failure alert.
5. A workgroup with a data limit, and a lifecycle rule on raw and results prefixes.
6. A first dashboard on a small aggregated table.
7. More datasets, following the same path.

By step 2 there is something to show. Every later step adds one service to a pipeline that already works, which is the least risky way to learn eight services at once.

Data lake implementation and cloud data pipelines on AWS are covered in the Data Engineering course.

Sources

• What is Amazon Athena? (Amazon Athena User Guide) — AWS Documentation
• What is AWS Glue? (AWS Glue User Guide) — AWS Documentation
• What is AWS Lambda? (AWS Lambda Developer Guide) — AWS Documentation
• What is Amazon Data Firehose? (Amazon Data Firehose Developer Guide) — AWS Documentation
• What is Step Functions? (AWS Step Functions Developer Guide) — AWS Documentation
• What is Amazon EventBridge Scheduler? (EventBridge Scheduler User Guide) — AWS Documentation
• What is Amazon Quick? (Amazon Quick User Guide) — AWS Documentation
• Configure per-query and per-workgroup data usage controls (Amazon Athena User Guide) — AWS Documentation