AWS Roadmap 2026 for Data Engineers

AWS describes itself as offering over 200 fully featured services, and a data engineer uses perhaps a dozen of them regularly. The hard part of learning AWS is not any single service. It is knowing which dozen matter, and in which order they start to make sense. This roadmap orders them the way data actually moves through a pipeline on AWS: land it, catalogue it, query it, transform it, serve it, schedule it, and secure it. Each stage ends with something you build, so you always know whether you are ready to move on.

It assumes you are aiming for a data engineering role and already write basic SQL and Python. If you are still deciding between data engineering and other data roles, or want the tool-agnostic version of this path, start with the data engineering roadmap and come back here when you pick AWS.

Before you open the console

Three skills do more for your AWS progress than any service tutorial:

• SQL beyond SELECT. Joins, GROUP BY, window functions and CTEs. Athena and Redshift are both SQL engines, so much of your AWS work is SQL.
• Python for scripts. Reading and writing files, working with JSON, functions, and error handling. Glue jobs and Lambda functions are Python programs with an AWS entry point.
• The command line and Git. The AWS CLI, shell scripts and version-controlled code are how real pipelines are built and reviewed. The official exam guide for the data engineering certification lists Git as expected general knowledge.

If one of these is shaky, fix it first. Everything below gets harder, not easier, when you are also fighting the language.

Set up an account that cannot surprise you

A new AWS account today starts on the Free Tier's free plan. AWS gives $100 in credits at sign-up and up to $100 more as you try key services. The free-plan account closes on its own after six months, or when the credits run out, unless you upgrade it to a paid plan. The details are on the AWS Free Tier page. Read them before you create anything, because the rules have changed over the years and older tutorials describe a different model.

Before your first real resource:

1. Stop using the root user. Secure it with MFA and create a separate identity for daily work. AWS's own guidance is to use the root user only for the few tasks that require it.
2. Create a budget with an email alert. AWS Budgets notifies you when actual or forecast spend crosses a threshold you choose. Set it low.
3. Pick one Region and stay in it. Resources you forget in another Region still cost money and are easy to miss.
4. Tear down as you go. Delete clusters, endpoints and crawlers when a stage is finished. Storage left in S3 is cheap; forgotten compute is not.

The roadmap at a glance

Stage 1: Store data and control who touches it

Amazon S3 is the centre of almost every AWS data platform, so learn it properly rather than as "a place for files". Understand buckets, keys and prefixes, the storage classes and lifecycle rules, and S3's consistency model: reads after a successful write or delete return the new state, in all Regions, as the S3 documentation describes. Our S3 guide covers storage classes and lifecycle policies in detail.

Learn IAM at the same time, not later. Almost every service in the rest of this roadmap acts through an IAM role, and many "it doesn't work" moments on AWS turn out to be permission errors. Know the difference between users and roles, identity-based and resource-based policies, and how an explicit deny wins. The IAM guide walks through each.

Build: one bucket with raw/, clean/ and curated/ prefixes, a lifecycle rule on raw/, and a role whose policy allows reading raw/ and writing clean/, nothing more. Test it by trying an action the policy should refuse.

Stage 2: Catalogue the lake and query it with SQL

Files in S3 become tables once their schema is registered in the AWS Glue Data Catalog. A crawler can infer the schema, or you can define it yourself. Athena then queries those tables in place with SQL, and it bills by the amount of data each query scans (Athena pricing). That billing model is the reason columnar formats and partitioning matter: a query that reads two columns of one month's partition scans a fraction of the bytes.

The fastest way to feel this is to convert a CSV table into partitioned Parquet with one CREATE TABLE AS SELECT (CTAS) query. Its WITH clause sets the output format to Parquet, an external_location under your bucket's curated/ prefix and a partitioned_by array naming the partition column, such as a trip_month derived from the pickup timestamp. The body is an ordinary SELECT from the CSV table.

Two rules from the Athena CTAS documentation are easy to miss. Partition columns must come last in the SELECT list. And a single CTAS query can write at most 100 partitions, so a larger backfill has to be split into a CTAS followed by INSERT INTO statements.

Build: run the same aggregate query against the CSV table and the Parquet table. Record the "Data scanned" figure Athena shows for each, and write two sentences explaining the difference.

Stage 3: Transform data with Glue jobs and Lambda

A Glue ETL job is a Spark program that AWS runs for you. You write it in PySpark, with Glue's DynamicFrame wrappers or plain Spark DataFrames, and pay for the compute while it runs. Learn one feature early, because it separates toy jobs from production ones: job bookmarks. With bookmarks enabled, Glue records which input it has already processed and reads only new data on the next run. According to the Glue documentation, bookmarks are disabled by default. They track sources, not targets, so resetting a bookmark does not clean up output you already wrote. That is why an incremental job also needs an idempotent write strategy, such as overwriting a partition instead of appending to it.

Lambda handles the small, event-driven work: reacting to a file landing in S3, validating it, and starting the heavier job. Know its limits before you design around it. A single invocation runs for at most 15 minutes, and memory ranges from 128 MB to 10,240 MB (Lambda quotas). Anything bigger belongs in Glue.

Build: a Glue job that reads new files from raw/, cleans them, and writes Parquet partitions to clean/. Run it twice with no new input, and prove the second run adds nothing. Our Glue tutorial covers the job setup.

Stage 4: Serve data from a warehouse

Athena is enough for exploration and many reports. You reach for Amazon Redshift when many users run the same heavy queries repeatedly and need consistent performance. Redshift Serverless removes cluster sizing from your first steps, so you can concentrate on the two skills that transfer everywhere:

• Loading with COPY. Redshift's loading guidance is to use the COPY command, loading many files in parallel, rather than row-by-row INSERT statements (loading best practices).
• Table design. Redshift distributes rows across compute slices using AUTO, EVEN, KEY or ALL distribution. If you don't specify one, it uses AUTO and adjusts as the table grows (distribution styles). Learn why co-locating two large tables on their join key avoids moving data at query time.

Build: model a small star schema (one fact table, two or three dimensions), load it from curated/ with COPY, and write the query a manager would actually ask for. If you can't explain why AUTO distribution is or isn't good enough for each table, you have more to learn here.

Stage 5: Orchestrate, retry and monitor

A pipeline that only works when you press Run is a script. Orchestration makes it a pipeline. AWS Step Functions is the native option, and the choice between its two workflow types is a real design decision. Standard workflows run for up to a year with exactly-once execution, which suits steps that must not repeat. Express workflows run for up to five minutes. Asynchronous Express workflows are at-least-once, so the steps must be idempotent (workflow types). Teams already using Apache Airflow can run it on Amazon MWAA instead; the concepts carry over.

Build: a daily Step Functions workflow, triggered by an EventBridge schedule, that runs the Glue job and then the Redshift load. Add a retry on the Glue step and a CloudWatch alarm on failure. Then break it deliberately and check that the alarm fires.

Stage 6: Add streaming when a use case needs it

Many data engineering jobs never touch streaming. Learn it once batch is comfortable, not before. The two services to understand are:

• Kinesis Data Streams, where data is split across shards. Each shard accepts up to 1,000 records or 1 MB per second of writes. Records are kept for 24 hours by default, extendable to 365 days at extra cost (Kinesis concepts).
• Amazon Data Firehose (formerly Kinesis Data Firehose), which buffers incoming records by size or time and delivers them to S3, Redshift and other destinations without you writing a consumer (Firehose overview).

Build: send simulated click events through Firehose into partitioned S3 prefixes, then query the last hour of events with Athena.

Stage 7: Secure and govern the data

Security is not a final polish step, but it is easier to learn once you have a working pipeline to secure. Cover encryption at rest with AWS KMS keys, fine-grained table and column permissions with AWS Lake Formation, and auditing with CloudTrail. In the data engineering certification, security and governance is a scored domain of its own.

Where the AWS certification fits

The AWS Certified Data Engineer – Associate (DEA-C01) exam guide is a useful free checklist of what AWS considers core data engineering skill. Its four domains and their share of scored content are:

The exam has 50 scored questions plus 15 unscored ones, and the passing mark is 720 on a 100–1,000 scale. Notice who it is written for: the guide describes a target candidate with the equivalent of 2–3 years of data engineering experience and 1–2 years of hands-on AWS work. For a beginner, that makes the guide more useful as a syllabus than as a first goal. Build the seven stages above, then use the DEA-C01 exam guide article to close the gaps the domains reveal.

What to leave for later

Skipping things is part of a roadmap. These are worth learning eventually, but not in your first pass:

• Amazon EMR, until you need Spark configurations that Glue does not expose.
• Amazon MSK (managed Kafka), unless a target employer runs Kafka.
• AWS DMS, until you have a database migration or change-data-capture problem to solve.
• Deep networking (VPC design, PrivateLink). Learn enough to understand why a Glue job can't reach a database in a private subnet, and stop there.

Put it together in one project

The strongest proof of this roadmap is a single pipeline that uses every stage: files land in S3, a Glue job cleans them incrementally, Athena serves exploration, Redshift serves a dashboard, and Step Functions runs it daily with alerts. Write a README explaining each design choice and what it cost to run. Our list of AWS data engineering projects has variations on this build, and the AWS interview questions for data engineers show how interviewers probe the same decisions.

If you want to work through this stack in a structured course, the GloryTecks Data Engineering course covers S3, Glue and Redshift alongside Spark and Airflow, with project work.