AWS Data Engineer Guide: Skills and Services

Most lists of AWS data engineer skills are really lists of AWS services: S3, Glue, Redshift, Kinesis, EMR, Lambda, and so on. Services matter, but they are not the whole job. A useful way to check what the job involves is to look at what data engineers who work on AWS actually use, and how that differs from data engineers who do not.

The 2025 Stack Overflow Developer Survey (Stack Overflow survey 2025) records respondents' roles and the databases, tools and languages they did extensive work with in the past year. The comparison below splits data engineers by whether they used AWS.

import pandas as pd

so = pd.read_csv("results.csv", low_memory=False)          # 2025 Stack Overflow Developer Survey
de = so[so.DevType.eq("Data engineer") & so.PlatformHaveWorkedWith.notna()].copy()
de["group"] = de.PlatformHaveWorkedWith.str.contains("Amazon Web Services", regex=False).map({True: "AWS", False: "no AWS"})
print("group sizes:", de.group.value_counts().to_dict())

def compare(column, items):
    rows = {}
    for item in items:
        answered = de.dropna(subset=[column])
        hit = answered[column].str.split(";").apply(lambda xs: item in xs)
        rows[item] = hit.groupby(answered.group).mean()
    return (pd.DataFrame(rows).T * 100).round().astype(int).astype(str) + "%"

print("\nT1 data stores used in the past year")
print(compare("DatabaseHaveWorkedWith", ["PostgreSQL", "Snowflake", "Amazon Redshift", "Dynamodb",
                                          "Databricks SQL", "BigQuery", "Microsoft SQL Server"]).to_string())
print("\nT2 engineering tools and other clouds")
print(compare("PlatformHaveWorkedWith", ["Docker", "Terraform", "Kubernetes", "Datadog",
                                          "Google Cloud", "Microsoft Azure"]).to_string())
print("\nT3 languages")
print(compare("LanguageHaveWorkedWith", ["Python", "SQL", "Bash/Shell (all shells)", "Java", "Scala"]).to_string())

aws = de[de.group.eq("AWS")].WorkExp.dropna()
print(f"\nT4 AWS-using data engineers: median {aws.median():.0f} years of professional experience,",
      f"{(aws <= 5).mean():.0%} with five years or fewer")

group sizes: {'AWS': 272, 'no AWS': 224}

T1 data stores used in the past year
group                 AWS no AWS
PostgreSQL            72%    56%
Snowflake             31%    15%
Amazon Redshift       21%     2%
Dynamodb              23%     3%
Databricks SQL        27%    23%
BigQuery              28%    27%
Microsoft SQL Server  34%    46%

T2 engineering tools and other clouds
group            AWS no AWS
Docker           74%    65%
Terraform        43%    21%
Kubernetes       36%    28%
Datadog          17%     5%
Google Cloud     33%    30%
Microsoft Azure  25%    49%

T3 languages
group                    AWS no AWS
Python                   97%    91%
SQL                      88%    81%
Bash/Shell (all shells)  66%    55%
Java                     29%    23%
Scala                    19%    11%

T4 AWS-using data engineers: median 10 years of professional experience, 24% with five years or fewer

Four things the data says

1. The warehouse is often not an AWS service. Among data engineers who used AWS, 21% had used Amazon Redshift, while 31% had used Snowflake and 72% PostgreSQL. Working on AWS does not mean everything runs on AWS-native services. Teams often pair S3 and AWS compute with Snowflake or Databricks. Learn warehouse concepts, such as modelling, partitioning and cost, that transfer across products, rather than one product's console.

2. Infrastructure as code is part of the job. Terraform use was twice as common among AWS-using data engineers: 43% against 21%. Docker (74%) and Kubernetes (36%) were also more common. On AWS, data engineers often create and change the buckets, roles, jobs and schedules their pipelines need, and doing that by hand in the console does not scale or survive review.

3. The shell and a JVM language still matter. Python and SQL were near universal in both groups. But Bash was used by two-thirds of AWS data engineers, and Scala by almost one in five, reflecting Spark on EMR and Glue. Comfort on the command line, with the AWS CLI and scripts, is assumed.

4. Operational databases come with the territory. DynamoDB, used by 23% of the AWS group against 3% of others, often sits upstream of the pipeline as a source, and its export and stream features shape how data gets out.

The AWS group was also experienced: median ten years of professional work, with a quarter at five years or fewer. AWS's own data engineer certification recommends about two to three years of data engineering and one to two years of hands-on AWS before the exam (AWS: Data Engineer Associate). It is a role people usually grow into.

Reading the numbers with care

These are a few hundred self-selected respondents, not a sample of job adverts. The survey asks what people did extensive work with in the past year, which tells you what the work involves, not what an employer will list as a requirement. Shares of this size can move by several points from year to year. Treat the gaps between the groups as the signal, such as Terraform at twice the rate or Redshift well behind Snowflake, rather than the exact percentages. Before you commit months to one tool, check the job adverts in your own city against these patterns.

The skills profile, in priority order

1. SQL and Python, to a level where you can write window functions and test a transformation.
2. Data modelling and warehouse concepts that transfer between Redshift, Snowflake and Athena.
3. Spark, in PySpark at least, for the jobs that outgrow a single machine.
4. Infrastructure as code, Terraform or AWS CloudFormation, plus Docker basics.
5. AWS security and networking basics: IAM roles and policies, encryption, VPC endpoints.
6. Orchestration and monitoring, so pipelines retry, alert and can be rerun.

The services, by task

You do not need every service. Learn the ones that serve the tasks you will actually do.

For the choice of transformation engine, see Glue vs EMR vs Lambda; for building the lake itself, a data lake on S3 and Glue; for orchestration, Step Functions; and for permissions, IAM explained.

How to use this when learning

Follow the order in the AWS roadmap for data engineers, but add two things that service lists leave out. First, write every resource for your practice project in Terraform or CloudFormation from the start, so you can destroy and recreate it, which also keeps your bill near zero. Second, connect at least one non-AWS system: a PostgreSQL source, or a Snowflake or Databricks trial as the warehouse. That reflects how the survey says real AWS data stacks look.

When you are ready for a certificate, the DEA-C01 exam guide covers the domains and a study plan.

AWS data engineering, from SQL and Spark to pipelines on S3 and Glue, is taught in the Data Engineering course.

Sources

• 2025 Stack Overflow Developer Survey — Stack Overflow
• AWS Certified Data Engineer - Associate (AWS) — Amazon Web Services