AWS Interview Questions for Data Engineers

AWS questions in data engineering interviews rarely stop at "what is S3?". They start with a service, then probe whether you understand how it behaves under load, when it fails, what it costs, and how it fits with the rest of a pipeline. The questions below follow that pattern. Each has the core of a strong answer, grounded in AWS's own documentation, and the follow-up you should expect next. The last section has three scenario questions, where interviewers see whether you can put the pieces together.

If you are still learning the services, start with the AWS roadmap for data engineers, which covers them in build order. This page assumes you have used them.

Storage: Amazon S3

1. S3 is not a file system. What does that change for pipelines?

S3 is an object store. Every object is addressed by bucket and key, and "folders" are just key prefixes. Three consequences matter. First, there is no rename: moving a "folder" means copying every object and deleting the originals. Second, there are no atomic updates across keys; AWS states plainly that you cannot make the update of one key depend on another (S3 consistency model). Third, S3 does provide strong read-after-write consistency for PUTs and DELETEs in all Regions, so a successful write is immediately visible to reads and listings.

Follow-up to expect: "So how do you publish a new version of a table atomically?" Write to a new location and switch the catalog's pointer, or use a table format such as Apache Iceberg, which Athena supports, that commits changes atomically.

2. How do you choose storage classes for a data lake?

Match the class to the access pattern. S3 Standard suits data you query often. S3 Intelligent-Tiering suits unknown or changing patterns, because it moves objects between tiers for you. Standard-IA suits data read rarely but needed quickly. The Glacier classes suit archives. Automate the moves with lifecycle rules rather than by hand. The trap to mention: data in archival classes is not directly queryable, and Athena skips such objects rather than failing, so a query can silently return partial results. Our S3 guide covers the classes and that Athena behaviour in detail.

3. How do you upload very large files reliably?

With multipart upload. AWS recommends it for objects of 100 MB or larger (multipart upload overview). Parts upload in parallel for throughput, and a failed part can be retried without restarting the whole file. A detail worth adding: parts of an upload that is never completed or aborted keep incurring storage charges, so add a lifecycle rule that aborts incomplete multipart uploads after a few days.

4. How would you lay out a data lake in S3?

Separate zones by processing stage, such as raw/, clean/ and curated/, so reprocessing never touches the original data. Within each zone, use Hive-style partition prefixes such as dt=2026-09-24/ that match the most common query filters. Store curated data in a columnar, compressed format such as Parquet. And keep files reasonably large: Athena's performance guidance warns that many small files slow queries, because each file adds listing and request overhead, and very large numbers of requests can hit S3's per-prefix request limits.

Catalogue and query: Glue Data Catalog and Athena

5. What does the Glue Data Catalog do, and do you need crawlers?

The Data Catalog stores table definitions: schemas, formats, locations and partitions for data in S3. Athena, Glue jobs, EMR and Redshift Spectrum all read it. A crawler scans data and infers schemas and partitions, which is convenient for exploration. In production it is often safer to define tables explicitly, so that a malformed file can't silently change a schema, and add partitions through the catalog API or partition projection. Athena's guidance also recommends against MSCK REPAIR TABLE for routine partition maintenance, because it is slow and never removes old partitions.

6. How do you make Athena queries cheaper and faster?

Athena bills by the data each query scans, so every technique reduces bytes read. Store data in columnar formats (Parquet or ORC) so only the queried columns are read. Compress it. Partition by the columns most queries filter on, and filter on those partition columns. Convert raw data with a CTAS query or a Glue job. Then prove the improvement: compare "Data scanned" before and after, and use EXPLAIN to confirm which partitions a query will read.

Follow-up to expect: "Why might partitioning by hour be a mistake?" Too many small partitions create too many small files. Athena's own guidance is to optimise for the most common query and keep data sorted within partitions, rather than partition for rare queries.

7. What is partition projection, and when shouldn't you use it?

With partition projection, Athena calculates partition locations from rules in the table properties, instead of looking them up in the catalog. That speeds up queries on heavily partitioned tables and removes partition maintenance entirely. The caveats are what make an answer strong:

• When enabled, Athena ignores any partitions registered in the catalog for that table.
• If more than half the projected partitions are empty, AWS recommends traditional partitions instead.
• It only works for queries through Athena. Redshift Spectrum and EMR still read the catalog's partitions.

8. Why does Athena's guidance say partition keys should be strings?

Because Athena requires partition keys to be of type STRING to push partition filtering down into the Glue Data Catalog. With other types, a table with many partitions can perform worse. Cast date-like values in the query instead. This comes straight from Athena's performance-tuning documentation.

Transformation: Glue, EMR and Lambda

9. What is a DynamicFrame, and when would you use a DataFrame instead?

A DynamicFrame is Glue's wrapper around Spark data. Each record describes its own schema, and inconsistencies are kept as "choice" types, for example a field that is a number in some records and a string in others. You then resolve them explicitly with resolveChoice (DynamicFrame documentation). That suits messy, semi-structured sources. For clean data and anything that needs the full Spark SQL API, convert with toDF() and work with a DataFrame, converting back with fromDF() if you need a Glue sink.

10. How do Glue job bookmarks work, and where do they fall short?

Bookmarks record what a job has already processed, so the next run reads only new data. For S3 sources, Glue checks object modification times. The limits, from the bookmarks documentation, matter more than the feature:

• Bookmarks are disabled by default.
• They track sources, not targets. Resetting or rewinding a bookmark reprocesses input but does not clean up output already written.
• Changing a source path without changing its transformation_ctx makes Glue apply the old bookmark state to the new path, so files can be skipped.

11. Glue, EMR or Lambda: how do you choose?

The Lambda figures come from the Lambda quotas. A good answer ends with the trade-off: Glue buys convenience with less control, EMR the reverse. Our Glue vs EMR vs Lambda comparison goes further.

12. How do you make a Spark job on S3 safe to rerun?

Make the write idempotent: running the job twice for the same day must produce the same output as running it once. With partitioned Parquet output, overwrite only the partitions the job produces:

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

(daily_df.write
    .mode("overwrite")
    .partitionBy("dt")
    .parquet("s3://your-bucket/clean/orders/"))

The configuration line is the point of the answer. Spark's default is static mode, in which an overwrite first deletes every partition matching the write, and a plain partitioned overwrite matches them all. A rerun for one day would wipe every other day. In dynamic mode, Spark replaces only the partitions that receive data (Spark SQL configuration, spark.sql.sources.partitionOverwriteMode).

Warehouse: Amazon Redshift

13. How do distribution styles work in Redshift?

Rows are spread across compute slices using one of four distribution styles. EVEN spreads rows round-robin. KEY places rows with the same value of one column on the same slice, so two large tables distributed on their join key join without moving data. ALL copies a table to every node. AUTO, the default when you specify nothing, starts small tables as ALL and changes the style as the table grows. A nuance worth adding: Redshift's own documentation says there is little benefit in making small dimension tables ALL, because redistributing a small table at query time is cheap. For sort keys and the rest of Redshift tuning, see our Redshift interview questions.

14. What is the right way to load data into Redshift?

The COPY command from S3, loading many files in parallel, rather than INSERT statements, which the loading best practices discourage for bulk data. Split input into multiple files so every slice has work, compress them, and verify the load. For updates, COPY into a staging table and merge into the target in one transaction, so readers never see a half-loaded table.

15. Athena, Redshift or Redshift Spectrum: when would you use each?

Athena for ad hoc, serverless SQL over the lake, paying per query, with nothing to manage. Redshift for dashboards and repeated heavy queries that need consistent performance and concurrency, with data loaded into its managed storage. Redshift Spectrum for joining warehouse tables with data that stays in S3, such as recent data in Redshift and history in the lake. Frame the answer around access patterns and cost, not features.

Streaming

16. Kinesis Data Streams or Amazon Data Firehose?

Kinesis Data Streams is a durable, replayable stream that you read with your own consumers. Capacity comes in shards: each accepts up to 1,000 records or 1 MB per second of writes. Data is retained for 24 hours by default, extendable to 365 days (Kinesis concepts). Amazon Data Firehose is managed delivery. It buffers records by size or time and writes them to S3, Redshift and other destinations, with no consumer code (Firehose overview). Choose Data Streams when you need custom processing, several consumers or replay; choose Firehose when you just need the data landed.

17. How do you handle ordering and duplicates in a stream?

Ordering is only meaningful per partition key: records with the same key go to the same shard, and their sequence numbers generally increase over time. Choose the partition key so that events which must stay in order share a key, such as a user ID, while still spreading load across shards. For duplicates, assume records can arrive more than once, for example after retries, and design the consumer to be idempotent. Upserting on a unique event ID is the usual approach.

Orchestration and operations

18. Step Functions Standard or Express workflows?

Standard workflows run for up to a year with exactly-once execution, which suits steps that must not repeat, such as starting a cluster or loading a warehouse. Express workflows run for up to five minutes. Asynchronous Express workflows are at-least-once, so every step must be idempotent; they are built for high-volume, short event processing (workflow types). The type can't be changed after creation. Our Step Functions article shows both in a pipeline.

19. How do you know a pipeline is healthy?

A job that "succeeded" can still have produced wrong or empty data, so monitor three things. Execution: job failures, retries and duration, with CloudWatch alarms. Freshness: when did the newest partition land, and alert if it is late. Data quality: row counts compared with the previous run, null rates on key columns, and uniqueness of keys, checked before data is published. Mention who gets paged, and what the runbook says.

Security and governance

20. How do you secure a data lake on AWS?

In layers:

1. Identity. Each job gets its own IAM role with least-privilege access to specific prefixes. Roles issue temporary credentials, so no long-lived keys sit in code.
2. Network and exposure. Keep S3 Block Public Access on, which is the default for new buckets.
3. Encryption. Use SSE-KMS with customer-managed keys where you need control over the key policy. The key policy must also allow the job roles to use the key; for example, a multipart upload to a KMS-encrypted bucket needs kms:GenerateDataKey and kms:Decrypt on the key.
4. Fine-grained access. Use Lake Formation for table-, column- and row-level permissions across Athena, Glue and Redshift Spectrum.
5. Audit. Use CloudTrail to record who accessed what.

21. Why do jobs use IAM roles instead of IAM users?

A role has no permanent credentials. The service that assumes it, such as Glue, Lambda or EC2, receives temporary credentials that rotate automatically. Access keys for an IAM user are long-lived and tend to end up in configuration files and repositories. The point to make is that you would never embed access keys in a Glue script.

Scenario questions

22. Design a daily pipeline from a transactional database to a dashboard

A strong answer walks the data through each stage and justifies each choice:

1. Extract changed rows from the source database: incremental extraction by an updated-at column, or change data capture if deletes matter. Land them unmodified in raw/.
2. Transform with a Glue job that cleans, de-duplicates on the primary key, and writes partitioned Parquet to curated/ with dynamic partition overwrite, so reruns are safe.
3. Serve from Redshift, loaded with COPY through a staging table, if many users hit the dashboard. Athena is enough if usage is light.
4. Orchestrate with a Standard Step Functions workflow on a schedule, with retries on each step.
5. Monitor failures, freshness and row counts, and alert a named owner.
6. Secure with a role per job and KMS encryption, and state the monthly cost drivers: Glue job time, Redshift, and scanned data.

23. An Athena report became slow and expensive this month. What do you check?

Start from the evidence: compare "Data scanned" and runtime with last month. Common causes include:

• a new query or dashboard filter that no longer uses the partition columns;
• a change upstream that started writing many small files, or CSV instead of Parquet;
• partitions growing in number without projection or pruning;
• someone querying SELECT *.

Use EXPLAIN to see which partitions the query reads, and fix the data layout before reaching for bigger tools.

24. A Glue job reprocessed yesterday's data and created duplicates. Why, and how do you prevent it?

Likely causes: the job bookmark was reset or rewound, which reprocesses input without cleaning output; the source path changed while the transformation_ctx stayed the same; or bookmarks were never enabled and the job appends. The durable fix is not just "fix the bookmark". It is making the write idempotent, by overwriting the day's partition or merging on a unique key, so that any rerun is harmless. Mention adding a uniqueness check to the monitoring from question 19.

Preparing for the interview

Answer out loud, with the follow-up in mind: for every service, be ready to say what it costs, how it fails, and what you would use instead. Build one end-to-end pipeline yourself, because scenario questions are much easier to answer from memory of real decisions. Our list of AWS data engineering projects suggests builds you can use for this practice. The S3, Glue and Redshift topics above are also part of the syllabus of the GloryTecks Data Engineering course.