AWS Data Engineering Projects for Your Portfolio

An AWS project in a portfolio has to do two things at once. It has to show that you can use the services, and it has to show that you can build a pipeline that is correct, which is a different skill and the one interviewers probe. A repository that provisions ten services and moves a toy CSV shows neither.

The five projects below use real public data, each centres on one AWS pattern, and each is designed to run on small data at low cost. New accounts can start on a Free plan with credits (AWS docs: Free Tier); check which services your plan includes. For projects that are not tied to a cloud, see data engineering projects for your portfolio. Two of these done properly are worth more than five sketched.

Before you start

• Set a budget alert and use the Free plan. The AWS Free Tier explains the plans and what burns credits.
• Define everything as code. CloudFormation, the CDK or Terraform. A reviewer should be able to read your infrastructure, and you should be able to delete it with one command.
• Develop locally first. Write transformation logic as plain Python or PySpark and test it on a sample file before running it on AWS.
• Use small data. One month, not ten years. Design for scale; run at small size.

Project 1: a serverless data lake with measured query cost

Data: New York yellow taxi trips, published monthly as Parquet with a zone lookup table (NYC TLC: trip record data).

Services: S3, Glue Data Catalog, Athena.

Build: land a few months of files in a raw zone. Convert one month to CSV as well, so you have a bad format to compare with. Define tables in the catalog, first over the CSV, then over partitioned Parquet in a curated zone. Run the same three queries against both and record "Data scanned" from Athena's query statistics for each.

It proves: lake layout, cataloguing, partitioning and file formats, and that you understand what drives cost. Athena bills by bytes scanned (AWS: Athena pricing), so a table of before-and-after figures is evidence, not assertion.

Cost caution: set a per-query data limit on the workgroup before running anything.

Stretch: partition projection, and a lifecycle rule on the results bucket. Building a data lake on AWS covers the layout.

Project 2: event-driven ingestion that survives bad input

Data: the U.S. Geological Survey's earthquake feed, GeoJSON updated every minute (USGS: GeoJSON feed).

Services: EventBridge Scheduler, Lambda, S3, SQS for failures.

Build: a scheduled function fetches the feed and writes the raw JSON to S3 under the fetch time. An S3 event triggers a second function that validates the file, flattens it, and writes one cleaned object per raw object to a second bucket. Send failed events to a queue and alarm on it.

It proves: event-driven design and its hazards. Events are revised after first publication, so the same earthquake arrives many times; your handler must be idempotent and your downstream table must upsert on the event ID. Show the handler being called twice with the same event and producing one output.

Cost caution: never write output to the bucket that triggers the function.

Stretch: a Step Functions workflow replacing the second function once files grow. Event-driven pipelines with Lambda and S3 builds the core pattern.

Project 3: an orchestrated batch pipeline with tests

Data: taxi trips for the same month in three different years. The schema drifts between them: a column is renamed, types change, a new column appears.

Services: S3, Glue job, Glue Data Catalog, Step Functions, Athena, SNS, CloudWatch.

Build: a PySpark job that takes a month as a parameter, applies named validation rules and counts what each rejects, handles the schema differences explicitly, and overwrites that month's partition. A state machine runs the job, then an Athena query that reconciles row counts, and publishes to SNS on failure (Step Functions docs: what is Step Functions). Publish rows loaded and rejected share as custom metrics, with an alarm.

It proves: the heart of the job: parameterised runs, idempotent loads, data quality, schema handling, orchestration and monitoring. Run it twice for one month and show identical output.

Cost caution: set the Glue job to two workers and give it a timeout. The default allocation is far more than this needs.

Stretch: unit tests for the transform in CI, and a backfill across all three years. Building an ETL pipeline on AWS shows the structure.

Project 4: a streaming path with honest file counts

Data: Wikimedia's EventStreams service, which exposes recent changes to Wikimedia projects as a continuous stream (Wikimedia: EventStreams). It is intended for small-scale external tools, so consume politely and briefly.

Services: a small producer script, Kinesis Data Streams, Amazon Data Firehose, S3, Athena.

Build: the producer reads the stream and puts records with a chosen partition key. Firehose delivers to S3 on a buffer interval. Measure how many objects an hour of data produces at two interval settings, then write a compaction job that merges a day's small objects into a few Parquet files and deduplicates on the event ID.

It proves: shards and partition keys, at-least-once delivery, and the small-files problem that streaming creates, with your own measurements.

Cost caution: a provisioned stream bills per shard-hour while it exists. Delete the stream and the Firehose stream when you stop for the day.

Stretch: compare two partition keys for shard balance. Streaming data with Kinesis to S3 and Redshift covers the path.

Project 5: a warehouse with a reasoned design

Data: the curated taxi tables from Project 1 or 3.

Services: Redshift Serverless, S3, an IAM role.

Build: design a star schema: a trip fact with date, zone and payment dimensions. Load it with COPY from Parquet through a staging table. Declare keys and then test them, since Redshift does not enforce them. Read a query plan, try a distribution and sort key against AUTO, and record what changed. Keep one older month in S3 as an external table and query across both.

It proves: dimensional modelling, bulk loading, the gap between declared and enforced constraints, and performance reasoning from evidence.

Cost caution: set the base capacity to the minimum and a usage limit before the first query. Do not create a provisioned cluster for practice.

Stretch: a materialized view for a dashboard query. Loading data into Redshift with COPY covers the load.

Presenting the work

A reviewer gives a repository a few minutes. Make those minutes count:

• A README that opens with the question the project answers, then an architecture diagram, then how to deploy and how to delete.
• Evidence. Screenshots of a state machine run, an Athena "Data scanned" comparison, a reconciliation report, an alarm in the ALARM state after you fed it bad data.
• A decisions section. Why this partition key, why Glue and not Lambda, why this distribution key, and what you tried that did not work.
• A cost section. What the project cost to run and which settings kept it low. It is exactly what an employer worries about.
• Infrastructure as code in the repository, with no credentials, account numbers or bucket names that matter.
• Limits. What breaks at a hundred times the volume, and what you would change.

Then be ready to explain every line. The project's real purpose is the conversation it starts.

Mistakes to avoid

• Console-only projects. If it cannot be recreated from the repository, it cannot be reviewed.
• Following a tutorial exactly. Change the dataset, add a failure case, measure something. The part you added is the part that counts.
• Many services, no correctness. Reconciliation and a rerun test impress more than an extra service.
• Leaving resources running, or publishing access keys.
• Hiding the mess. Real data has faults. Showing how you handled them is the point.

An end-to-end ETL pipeline, a cloud data warehouse, a real-time streaming pipeline and a data lake implementation are the projects built in the Data Engineering course.

Sources

• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission
• GeoJSON summary feed (U.S. Geological Survey) — U.S. Geological Survey
• EventStreams HTTP service (Wikimedia) — Wikimedia
• Amazon Athena pricing (Amazon Web Services) — Amazon Web Services
• Explore AWS services with AWS Free Tier (AWS Billing and Cost Management User Guide) — AWS Documentation
• What is Step Functions? (AWS Step Functions Developer Guide) — AWS Documentation