Building a Data Lake on AWS with S3 and Glue

A data lake on AWS is less a product than a set of decisions. The storage is Amazon S3, the index is the AWS Glue Data Catalog, and the query engines come and go. What makes it a lake and not a pile of files is how the bucket is laid out, what is allowed into each part of it, and whether every dataset is described in the catalog.

AWS's storage whitepaper defines a data lake as a centralised repository for structured, semi-structured and unstructured data that can be catalogued, secured and analysed by a diverse set of users and tools, with compute decoupled from storage (AWS whitepaper: storage best practices). For how that differs from a warehouse, see data lake vs data warehouse vs lakehouse. This guide is about building one.

Step 1: decide the zones

Separate data by how far it has been processed. Two zones are enough to start:

• Raw. Files exactly as they arrived, never edited. This is the record of what the source sent, and the material for reprocessing when a transformation turns out to be wrong.
• Curated. Cleaned, typed, deduplicated data in an analytical format, organised for queries.

Larger platforms add a staging zone for work in progress and an analytics or "gold" zone for aggregated, business-ready tables. Add them when there is a need, not in advance.

Zones can be prefixes in one bucket or separate buckets. Separate buckets make permissions and lifecycle rules simpler to reason about; prefixes are fine for a small lake.

Step 2: design the keys

An S3 key is the only structure a lake has, so make it carry meaning:

s3://demo-lake/raw/<source>/<dataset>/ingest_date=YYYY-MM-DD/<original file>
s3://demo-lake/curated/<table>/<partition_key>=<value>/part-0000.parquet

• Raw is organised by arrival. Partition by the date the file was received, which you control, not by dates inside the data.
• Curated is organised by query. Partition by the column analysts filter on; see partitioning data in S3.
• Use lowercase folders in the form key=value, the Hive style that Glue, Athena and Spark all recognise as partitions.
• One table per prefix. Do not mix datasets under a shared folder.

Step 3: land, clean, catalog

The following builds a miniature lake end to end with boto3: a month of New York taxi trips (NYC TLC: trip record data) is landed raw, a cleaned copy is written as Parquet to the curated zone, and the curated table is registered in the Glue Data Catalog. It runs against moto, a local emulator of the S3 and Glue APIs, so the calls are the real ones but no AWS account is involved.

import io, os
import boto3
import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.parquet as pq
from moto import mock_aws

os.environ.update(AWS_ACCESS_KEY_ID="testing", AWS_SECRET_ACCESS_KEY="testing",
                  AWS_DEFAULT_REGION="us-east-1")
BUCKET, SOURCE = "demo-lake", "yellow_tripdata_2024-01.parquet"
KEEP = ["tpep_pickup_datetime", "tpep_dropoff_datetime", "PULocationID", "DOLocationID",
        "passenger_count", "trip_distance", "payment_type", "fare_amount", "total_amount"]


@mock_aws
def main():
    s3, glue = boto3.client("s3"), boto3.client("glue")
    s3.create_bucket(Bucket=BUCKET)

    # 1. Raw zone: the file exactly as received, under the date it arrived
    raw_key = f"raw/tlc/yellow/ingest_date=2024-02-05/{SOURCE}"
    s3.upload_file(SOURCE, BUCKET, raw_key)

    # 2. Curated zone: cleaned, trimmed to the columns analysts use, partitioned by month
    trips = pq.read_table(SOURCE, columns=KEEP)
    month = pc.strftime(trips["tpep_pickup_datetime"], format="%Y-%m")
    good = pc.and_(pc.equal(month, "2024-01"), pc.greater_equal(trips["fare_amount"], 0))
    clean = trips.filter(good)
    print(f"raw rows {trips.num_rows:,}  kept {clean.num_rows:,}  rejected {trips.num_rows - clean.num_rows:,}")
    buffer = io.BytesIO()
    pq.write_table(clean, buffer, compression="zstd")
    curated_prefix = "curated/trips/"
    s3.put_object(Bucket=BUCKET, Key=curated_prefix + "pickup_month=2024-01/part-0000.parquet",
                  Body=buffer.getvalue())

    # 3. Catalog: describe the curated table so query engines can find it
    glue.create_database(DatabaseInput={"Name": "lake"})
    types = {pa.timestamp("us"): "timestamp", pa.int32(): "int", pa.int64(): "bigint", pa.float64(): "double"}
    storage = {
        "Columns": [{"Name": f.name.lower(), "Type": types[f.type]} for f in clean.schema],
        "Location": f"s3://{BUCKET}/{curated_prefix}",
        "InputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
        "OutputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
        "SerdeInfo": {"SerializationLibrary": "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"},
    }
    glue.create_table(DatabaseName="lake", TableInput={
        "Name": "trips", "TableType": "EXTERNAL_TABLE", "StorageDescriptor": storage,
        "PartitionKeys": [{"Name": "pickup_month", "Type": "string"}],
        "Parameters": {"classification": "parquet"}})
    glue.create_partition(DatabaseName="lake", TableName="trips", PartitionInput={
        "Values": ["2024-01"],
        "StorageDescriptor": {**storage, "Location": storage["Location"] + "pickup_month=2024-01/"}})

    # What exists now
    print()
    for obj in s3.list_objects_v2(Bucket=BUCKET)["Contents"]:
        print(f"{obj['Size'] / 2**20:6.1f} MB  s3://{BUCKET}/{obj['Key']}")
    table = glue.get_table(DatabaseName="lake", Name="trips")["Table"]
    print()
    print("catalog table lake.trips ->", table["StorageDescriptor"]["Location"])
    print("  columns:", ", ".join(f"{c['Name']} {c['Type']}" for c in table["StorageDescriptor"]["Columns"][:4]), "...")
    print("  partition keys:", [k["Name"] for k in table["PartitionKeys"]])
    parts = glue.get_partitions(DatabaseName="lake", TableName="trips")["Partitions"]
    print("  partitions:", [p["Values"] for p in parts])


main()

raw rows 2,964,624  kept 2,927,159  rejected 37,465

  40.6 MB  s3://demo-lake/curated/trips/pickup_month=2024-01/part-0000.parquet
  47.6 MB  s3://demo-lake/raw/tlc/yellow/ingest_date=2024-02-05/yellow_tripdata_2024-01.parquet

catalog table lake.trips -> s3://demo-lake/curated/trips/
  columns: tpep_pickup_datetime timestamp, tpep_dropoff_datetime timestamp, pulocationid int, dolocationid int ...
  partition keys: ['pickup_month']
  partitions: [['2024-01']]

Three things happened.

The raw file was stored untouched. All 2,964,624 rows, including the bad ones, are preserved under the date they arrived.

The curated layer applied rules. Trips dated outside January 2024 and trips with a negative fare were removed, 37,465 rows, and only the nine columns analysts use were kept. The result is a 40.6 MB Parquet file in a month partition. In a real lake those rules, and the count of rows each rejects, belong in a tested job.

The catalog now describes the table. A database lake holds a table trips with its columns, its S3 location, a partition key and one registered partition. Nothing in S3 changed in this step. The catalog entry is what turns a folder of files into something Athena, Redshift and Glue jobs can all refer to by name.

Step 4: choose how tables get into the catalog

The Glue Data Catalog is a centralised metadata repository holding the location, schema and properties of each dataset (Glue docs: Data Catalog). There are two ways to fill it.

A crawler classifies the data, groups it into tables or partitions using heuristics, and writes the metadata (Glue docs: crawlers). That is convenient, and heuristics are not contracts. For curated tables, defining the schema in code keeps it stable and reviewable. The Glue Data Catalog and crawlers goes further.

Step 5: transform at scale

The example cleans one file in memory. Real lakes run that step as a job: AWS Glue provides serverless, Spark-based ETL jobs that read and write catalog tables (Glue docs: what is AWS Glue), and EMR or Athena CREATE TABLE AS statements are alternatives. Whatever runs it, the job should:

• Read from raw, write to curated, and never modify raw.
• Write Parquet with compression; see Parquet and compression for cheaper Athena queries.
• Overwrite a whole partition per run, so reruns do not duplicate rows.
• Register new partitions as part of the run.

Step 6: query it

With the table in the catalog, Amazon Athena can query it immediately with SQL, with no further setup beyond a location for results. The same catalog entry serves Redshift external tables and Spark jobs.

Step 7: secure and control cost

• Encryption. S3 encrypts new objects by default with S3 managed keys (S3 docs: default encryption). Use KMS keys where you need control over who can decrypt.
• No public access. Keep S3 Block Public Access on for lake buckets.
• Least privilege. Pipelines write; analysts read curated only. Few people should be able to read raw, which often holds personal data that curated tables mask or drop.
• Fine-grained access. When column or row rules are needed, AWS Lake Formation adds them on top of the catalog.
• Lifecycle rules. Tier or expire old raw data and query results; see S3 lifecycle policies.

How lakes become swamps

• No catalog entry. Data nobody can find does not exist.
• Edited raw files. Once raw is changed, nothing can be reproduced.
• CSV and JSON in the curated zone.
• Thousands of tiny files from frequent small writes, never compacted.
• No owner per dataset, so nobody knows whether it is still correct.
• Everything readable by everyone.

A lake that avoids these six is in good shape, whatever else it lacks.

Data lake implementation with Amazon S3 and AWS Glue is covered, as a project, in the Data Engineering course.

Sources

• Storage Best Practices for Data and Analytics Applications (AWS Whitepaper) — AWS Whitepapers
• Data discovery and cataloging in AWS Glue (AWS Glue User Guide) — AWS Documentation
• Using crawlers to populate the Data Catalog (AWS Glue User Guide) — AWS Documentation
• What is AWS Glue? (AWS Glue User Guide) — AWS Documentation
• Configuring default encryption (Amazon S3 User Guide) — AWS Documentation
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission