AWS Cost Optimization for Data Workloads

Cloud bills for data platforms rarely grow because of one bad decision. They grow because each service bills for a different thing, nobody holds the whole picture, and defaults are set for convenience. AWS's Well-Architected Framework defines cost optimisation as the ability to run systems to deliver business value at the lowest price point (AWS: Well-Architected cost optimization). For data work that reduces to a short discipline: know what each service meters, pull the one or two levers that matter for it, and put guard rails around the rest.

Rates vary by Region and change. This guide deals in meters and ratios, not prices.

What each service bills for

Usually a few rows carry most of the bill. Find out which before optimising anything.

The lever with the largest ratio: how data is stored

For query engines that bill by data scanned, storage layout is the cost. To show how far the levers stack, we stored one month of New York taxi trips four ways and measured what a query for one day's fares by payment type would have to read. This ran on a local disk; the sizes stand in for bytes scanned.

import gzip, io, os, shutil
import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.csv as pacsv
import pyarrow.dataset as ds
import pyarrow.parquet as pq

table = pq.read_table("yellow_tripdata_2024-01.parquet")
need = ["payment_type", "fare_amount"]            # the query: fares by payment type for one day
DAY = "2024-01-15"


def column_bytes(path, columns):
    meta, total = pq.ParquetFile(path).metadata, 0
    for rg in range(meta.num_row_groups):
        for c in range(meta.num_columns):
            col = meta.row_group(rg).column(c)
            if col.path_in_schema in columns:
                total += col.total_compressed_size
    return total


# Step 0: the month as one CSV file, read in full
pacsv.write_csv(table, "trips.csv")
steps = [("raw CSV, whole month", os.path.getsize("trips.csv"))]

# Step 1: compress it
with open("trips.csv", "rb") as f, gzip.open("trips.csv.gz", "wb") as g:
    shutil.copyfileobj(f, g)
steps.append(("gzip CSV, whole month", os.path.getsize("trips.csv.gz")))

# Step 2: Parquet, reading only the two columns the query needs
pq.write_table(table, "trips.parquet", compression="zstd")
steps.append(("Parquet, two columns, whole month", column_bytes("trips.parquet", need)))

# Step 3: partition by day as well, and read only that day's file
dated = table.append_column("pickup_date", pc.strftime(table["tpep_pickup_datetime"], format="%Y-%m-%d"))
shutil.rmtree("by_date", ignore_errors=True)
ds.write_dataset(dated, "by_date", format="parquet",
                 file_options=ds.ParquetFileFormat().make_write_options(compression="zstd"),
                 partitioning=ds.partitioning(pa.schema([dated.schema.field("pickup_date")]), flavor="hive"))
day_dir = os.path.join("by_date", f"pickup_date={DAY}")
day_bytes = sum(column_bytes(os.path.join(day_dir, f), need) for f in os.listdir(day_dir))
steps.append(("Parquet, two columns, one day", day_bytes))

base = steps[0][1]
print(f"{'data a query must read':36} {'size':>12} {'of raw':>9}")
for label, size in steps:
    shown = f"{size / 2**20:.1f} MB" if size >= 2**20 else f"{size / 1024:.0f} KB"
    print(f"{label:36} {shown:>12} {size / base:>9.3%}")

data a query must read                       size    of raw
raw CSV, whole month                     306.5 MB  100.000%
gzip CSV, whole month                     51.5 MB   16.806%
Parquet, two columns, whole month          3.5 MB    1.134%
Parquet, two columns, one day               92 KB    0.029%

The same question, on the same data, reads 306.5 MB from raw CSV and 92 KB from day-partitioned Parquet: three hundredths of one percent. Each step contributes:

• Compression takes it to about 17 percent.
• A columnar format means only the two needed columns are read: about 1 percent.
• Partitioning restricts that to one day's file.

AWS's own claim is more modest, that compressing, partitioning and converting to columnar formats can save up to 90 percent per query (AWS: Athena pricing). The measured ratio here is larger because the query is narrow, two columns and one day of 19 columns and a month. A query that reads every column of every day gains only the compression. Note also Athena's 10 MB minimum per query: savings below that floor are not passed on.

The detail is in Parquet and compression for cheaper Athena queries and partitioning data in S3. The same layout reduces Glue and EMR run times, because jobs read less, and Redshift external table costs.

Storage

• Lifecycle rules on every bucket: expire temporary prefixes and query results, tier or archive old raw data, abort incomplete multipart uploads, expire old versions.
• Mind the traps. Objects under 128 KB are not transitioned by default because the per-object request can outweigh the saving, and leaving a storage class before its minimum duration is charged (S3 docs: lifecycle transitions).
• Compact small files. Fewer, larger objects cost less to store in colder classes, to list and to query.
• Delete what is derived and unused. Old staging tables and abandoned experiments are pure cost.

S3 lifecycle policies covers the rules.

Transformation: Glue and EMR

Glue bills per DPU-hour, by the second, with a one-minute minimum. A Spark job is given 10 DPUs by default (AWS: Glue pricing).

• Set the worker count. The default is the commonest waste. Start small and increase only if the run time demands it.
• Use the right job type. A Python shell job at a fraction of a DPU, or a Lambda function, for work that does not need Spark.
• Process incrementally. Read only new partitions, not the full history each night.
• Set a timeout on every job so that a hung run cannot bill for hours.
• Do not crawl on a schedule when nothing has changed.

EMR bills per second on top of the EC2 instances. Its pricing page makes the key point: a 10-node cluster running for 10 hours costs the same as a 100-node cluster running for one hour (AWS: EMR pricing).

• Use transient clusters that terminate after the job. An idle cluster costs what a busy one does.
• Spot Instances for task nodes.
• Right-size from utilisation metrics.

See AWS Glue jobs, Amazon EMR explained and, for instance choice and purchasing options, Amazon EC2 explained.

Query and warehouse

• Athena: a per-query data limit on every workgroup, so one careless query is cancelled instead of billed. Pre-aggregate tables that dashboards read repeatedly.
• Redshift Serverless: set the base capacity deliberately and a usage limit; it bills for capacity while workloads run.
• Redshift provisioned: steady workloads suit it; a cluster sized for the peak and idle overnight does not.
• Hot and cold: keep recent, frequently queried data in the warehouse and history in S3.

The costs nobody budgets for

• NAT gateways. Data pipelines push large volumes to and from S3. A gateway endpoint gives a VPC access to S3 without a NAT device, at no additional charge (VPC docs: gateway endpoints for S3).
• KMS requests. A job reading millions of SSE-KMS objects makes millions of KMS calls. S3 Bucket Keys can reduce KMS request costs by up to 99 percent (S3 docs: bucket keys).
• Logs. Verbose job logs kept for ever. Set retention on every log group.
• Data transfer between Regions, and out of AWS. Keep compute in the same Region as the data.
• Forgotten resources: development clusters, stopped instances with large disks, unused streams, old snapshots.
• Streaming retention. Extending a stream's retention beyond the default is charged.

Guard rails

Optimisation done once decays. Three mechanisms keep it in place.

Budgets. AWS Budgets tracks spend against a target and alerts on both actual and forecasted cost; it can also apply an action, such as an IAM policy, when a threshold is reached (AWS docs: AWS Budgets). Create one per account and one per major workload. A forecast alert on day ten is worth more than an invoice on day thirty.

Tags. A tag is a key and value on a resource; once activated as cost allocation tags, they organise costs in reports (AWS docs: cost allocation tags). Tag by pipeline, team and environment. Without tags, the bill says "Glue" and nothing more.

Limits in the services themselves. Athena workgroup limits, job timeouts, Redshift usage limits, maximum workers on autoscaling. These stop a mistake while it is happening, where a budget only reports it.

Cost Explorer, AWS's tool for viewing and analysing costs and usage (AWS docs: Cost Explorer), then answers "what changed?": view cost by service, by tag and by day, and look at the days where the line jumps.

A working order

1. Open Cost Explorer and rank services by cost for the last three months.
2. Take the top three. For each, name the meter and the lever from the table above.
3. Fix storage layout first if Athena, Glue or EMR is on the list.
4. Remove waste: idle clusters, default worker counts, missing lifecycle rules, forgotten resources.
5. Add budgets, tags and service limits.
6. Review monthly, and after any new pipeline.

The principle underneath all of it: pay for work, not for waiting. A design that scales to zero when idle and reads only what it needs is cheap almost by construction; building a serverless analytics stack on AWS describes one.

AWS data services and the design of cost-aware data pipelines are covered in the Data Engineering course.

Sources

• Cost optimization (AWS Well-Architected Framework) — AWS Documentation
• Amazon Athena pricing (Amazon Web Services) — Amazon Web Services
• AWS Glue pricing (Amazon Web Services) — Amazon Web Services
• Amazon EMR pricing (Amazon Web Services) — Amazon Web Services
• Transitioning objects using Amazon S3 Lifecycle (Amazon S3 User Guide) — AWS Documentation
• Reducing the cost of SSE-KMS with Amazon S3 Bucket Keys (Amazon S3 User Guide) — AWS Documentation
• Gateway endpoints for Amazon S3 (AWS PrivateLink) — AWS Documentation
• Managing your costs with AWS Budgets (AWS Billing and Cost Management User Guide) — AWS Documentation
• Analyzing your costs and usage with AWS Cost Explorer (AWS Billing and Cost Management User Guide) — AWS Documentation
• Organizing and tracking costs using AWS cost allocation tags (AWS Billing and Cost Management User Guide) — AWS Documentation