Amazon S3 Explained: Storage, Classes and Best Practices

Every object in Amazon S3 carries a storage class, and the class decides three things: what you pay to keep the object, what you pay to read it, and how quickly it comes back. If you do not choose, S3 uses S3 Standard (AWS: storage classes). This guide compares the classes on the numbers AWS publishes, gives a four-question method for placing a dataset, and ends with the bucket defaults worth setting on day one.

How storage classes work

The class is a property of each object, not of the bucket. One bucket can hold fresh files in S3 Standard, last quarter’s exports in S3 Standard-IA and seven-year-old audit logs in S3 Glacier Deep Archive, all under the same key structure.

All current classes are designed for 99.999999999% durability. What changes between them is availability, how many Availability Zones hold the data, whether reads are instant or need a restore, and two billing rules that catch people out: a minimum storage duration and a minimum billable object size.

The classes side by side

Source: the comparison table in Understanding and managing Amazon S3 storage classes. AWS still lists Reduced Redundancy Storage, but recommends against it because S3 Standard is more cost-effective.

Choosing a class: four questions

Work through these in order for each dataset. The first "yes" usually settles it.

1. Is it read more than about once a month, or does something query it on a schedule? Keep it in S3 Standard. Pipelines that re-read raw files, and tables behind a daily dashboard, belong here.
2. Do you genuinely not know how often it will be read? Use S3 Intelligent-Tiering. It moves each object to a cheaper tier after 30 days without access, and to the Archive Instant Access tier after 90, with no retrieval fees. You pay a small monitoring and automation fee per object instead.
3. Is it rarely read, but must come back instantly when it is? Use S3 Standard-IA, or S3 Glacier Instant Retrieval if reads are closer to quarterly and the data will stay at least 90 days.
4. Can the reader wait minutes or hours? Archive it in S3 Glacier Flexible Retrieval or, for data you keep for compliance and almost never open, S3 Glacier Deep Archive.

One more check before choosing either One Zone class: can you re-create the data if an Availability Zone is lost? AWS recommends S3 Standard-IA for a primary copy that cannot be re-created. It suggests S3 One Zone-IA for data you can rebuild, such as a replica made with Cross-Region Replication.

The two billing rules that change the answer

S3 Standard-IA and S3 One Zone-IA charge for at least 30 days of storage. An object you delete, overwrite or transition after 10 days is still billed for the full 30. They also bill every object as at least 128 KB. The Glacier classes have longer minimums: 90 days for Instant and Flexible Retrieval, 180 days for Deep Archive.

The 128 KB rule matters for data lakes full of small files. Take one million JSON event files of about 10 KB each: roughly 10 GB of data. In S3 Standard-IA each file is billed as 128 KB, so you pay for about 128 GB, nearly thirteen times what you store. For small files, compact them into larger Parquet files first, or leave them in S3 Standard. Intelligent-Tiering does not help either: objects under 128 KB are never monitored and always stay in its Frequent Access tier.

Getting data back from the archive classes

S3 Glacier Instant Retrieval objects are read with an ordinary GET. S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects are archived: you first request a restore, which makes a temporary copy readable for the number of days you ask for. The speed depends on the retrieval tier (AWS: retrieval options):

# Ask for a Bulk restore that stays readable for 7 days
aws s3api restore-object \
  --bucket my-data-lake \
  --key archive/2019/events.parquet \
  --restore-request '{"Days": 7, "GlacierJobParameters": {"Tier": "Bulk"}}'

# Check progress: the Restore field shows ongoing-request="false" when it is ready
aws s3api head-object --bucket my-data-lake --key archive/2019/events.parquet

Setting the class in practice

You can set the class on upload with the x-amz-storage-class header, which the AWS CLI exposes as --storage-class (AWS: setting the storage class):

aws s3 cp exports/2026-08.parquet s3://my-data-lake/exports/2026-08.parquet --storage-class STANDARD_IA

For data that ages, a lifecycle rule is better than choosing at upload: write to S3 Standard, then let S3 transition or expire objects on a schedule (AWS: lifecycle). This rule moves everything under logs/ to Standard-IA after 30 days and to Flexible Retrieval after 90, and deletes it after a year:

{
  "Rules": [
    {
      "ID": "age-out-logs",
      "Filter": { "Prefix": "logs/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-data-lake \
  --lifecycle-configuration file://lifecycle.json

Lifecycle transitions are billed per request, and moving an object out of Standard-IA before its 30 days are up still incurs the charge for the remainder. Rule design (versioned buckets, tag filters, overlapping rules) is covered in S3 lifecycle policies and storage cost savings.

Bucket defaults for data engineering

• Leave Block Public Access on. New buckets, access points and objects do not allow public access by default (AWS: Block Public Access). If something must be public, make it a separate bucket with its own review, never a policy change on the data lake.
• Know what encryption you already have. Since 5 January 2023, S3 encrypts every new upload with SSE-S3 at no extra cost (AWS: server-side encryption). Choose SSE-KMS only when you need to control and audit use of the key itself.
• Make prefixes match how data ages and how it is queried. Lifecycle rules filter on prefixes and tags, and query engines prune on partition prefixes. A layout like raw/source=…/dt=…/ serves both; see partitioning data in S3 for Athena.
• Measure before you move data to Standard-IA. Storage Class Analysis observes access patterns for a prefix or tag filter and helps you decide when to transition Standard data to Standard-IA (AWS: Storage Class Analysis). The 30 in your lifecycle rule should be a number you measured, not a guess.

Mistakes that cost money

• Transitioning millions of small files to an infrequent-access class, where each one is billed as 128 KB.
• Archiving a prefix a dashboard or scheduled job still reads, then paying for urgent Expedited restores.
• Storing the only copy of data you cannot rebuild in a One Zone class.
• Writing temporary files to Standard-IA and deleting them within days: you still pay for 30.

Where to go next

For how S3 fits into a pipeline, read building a data lake on AWS with S3 and Glue. If you are deciding whether S3 is the right storage at all, S3 vs EBS vs EFS covers that. S3, Glue and Redshift are part of the AWS module in our Data Engineering course.

Sources

• Understanding and managing Amazon S3 storage classes — AWS documentation
• Understanding archive retrieval options — AWS documentation
• Setting the storage class of an object — AWS documentation
• Managing the lifecycle of objects — AWS documentation
• Restoring an archived object — AWS documentation
• Query restored Amazon Glacier objects — Amazon Athena documentation
• Amazon S3 analytics – Storage Class Analysis — AWS documentation
• Blocking public access to your Amazon S3 storage — AWS documentation
• Protecting data with server-side encryption — AWS documentation