S3 Lifecycle Policies and Storage Cost Savings

An S3 bucket never tells you it is full. Objects accumulate, the bill rises a little each month, and nobody deletes anything because nobody is sure what is still needed. S3 Lifecycle is the mechanism for deciding in advance. A lifecycle configuration is a set of rules that Amazon S3 applies to groups of objects, with two types of action (S3 docs: managing the lifecycle of objects):

• Transition actions move objects to another storage class after a number of days.
• Expiration actions delete objects at the end of their life.

Storage classes themselves, what each costs relative to the others and how retrieval works, are covered in Amazon S3 explained. This article is about the rules.

Anatomy of a rule

Each rule has an ID, a status, a filter that selects objects, by key prefix, tags or size, and one or more actions. A bucket's configuration can hold many rules, so different prefixes can age differently.

Transitions only go one way. S3 supports what the guide calls a waterfall: from Standard down through Standard-IA, Intelligent-Tiering, One Zone-IA and Glacier Instant Retrieval to Glacier Flexible Retrieval and Glacier Deep Archive (S3 docs: transitioning objects). A lifecycle rule cannot move an object back up.

A rule set for a data lake bucket

A typical analytics bucket holds three kinds of object with three different futures: raw files that must be kept for years but are rarely read, query results that are worthless after a few days, and debris. The following configuration handles each. It was applied with boto3 against moto, a local emulator of the AWS API, and read back.

import os
import boto3
from moto import mock_aws

os.environ.update(AWS_ACCESS_KEY_ID="testing", AWS_SECRET_ACCESS_KEY="testing",
                  AWS_DEFAULT_REGION="us-east-1")

RULES = [
    {   # raw landing files: cheaper storage after a month, archive after a year, delete after 7 years
        "ID": "raw-tiering",
        "Filter": {"Prefix": "raw/"},
        "Status": "Enabled",
        "Transitions": [{"Days": 30, "StorageClass": "STANDARD_IA"},
                        {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}],
        "Expiration": {"Days": 2555},
    },
    {   # query results and temporary files: delete after a week
        "ID": "expire-query-results",
        "Filter": {"Prefix": "athena-results/"},
        "Status": "Enabled",
        "Expiration": {"Days": 7},
    },
    {   # whole bucket: clean up failed uploads and old versions
        "ID": "housekeeping",
        "Filter": {"Prefix": ""},
        "Status": "Enabled",
        "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7},
        "NoncurrentVersionExpiration": {"NoncurrentDays": 30},
    },
]


@mock_aws
def main():
    s3 = boto3.client("s3")
    s3.create_bucket(Bucket="demo-data-lake")
    s3.put_bucket_versioning(Bucket="demo-data-lake", VersioningConfiguration={"Status": "Enabled"})
    s3.put_bucket_lifecycle_configuration(Bucket="demo-data-lake", LifecycleConfiguration={"Rules": RULES})

    stored = s3.get_bucket_lifecycle_configuration(Bucket="demo-data-lake")["Rules"]
    for rule in stored:
        actions = []
        for t in rule.get("Transitions", []):
            actions.append(f"day {t['Days']} -> {t['StorageClass']}")
        if "Expiration" in rule:
            actions.append(f"day {rule['Expiration']['Days']} -> delete")
        if "AbortIncompleteMultipartUpload" in rule:
            actions.append(f"abort unfinished uploads after {rule['AbortIncompleteMultipartUpload']['DaysAfterInitiation']} days")
        if "NoncurrentVersionExpiration" in rule:
            actions.append(f"delete old versions after {rule['NoncurrentVersionExpiration']['NoncurrentDays']} days")
        prefix = rule["Filter"].get("Prefix") or "(whole bucket)"
        print(f"{rule['ID']:22} {prefix:16} {'; '.join(actions)}")


main()

raw-tiering            raw/             day 30 -> STANDARD_IA; day 365 -> DEEP_ARCHIVE; day 2555 -> delete
expire-query-results   athena-results/  day 7 -> delete
housekeeping           (whole bucket)   abort unfinished uploads after 7 days; delete old versions after 30 days

The emulator confirms that the request is well formed and shows what is stored. It does not move or delete objects, and it does not enforce every constraint of the real service, so treat this as the shape of a configuration and test timings in a real, non-production bucket.

What each rule does:

• raw-tiering. Raw files go to Standard-IA after 30 days and to Glacier Deep Archive after a year, and are deleted after seven years. The day counts are illustrative; yours come from how the data is used and how long it must be kept.
• expire-query-results. Athena writes a result file for every query. A seven-day expiry stops that prefix growing for ever.
• housekeeping. Two actions that apply to the whole bucket and are easy to forget, covered below.

The two rules every bucket should have

Abort incomplete multipart uploads. Large uploads are sent in parts. If an upload fails or is abandoned, the parts already sent stay in the bucket, invisible in normal listings and billed as storage. AWS recommends, as a best practice, a lifecycle rule with the AbortIncompleteMultipartUpload action, which stops uploads not completed within the set number of days and deletes their parts (S3 docs: incomplete multipart uploads).

Expire noncurrent versions. With versioning enabled, overwriting or deleting an object keeps the old version. That protects against mistakes and quietly multiplies storage. In a versioned bucket an expiration action does not remove data at all: it adds a delete marker and makes the current version noncurrent. Only NoncurrentVersionExpiration permanently deletes old versions (S3 docs: expiring objects).

Where the savings go missing

Moving data to a cheaper class is not automatically cheaper. Four charges and behaviours decide the outcome.

Transition requests are charged per object. A million tiny files cost a million requests to move. For this reason S3 does not transition objects smaller than 128 KB by default; the guide says the transition cost can outweigh the storage saving. A data lake full of small files gains little from tiering until the files are compacted.

Minimum storage durations. Leave a storage class early, by deleting, overwriting or transitioning, and you are charged for the remainder of its minimum. The Glacier classes have minimums of 90 days for Instant Retrieval and Flexible Retrieval and 180 days for Deep Archive (S3 docs: Glacier storage classes). A rule that archives data which is then deleted a month later costs more than leaving it alone.

Retrieval. Infrequent-access and archive classes carry retrieval charges. Objects in Glacier Flexible Retrieval and Deep Archive are not available in real time at all: they must be restored first, which for Flexible Retrieval takes from minutes to 12 hours.

Analytics stops seeing archived data. Athena skips objects in those two archive classes unless a table property is set, and even then the objects must have been restored (Athena docs: restored Glacier objects). Archive a partition and it silently disappears from query results. Decide with the people who query the data, and remove or document the affected partitions.

Behaviours worth knowing before you apply a rule

• Rules apply to existing objects. Add a 30-day expiry today and every matching object already older than 30 days is queued for removal.
• A bucket policy cannot block lifecycle actions. Even a policy that denies everything does not stop a lifecycle rule.
• One action per object per day. When several rules match, permanent deletion takes precedence over transition.
• Actions are asynchronous. There can be a delay between an object becoming eligible and S3 acting; billing changes apply from eligibility.

The first point deserves emphasis. A lifecycle rule is a scheduled bulk delete. Check the prefix twice, and if in doubt, enable versioning first so a mistake is recoverable.

When you do not know the access pattern

Lifecycle rules assume you can predict when data goes cold. If you cannot, the guide's suggestion is the S3 Intelligent-Tiering storage class, which moves objects between access tiers automatically according to how they are used. A single lifecycle rule that transitions objects into Intelligent-Tiering can replace a ladder of hand-picked day counts.

A working method

1. List the prefixes in the bucket and write down, for each, how long data is read and how long it must be kept.
2. Add the two housekeeping actions to every bucket.
3. Expire temporary prefixes: query results, staging, logs with no retention requirement.
4. Tier or archive long-lived data, checking object sizes and minimum durations first.
5. Apply to a test bucket and confirm in the console which objects each rule matches.
6. Review storage by class a month later and adjust.

Storage is one line of an analytics bill. Query and compute costs follow different rules: Amazon Athena explained covers the per-query side, and AWS cost optimisation for data workloads covers the rest.

Amazon S3 and data lake storage on AWS are covered in the Data Engineering course.

Sources

• Managing the lifecycle of objects (Amazon S3 User Guide) — AWS Documentation
• Transitioning objects using Amazon S3 Lifecycle (Amazon S3 User Guide) — AWS Documentation
• Expiring objects (Amazon S3 User Guide) — AWS Documentation
• Configuring a bucket lifecycle configuration to delete incomplete multipart uploads (Amazon S3 User Guide) — AWS Documentation
• Understanding S3 Glacier storage classes for long-term data storage (Amazon S3 User Guide) — AWS Documentation
• Query restored Amazon Glacier objects (Amazon Athena User Guide) — AWS Documentation