Google Cloud Storage: Buckets, Classes and Best Practices

Every data platform on Google Cloud starts with files landing somewhere, and that somewhere is Google Cloud Storage. It looks simple, a bucket and some objects, and most of the decisions that matter are made in the first five minutes of creating a bucket: its name, its location, its storage class. Several of them cannot be changed afterwards. This guide covers what to decide and why.

Buckets and objects

Cloud Storage stores data as objects inside containers called buckets. A bucket belongs to a project. An object is an immutable piece of data of any format; to change it, you replace it.

Rules about buckets worth knowing before you create one (Google Cloud: about buckets):

• Names are global. Every bucket name is unique across all of Cloud Storage, not only your project.
• Names are public. Anyone can probe whether a bucket name exists. Google's guidance is to keep personal data, project names and anything sensitive out of bucket names.
• Names are permanent. A bucket cannot be renamed. You create a new one and move the data.
• Format: 3 to 63 characters, lowercase letters, numbers, dashes, underscores and dots, starting and ending with a letter or number.
• Buckets cannot be nested.

Inside a bucket, "folders" are normally just a naming convention: an object called raw/2026/10/orders.parquet is one object whose name contains slashes. A bucket created with hierarchical namespace enabled has real folders, which can be renamed in one operation. That setting, too, is fixed at creation.

Location

A bucket's location is where its data physically lives. There are three main types (Google Cloud: bucket locations):

For analytics, the usual answer is a region, the same one as the compute that will read the data. Google's own recommendation table puts analytics workloads there. A bucket in the United States multi-region read by a BigQuery dataset or a Spark cluster in a single region pays for data transfer that co-location would have avoided.

Storage classes

Every object has a storage class. The class changes what you pay to keep the data and what you pay to touch it (Google Cloud: storage classes):

Two facts set Cloud Storage apart from some other providers, and both come from the same page:

• All classes are online. Google states that Archive data is available within milliseconds, not hours or days. Colder classes cost more to read; they are not slower to reach.
• All classes have the same durability, quoted as 99.999999999% a year. What drops slightly in colder classes is availability.

There is also a Rapid class, for zonal buckets aimed at I/O-intensive work such as machine learning. It is a specialised option, not part of the usual warm-to-cold ladder.

The minimum duration is a real cost

The minimum storage duration is not a restriction on deleting. It is a billing rule. Delete, replace or move an object before its minimum, and you are charged as if it had been stored for the whole minimum (Google Cloud: storage pricing).

That turns some apparently thrifty choices into expensive ones. Temporary files written to a Coldline bucket and deleted after a week are billed for ninety days each. Data that is overwritten daily should be in Standard, whatever its size.

Lifecycle rules

Lifecycle rules move or delete objects automatically. A rule has one action and one or more conditions, and it is set on the bucket (Google Cloud: Object Lifecycle Management).

{
  "rule": [
    { "action": { "type": "SetStorageClass", "storageClass": "NEARLINE" }, "condition": { "age": 30 } },
    { "action": { "type": "SetStorageClass", "storageClass": "COLDLINE" }, "condition": { "age": 90 } },
    { "action": { "type": "SetStorageClass", "storageClass": "ARCHIVE" }, "condition": { "age": 365 } },
    { "action": { "type": "Delete" }, "condition": { "age": 730 } }
  ]
}

Conditions can also match a name prefix or suffix, the current storage class, a creation date, or the number of newer versions.

When an object satisfies several rules at once, only one action is taken. The documentation gives the order: Delete takes precedence over any change of class, and among changes of class, the one to the class with the lowest at-rest price wins. Transitions also only go one way, towards colder classes. The script below is a model of those rules applied to the configuration above. It is not Cloud Storage; it evaluates the rules on sample objects.

# A model of how Object Lifecycle Management chooses one action when several rules match,
# following the documented precedence. It is not Cloud Storage; it evaluates rules on sample ages.
COLDNESS = {"STANDARD": 0, "NEARLINE": 1, "COLDLINE": 2, "ARCHIVE": 3}      # higher = cheaper at rest
MINIMUM_DAYS = {"STANDARD": 0, "NEARLINE": 30, "COLDLINE": 90, "ARCHIVE": 365}

RULES = [                                       # (minimum age in days, action, target class)
    (30,  "SetStorageClass", "NEARLINE"),
    (90,  "SetStorageClass", "COLDLINE"),
    (365, "SetStorageClass", "ARCHIVE"),
    (730, "Delete", None),
]


def decide(age_days, current_class):
    matching = [(action, target) for min_age, action, target in RULES if age_days >= min_age]
    if any(action == "Delete" for action, _ in matching):
        return "Delete"                                           # Delete takes precedence
    targets = [t for action, t in matching if action == "SetStorageClass"]
    if not targets:
        return "no action"
    coldest = max(targets, key=COLDNESS.get)                      # lowest at-rest price wins
    if COLDNESS[coldest] <= COLDNESS[current_class]:
        return "no action"                                        # transitions only go colder
    return f"SetStorageClass {coldest}"


print(f"{'age':>4}  {'current class':14} {'rules matched':>13}  action taken")
for age, current in [(10, "STANDARD"), (45, "STANDARD"), (120, "NEARLINE"), (120, "STANDARD"),
                     (400, "COLDLINE"), (400, "ARCHIVE"), (800, "ARCHIVE")]:
    matched = sum(age >= min_age for min_age, _, _ in RULES)
    print(f"{age:4}  {current:14} {matched:13}  {decide(age, current)}")

# Minimum storage duration: an object moved to Coldline and deleted 10 days later
stored, minimum = 10, MINIMUM_DAYS["COLDLINE"]
print(f"\nColdline object deleted after {stored} days: charged as if stored for {max(stored, minimum)} days")

 age  current class  rules matched  action taken
  10  STANDARD                   0  no action
  45  STANDARD                   1  SetStorageClass NEARLINE
 120  NEARLINE                   2  SetStorageClass COLDLINE
 120  STANDARD                   2  SetStorageClass COLDLINE
 400  COLDLINE                   3  SetStorageClass ARCHIVE
 400  ARCHIVE                    3  no action
 800  ARCHIVE                    4  Delete

Coldline object deleted after 10 days: charged as if stored for 90 days

Reading the rows:

• At 120 days, two rules match and Coldline wins, whether the object is still in Standard or already in Nearline.
• At 400 days an object already in Archive gets no action. Three rules match, and none can move it anywhere colder.
• At 800 days all four match and the object is deleted.

Three operational details from the documentation:

• Changes take time. A change to a lifecycle configuration can take up to 24 hours to take effect, and the old rules may still act in that window. Shortening a deletion age is safe; lengthening one can still delete objects under the old rule for a day.
• Lifecycle transitions do not trigger early deletion charges. A class change made by a lifecycle rule is exempt. A class change you make yourself, by rewriting the object, is not.
• Try rules on a prefix first. Google's advice is to test on development data, or on a small subset selected with a prefix condition.

Or let Autoclass decide

With Autoclass enabled, the bucket manages classes itself: objects that are not read move to colder classes, and an object that is read moves back to Standard (Google Cloud: Autoclass). By default the coldest class it uses is Nearline; Archive can be chosen. Objects under 128 KiB always stay in Standard. Lifecycle rules that set a storage class cannot be used on an Autoclass bucket.

Use lifecycle rules when access follows age, as it does for dated data files. Use Autoclass when you cannot predict access.

Protection against deletion

Soft delete is on by default. A deleted or overwritten object is kept for 7 days and can be restored; the period can be set from 7 to 90 days, or the feature disabled (Google Cloud: soft delete). It also protects against deleting a whole bucket.

Google attaches a warning: for a bucket that mainly holds temporary data, soft delete can significantly increase storage costs, because everything deleted is still stored and billed for the retention period. A staging bucket whose files are rewritten every hour is the classic case. Shorten or disable it there, deliberately.

Object versioning keeps earlier versions of an object when it is replaced. Google now recommends soft delete over versioning for protection against accidental deletion.

Access control

Two systems exist, and Google recommends one (Google Cloud: access control):

• Uniform bucket-level access, the recommended option, uses IAM alone. Permissions are granted on the bucket, or on groups of objects sharing a name prefix.
• Fine-grained access adds per-object access control lists, a legacy system kept for compatibility. Running two systems side by side makes it harder to see who can read what.

Enable uniform access on every new bucket. After 90 days it becomes permanent. For giving someone outside your organisation temporary access to one object, use a signed URL, not a public bucket. IAM on Google Cloud covers roles and policies.

A layout for a data platform

Keep them as separate buckets, in the same region as the compute, with names that say nothing sensitive. Separate buckets give separate permissions, lifecycle rules and cost lines. From the landing bucket, data usually moves on to the warehouse; BigQuery explained and loading data into BigQuery cover that step, and data lake vs data warehouse vs lakehouse the pattern.

Mistakes to avoid

• A cold class for short-lived data. The minimum duration is billed in full.
• A multi-region bucket read from one region.
• Soft delete left at default on a churning staging bucket.
• Sensitive words in a bucket name.
• Per-object ACLs on new buckets.
• A public bucket to share one file.
• Lifecycle rules applied to production without a trial on a prefix.

The same ideas under other names appear in Amazon S3 and Azure Data Lake Storage.

Google Cloud and data lake design are covered in the Data Engineering course.

Sources

• Storage classes (Google Cloud Documentation) — Google Cloud
• About Cloud Storage buckets (Google Cloud Documentation) — Google Cloud
• Bucket locations (Google Cloud Documentation) — Google Cloud
• Object Lifecycle Management (Google Cloud Documentation) — Google Cloud
• Cloud Storage pricing (Google Cloud) — Google Cloud
• Soft delete overview (Google Cloud Documentation) — Google Cloud
• Autoclass (Google Cloud Documentation) — Google Cloud
• Overview of access control (Google Cloud Documentation) — Google Cloud