Cost Controls and Budgets on Google Cloud

On your own computer, a careless query costs time. In the cloud it costs money, and the bill arrives after the money is spent. Cost control is therefore part of the engineering, not something finance does afterwards. Google Cloud budgets are the tool most people reach for first, and the one most often misunderstood: by default a budget sends alerts and stops nothing.

Three jobs: see, be told, be stopped

The cost tools do three different jobs. Most unpleasant surprises come from expecting one tool to do another's job.

A team needs all three. Seeing without being told means finding out at the end of the month. Being told without being stopped means an alert at midnight and a loss that runs until morning.

See: reports, export and labels

Reports. The Reports page in the console charts the costs of a billing account. You can group and filter by project, service, SKU, location and label, and it shows a forecast for the period (Google Cloud: billing reports). Start here: the first useful question is always which three things cost the most.

Export to BigQuery. For anything a chart cannot answer, export billing data to a BigQuery dataset, where it arrives through the day. The standard export has cost by service, SKU, project, label and location, and the detailed export adds individual resources. Turn it on when the billing account is created. A dataset in a multi-region location is backfilled for the current and previous month only, and a regional one is not backfilled at all (Google Cloud: export billing data to BigQuery).

Labels. A label is a key and a value attached to a resource, such as team: analytics or env: prod. Labels travel into the billing data, so they let you ask what a team or a pipeline costs. One catch: labels on a project do not propagate to the resources inside it (Google Cloud: labels for projects). Label the buckets, datasets and jobs themselves.

With export and labels in place, cost per team is one query, in the form Google documents (Google Cloud: example queries for billing export):

SELECT
  labels.value AS team,
  SUM(cost) AS cost
FROM `my-project.billing.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX`
LEFT JOIN UNNEST(labels) AS labels
  ON labels.key = "team"
GROUP BY team;

The label key sits in the join condition, not in a filter, so cost with no such label is kept and appears with an empty team. That row is the unlabelled spend, and it is usually the first thing worth fixing.

Be told: budgets and threshold rules

A budget has four parts (Google Cloud: budgets and budget alerts):

• Scope. The whole billing account, or chosen projects, folders, services, or resources carrying one label.
• Amount. A fixed figure, or whatever the previous period cost.
• Period. A month, a quarter, a year or a custom range.
• Threshold rules. Percentages of the amount at which an alert goes out. A new budget starts with 50%, 90% and 100%.

Each rule watches one of two things. An actual rule fires when the cost so far in the period passes the threshold. A forecasted rule fires when the projected cost at the end of the period passes it. The defaults are all actual, and budgets with a custom date range cannot use forecasted rules.

By default the emails go to the billing account's administrators and users, who are often not the people able to fix anything. A budget can also notify Cloud Monitoring notification channels, up to five, and publish to a Pub/Sub topic. See logging and monitoring on GCP for notification channels.

The same budget from the command line, with the flags in the reference (Google Cloud: gcloud billing budgets create):

gcloud billing budgets create \
  --billing-account=0X0X0X-0X0X0X-0X0X0X \
  --display-name="analytics-prod monthly" \
  --budget-amount=300USD \
  --filter-projects=projects/analytics-prod \
  --threshold-rule=percent=0.50 \
  --threshold-rule=percent=0.90 \
  --threshold-rule=percent=1.00 \
  --threshold-rule=percent=1.00,basis=forecasted-spend

Anomaly detection works beside budgets and needs no amount. It compares each project's cost with its own history, flags spikes, shows a root cause analysis and can email them (Google Cloud: cost anomalies). A budget asks whether you have spent more than you planned; anomaly detection asks whether today looks unlike your other days.

What an alert does, and what it does not

The script follows one invented month. A project with a USD 300 budget normally costs USD 8 a day. On day 12, a scheduled query starts scanning a whole table every hour and adds USD 30 a day. The script reports when each threshold rule fires and what the month costs under four responses.

It is a model. The projection is a straight line, not Google's forecast, and it ignores the delay before usage appears in billing data.

DAYS, BUDGET = 30, 300.0
OTHER, QUERIES, RUNAWAY = 3.0, 5.0, 30.0   # US dollars a day
INCIDENT_DAY = 12                           # an hourly query starts scanning a whole table
PRICE_PER_TIB, QUOTA_TIB = 6.25, 2          # on-demand list price and a custom daily quota


def month(fixed_from=None, quota_tib=None):
    """Spend for each day. fixed_from is the first day without the runaway query."""
    days = []
    for day in range(1, DAYS + 1):
        queries = QUERIES
        if day >= INCIDENT_DAY and (fixed_from is None or day < fixed_from):
            queries += RUNAWAY
        if quota_tib is not None:
            queries = min(queries, quota_tib * PRICE_PER_TIB)
        days.append(OTHER + queries)
    return days


RULES = (("100% forecast", 1.0, "forecast"), ("50% actual", 0.5, "actual"),
         ("90% actual", 0.9, "actual"), ("100% actual", 1.0, "actual"))
fired, spent = {}, 0.0
for day, cost in enumerate(month(), start=1):
    spent += cost
    projected = spent / day * DAYS          # a straight line to the end of the month
    for name, share, basis in RULES:
        value = projected if basis == "forecast" else spent
        if name not in fired and value >= BUDGET * share:
            fired[name] = (day, spent, projected)

print(f"budget USD {BUDGET:.0f}; a normal day costs USD {OTHER + QUERIES:.0f}; "
      f"from day {INCIDENT_DAY} a runaway query adds USD {RUNAWAY:.0f} a day")
print()
print("threshold rule   fires on   spent so far   projected month")
for name, _, _ in RULES:
    day, so_far, projected = fired[name]
    print(f"{name:15}  day {day:>2}     {so_far:>10.2f}   {projected:>15.2f}")

late, early = fired["100% actual"][0] + 1, fired["100% forecast"][0] + 1
print()
print("what the month costs                                total   against budget")
for label, days in (("nobody acts on the alerts", month()),
                    ("fixed the day after the 100% actual alert", month(fixed_from=late)),
                    ("fixed the day after the forecast alert", month(fixed_from=early)),
                    (f"nobody acts, but a {QUOTA_TIB} TiB daily query quota", month(quota_tib=QUOTA_TIB))):
    print(f"{label:46} {sum(days):>10.2f}   {sum(days) - BUDGET:>+14.2f}")

budget USD 300; a normal day costs USD 8; from day 12 a runaway query adds USD 30 a day

threshold rule   fires on   spent so far   projected month
100% forecast    day 12         126.00            315.00
50% actual       day 13         164.00            378.46
90% actual       day 16         278.00            521.25
100% actual      day 17         316.00            557.65

what the month costs                                total   against budget
nobody acts on the alerts                          810.00          +510.00
fixed the day after the 100% actual alert          420.00          +120.00
fixed the day after the forecast alert             270.00           -30.00
nobody acts, but a 2 TiB daily query quota         382.50           +82.50

• The forecasted rule fires first, on day 12, the day the problem starts, with USD 126 spent. The 100% actual rule fires on day 17, when the budget is already gone.
• Acting on the forecast keeps the month under budget, at USD 270. Waiting for the 100% alert costs USD 420.
• If nobody acts, the month costs USD 810. The budget did everything it was built to do. It sent four alerts and stopped nothing.
• A daily query quota changes the last row without anyone acting. With a 2 TiB limit, the same neglected month costs USD 382.50.

Alerts and limits do different work. An alert shortens the time until a person decides. A limit bounds the loss while no one is looking. The limit has a price too: once a daily quota is used up, every query in the project fails until it resets, including the ones that matter.

Be stopped: limits that hold

Quotas. A quota restricts how much of a resource a project can use, and in most cases a request beyond it is blocked. Quotas generally apply per project (Google Cloud: Cloud Quotas overview).

BigQuery limits. A maximum bytes billed setting refuses a single query that would scan more than you allow. Custom quotas cap the query data a project, or each user, may process in a day under on-demand pricing (Google Cloud: custom query quotas). Both are covered in BigQuery cost optimization.

Automated responses. A budget connected to a Pub/Sub topic publishes its status several times a day: the cost so far, the budget amount and the highest thresholds passed (Google Cloud: programmatic notifications). A small function subscribed to the topic can post to a chat channel or stop a resource. See Cloud Functions on GCP.

The most drastic response is to disable billing on the project. Google documents it and is blunt about the consequences. Every service in the project stops, including Free Tier ones, and resources might be irretrievably deleted. Because cost data arrives late, it still does not guarantee that spending stays within the budget (Google Cloud: disable billing with notifications). It suits a personal sandbox. It does not suit a project that holds data anyone needs.

Spend caps. A newer kind of budget, in preview, pauses new usage of one eligible service in one project when its estimated cost passes the amount, until someone lifts the cap. The eligible services include the Gemini API, Gemini Enterprise Agent Platform (formerly Vertex AI), Cloud Run and Cloud Run functions. Storage, virtual machines and other persistent resources keep charging, and enforcement is not instant (Google Cloud: spend cap budgets).

Lower the rate last

Committed use discounts give lower prices in return for committing to a level of usage or spend for one or three years (Google Cloud: committed use discounts). They reduce the price of usage you are sure about. They are not a control, and a commitment made before the usage is understood locks the waste in.

A starting setup

1. Turn on billing export to BigQuery on the day the billing account is created.
2. Give each project its own budget, and add one for the whole account.
3. Add a forecasted rule to every monthly budget.
4. Send alerts to a channel the team reads.
5. Label resources by team and environment, and look at the unlabelled share each month.
6. Lower the BigQuery daily quotas to an amount you could afford to lose in a day.
7. In a sandbox project, consider an automatic shut-off. In production, do not.

If you are learning, do steps 2 and 3 before creating anything else. See the GCP free tier for what costs nothing in the first place.

Common mistakes

• Treating a budget as a limit. It is a notification.
• One budget for the whole account. A project whose cost doubles is invisible inside a total ten times its size.
• Rules on actual spend only. The 100% alert tells you the money is gone.
• Alerts sent to an inbox nobody reads.
• Labelling the project and assuming its resources inherit the label.
• Wiring an automatic billing shut-off to a project with real data in it.
• Giving budget access to no one on the team. The people who can act need to see the numbers. See GCP IAM.

Working with cloud platforms, Google Cloud among them, is part of the Data Engineering course.

Sources

• Create, edit, or delete budgets and budget alerts (Google Cloud Documentation) — Google Cloud
• gcloud billing budgets create (Google Cloud SDK reference) — Google Cloud
• Analyze billing data and cost trends with Reports (Google Cloud Documentation) — Google Cloud
• Export Cloud Billing data to BigQuery (Google Cloud Documentation) — Google Cloud
• Example queries for Cloud Billing data export (Google Cloud Documentation) — Google Cloud
• Create and update labels for projects (Google Cloud Documentation) — Google Cloud
• View and manage cost anomalies (Google Cloud Documentation) — Google Cloud
• Cloud Quotas overview (Google Cloud Documentation) — Google Cloud
• Create custom query quotas (Google Cloud Documentation) — Google Cloud
• Set up programmatic notifications (Google Cloud Documentation) — Google Cloud
• Disable billing usage with notifications (Google Cloud Documentation) — Google Cloud
• Manage spend cap budgets (Google Cloud Documentation) — Google Cloud
• Committed use discounts (Google Cloud Documentation) — Google Cloud