GCP Logging and Monitoring with Cloud Operations

A pipeline that crashes is the easy case: something is red, and someone looks. The hard case is the pipeline that does not run at all, or runs and loads nothing, and tells nobody. GCP logging and monitoring exist to turn both cases into a message that reaches a person. The services collect most of the data without being asked. The decisions are yours: what to keep, what to count and what deserves an alert.

Google's documentation calls the family Google Cloud Observability (Google Cloud: Observability overview). You will also meet the older names Cloud Operations and Stackdriver, and the second survives in the address of that documentation.

Three kinds of signal

For batch pipelines, logs and metrics do nearly all the work. Traces matter more for services that answer requests.

Where logs go

Cloud Logging collects logs from Google Cloud resources automatically (Google Cloud: Cloud Logging overview). What happens next is decided by the Log Router, which every project has. The router passes each entry to a set of sinks, and each sink sends matching entries to a destination (Google Cloud: route log entries).

Two sinks and two log buckets exist from the start:

Retention on a bucket you control can be set anywhere from 1 to 3650 days (Google Cloud: configure log buckets).

You can add sinks of your own. A sink can send entries to another log bucket, to Cloud Storage for cheap long-term keeping, to BigQuery for analysis, or to Pub/Sub for another system. Two rules are worth remembering. Every sink evaluates every entry independently, so one entry can go to several places. And a sink is not retroactive: it routes only what arrives after it is created.

Writing logs worth reading

On Cloud Run, anything a container writes to standard output or standard error is collected. If a line is a JSON object, Cloud Logging parses it into a structured entry. Two properties get special treatment: severity sets the level of the entry, and message becomes its display text (Google Cloud: logging in Cloud Run).

That is all structured logging requires: print one JSON object per event. The payoff is that every other property becomes a field you can filter on. A filter in the Logging query language compares fields and joins the comparisons with AND and OR (Google Cloud: Logging query language):

resource.type = "cloud_run_job" AND
resource.labels.job_name = "trips-hourly" AND
severity >= ERROR AND
jsonPayload.hour = 9

The same filter works in the Logs Explorer, in a sink, in a log-based metric and on the command line:

gcloud logging read 'resource.type="cloud_run_job" AND severity>=ERROR' \
  --project analytics-prod --limit 10

The Logs Explorer shows individual entries and does not aggregate. For counts and trends over logs, Observability Analytics runs SQL on a log bucket.

Audit logs: who did what

Audit logs are written by Google Cloud itself, not by your code. They answer who did what, where and when (Google Cloud: Cloud Audit Logs).

Data Access logs are off elsewhere because they can be very large, and turning them on can add to the bill. Reading them needs the Private Logs Viewer role, not the ordinary Logs Viewer. See GCP IAM.

From logs to numbers, and from numbers to alerts

A log-based metric turns log entries into a metric. A counter counts the entries that match a filter. A distribution extracts a number, such as a duration, from each one (Google Cloud: log-based metrics). Once it is a metric, it can be charted and watched like any other, alongside the metrics Cloud Monitoring already collects for most services (Google Cloud: Cloud Monitoring overview).

An alerting policy has a condition and one or more notification channels, such as email, Slack or PagerDuty. When the condition is met, Monitoring opens an alert and notifies the channels. The conditions that matter for pipelines (Google Cloud: alerting overview):

• Metric threshold. A value is above or below a line for a period.
• Metric absence. A time series has had no data for a period, up to 24 hours.
• Log-based. A particular message has appeared in the logs.

The failure that writes nothing

The script models an hourly pipeline for one day. Each run writes JSON log lines in the form above. The row counts are real: New York yellow taxi trips for each hour of 15 January 2024 (NYC TLC: trip record data). Two things go wrong, both invented. The load for hour 9 fails with an error. The loads for hours 14, 15 and 16 never start, as if the scheduler had stopped.

The script then applies two rules to its own log lines: a filter for errors, and an absence rule on the success message. It is a model in Python of what those rules compute, not Cloud Logging or Cloud Monitoring.

import json
from datetime import datetime, timedelta

import pandas as pd

trips = pd.read_parquet("../../dedata/yellow_tripdata_2024-01.parquet",
                        columns=["tpep_pickup_datetime"])
day = trips[(trips.tpep_pickup_datetime >= "2024-01-15")
            & (trips.tpep_pickup_datetime < "2024-01-16")]
rows_by_hour = day.tpep_pickup_datetime.dt.hour.value_counts().sort_index()

FAILED, NOT_RUN = {9}, {14, 15, 16}   # one load fails loudly; three never start
lines = []


def log(severity, message, when, **fields):
    """One JSON object per line, as a container writes to standard output."""
    lines.append(json.dumps({"severity": severity, "message": message,
                             "time": when.strftime("%Y-%m-%dT%H:%M:%SZ"), **fields}))


for hour, rows in rows_by_hour.items():
    if hour in NOT_RUN:
        continue
    run_at = datetime(2024, 1, 15, hour) + timedelta(minutes=65)
    log("INFO", "load started", run_at, hour=hour)
    if hour in FAILED:
        log("ERROR", "load failed", run_at + timedelta(seconds=40), hour=hour,
            error="schema mismatch: fare_amount")
        continue
    log("INFO", "load finished", run_at + timedelta(minutes=2), hour=hour, rows=int(rows))

print("three of the log lines as written:")
for line in lines[:2] + [l for l in lines if "ERROR" in l]:
    print(" ", line)

entries = [json.loads(line) for line in lines]
print()
print("entries by severity:", pd.Series([e["severity"] for e in entries]).value_counts().to_dict())

print()
print('filter  severity >= ERROR')
for e in entries:
    if e["severity"] == "ERROR":
        print(f"  {e['time']}  {e['message']}  hour={e['hour']}  {e['error']}")

print()
print('absence rule  no "load finished" entry for 90 minutes')
finished = [datetime.strptime(e["time"], "%Y-%m-%dT%H:%M:%SZ")
            for e in entries if e["message"] == "load finished"]
for earlier, later in zip(finished, finished[1:]):
    if later - earlier > timedelta(minutes=90):
        opened = earlier + timedelta(minutes=90)
        print(f"  alert opens {opened:%H:%M}, last success {earlier:%H:%M}, "
              f"next success {later:%H:%M}, silent for {later - earlier}")

three of the log lines as written:
  {"severity": "INFO", "message": "load started", "time": "2024-01-15T01:05:00Z", "hour": 0}
  {"severity": "INFO", "message": "load finished", "time": "2024-01-15T01:07:00Z", "hour": 0, "rows": 2237}
  {"severity": "ERROR", "message": "load failed", "time": "2024-01-15T10:05:40Z", "hour": 9, "error": "schema mismatch: fare_amount"}

entries by severity: {'INFO': 41, 'ERROR': 1}

filter  severity >= ERROR
  2024-01-15T10:05:40Z  load failed  hour=9  schema mismatch: fare_amount

absence rule  no "load finished" entry for 90 minutes
  alert opens 10:37, last success 09:07, next success 11:07, silent for 2:00:00
  alert opens 15:37, last success 14:07, next success 18:07, silent for 4:00:00

• The error filter finds one incident. The failed load at 10:05 wrote an ERROR entry, and a log-based alert on that severity would report it at once.
• The absence rule finds two. It opens at 10:37 for the failed load, and again at 15:37 for the three runs that never happened.
• The second incident has no error to find. Of 42 entries, 41 are INFO and one is ERROR. A pipeline that does not start writes nothing, and a rule that waits for an error waits forever.
• The cost of the absence rule is delay. It reports 90 minutes after the last success. The error alert was faster for the failure it could see.

So a pipeline needs both: an alert when an error appears, and an alert when success stops appearing. The second depends on logging success in the first place. A run that finishes silently cannot be missed when it is absent.

What it costs

At list prices read on 2 October 2026 (Google Cloud: Observability pricing):

• Logging charges USD 0.50 per GiB stored in log buckets, with the first 50 GiB per project each month free and 30 days of storage included.
• Keeping logs longer than 30 days costs USD 0.01 per GiB per month.
• The required bucket, named _Required, is free.
• Routing has no charge of its own, though the destination charges for what it stores.
• Monitoring does not charge for most Google Cloud metrics. Chargeable metrics, including log-based ones you define, are billed by volume after 150 MiB per billing account.

For most learning projects this is zero. On a busy system, the usual surprise is one chatty component logging every row. Add an exclusion filter to the _Default sink for what nobody reads. See cost controls and budgets.

A starting setup for one pipeline

1. Print one JSON line per event, with severity, message and the fields you will filter on: pipeline, run date, row counts.
2. Log a success line at the end of every run.
3. Create a counter metric on that success line, and an absence alert on the metric.
4. Add a log-based alert on ERROR entries.
5. Send both to a channel the team reads.
6. Decide retention on purpose. Route what must be kept for years to Cloud Storage.
7. Look at log volume once a month and exclude what is never read.

The same pattern fits an orchestrated pipeline: see building a data pipeline on GCP and Cloud Composer. For the equivalent on another cloud, see AWS CloudWatch for monitoring data pipelines.

Common mistakes

• Alerting only on errors.
• Logging free text. A sentence can be searched; a field can be filtered, counted and charted.
• Putting personal data or secrets in log lines. Logs are read by more people than the tables they describe.
• Alerting on everything. A channel that fires ten times a day is muted within a week.
• Assuming logs are kept. The _Default bucket keeps 30 days unless you change it.
• Creating a sink after the incident. It will not contain the incident.

Monitoring pipelines is part of the Data Engineering course.

Sources

• Google Cloud Observability overview (Google Cloud Documentation) — Google Cloud
• Cloud Logging overview (Google Cloud Documentation) — Google Cloud
• Route log entries (Google Cloud Documentation) — Google Cloud
• Configure log buckets (Google Cloud Documentation) — Google Cloud
• Logging and viewing logs in Cloud Run (Google Cloud Documentation) — Google Cloud
• Logging query language (Google Cloud Documentation) — Google Cloud
• Cloud Audit Logs overview (Google Cloud Documentation) — Google Cloud
• Log-based metrics overview (Google Cloud Documentation) — Google Cloud
• Cloud Monitoring overview (Google Cloud Documentation) — Google Cloud
• Alerting overview (Google Cloud Documentation) — Google Cloud
• Google Cloud Observability pricing (Google Cloud) — Google Cloud
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission