Cloud Functions on GCP: Serverless Explained

Most of a data platform runs on a schedule. Some of it should not: a file lands in a bucket, a message arrives, a webhook is called, and something small needs to happen now. Standing up a server for ten lines of code is absurd. A function is the answer: you supply the code and say what triggers it, and the platform runs it when the trigger fires and charges only for the time it runs.

On Google Cloud that product was called Cloud Functions for years, and most people still call it that. Google now calls it Cloud Run functions.

What changed with the name

The rename reflects a real change underneath. A function is now built into a container and deployed as a service on Cloud Run (Google Cloud: compare Cloud Run functions). There are two versions:

For anything new, use the current version. The first generation is what older tutorials describe, and its limits are the ones people remember.

Two kinds of function

You write a function with the open source Functions Framework and choose one of two signatures (Google Cloud: write Cloud Run functions):

• HTTP functions have a URL and answer requests. Use them for webhooks and small APIs.
• Event-driven functions receive an event, in the standard CloudEvents format, when something happens in your project: an object is created in a bucket, a message is published to a topic.

Data engineering mostly uses the second kind.

A function for a landing bucket

The function below reacts to a file arriving in a landing bucket and starts a load into BigQuery. It is the event-driven alternative to a scheduled load. The interesting part is not the load but everything around it.

import re

import functions_framework

DATA_FILE = re.compile(r"^trips/dt=(\d{4}-\d{2}-\d{2})/[^/]+\.parquet$")
submitted = {}          # stands in for BigQuery, which remembers the job IDs it has accepted


def start_load_job(job_id, uri, day):
    """In production this would call the BigQuery client with this job ID. Here it records the request."""
    if job_id in submitted:
        return "duplicate"                     # BigQuery rejects a job ID it has already accepted
    submitted[job_id] = (uri, day)
    return "submitted"


@functions_framework.cloud_event
def on_file(cloud_event):
    """Runs once for every object finalized in the landing bucket."""
    data = cloud_event.data
    bucket, name, generation = data["bucket"], data["name"], data["generation"]
    match = DATA_FILE.match(name)
    if not match:
        print(f"ignored    {name}")
        return
    day = match.group(1)
    job_id = f"load_trips_{day.replace('-', '')}_{generation}"       # the same event gives the same ID
    print(f"{start_load_job(job_id, f'gs://{bucket}/{name}', day):10} {name} -> {job_id}")

The same Functions Framework that runs it in the cloud can run it on your machine. The script below loads the function and sends it five events shaped like Cloud Storage's "object finalized" events. Nothing is deployed, and the load is a stub.

# Runs the function in main.py locally with Google's Functions Framework and sends it CloudEvents
# shaped like Cloud Storage "object finalized" events. Nothing is deployed and no Google service is
# called; the BigQuery load is a stub inside main.py.
import functions_framework

app = functions_framework.create_app(target="on_file", source="main.py", signature_type="cloudevent")
client = app.test_client()

EVENTS = [                                           # (event id, object name, generation)
    ("1001", "trips/dt=2026-10-01/part-000.parquet", "1759300000000001"),
    ("1001", "trips/dt=2026-10-01/part-000.parquet", "1759300000000001"),   # delivered a second time
    ("1002", "trips/dt=2026-10-01/_SUCCESS", "1759300000000002"),
    ("1003", "trips/2026-10-01/part-000.parquet", "1759300000000003"),      # wrong folder layout
    ("1004", "trips/dt=2026-10-02/part-000.parquet", "1759386400000004"),
]
for event_id, name, generation in EVENTS:
    response = client.post("/", json={"bucket": "example-landing", "name": name, "generation": generation},
                           headers={"ce-id": event_id, "ce-specversion": "1.0",
                                    "ce-type": "google.cloud.storage.object.v1.finalized",
                                    "ce-source": "//storage.googleapis.com/projects/_/buckets/example-landing"})
    assert response.status_code == 200, response.status_code

import main                                           # the module the framework loaded
print(f"\nevents received: {len(EVENTS)}; load jobs started: {len(main.submitted)}")

submitted  trips/dt=2026-10-01/part-000.parquet -> load_trips_20261001_1759300000000001
duplicate  trips/dt=2026-10-01/part-000.parquet -> load_trips_20261001_1759300000000001
ignored    trips/dt=2026-10-01/_SUCCESS
ignored    trips/2026-10-01/part-000.parquet
submitted  trips/dt=2026-10-02/part-000.parquet -> load_trips_20261002_1759386400000004

events received: 5; load jobs started: 2

Five events arrived and two loads were started. Each line shows a rule that every event-driven function needs:

• A real file starts a load.
• The same event delivered twice does not start two. Event delivery is at-least-once: Google's documentation says handlers might receive duplicate events and should be idempotent (Google Cloud: retry events). Here the job ID is built from the file's date and generation number, so a repeat produces the same ID, and BigQuery accepts a given job ID only once (Google Cloud: load data from Cloud Storage).
• Files that are not data are ignored. A bucket trigger fires for every object, including marker files and temporary files.
• A file in the wrong place is ignored, not guessed at. In production, log it or alert on it.

Deploying and triggering

gcloud run deploy on-file \
  --source . \
  --function on_file \
  --base-image python314 \
  --region asia-south1

gcloud eventarc triggers create on-file-trigger \
  --location=asia-south1 \
  --destination-run-service=on-file \
  --destination-run-region=asia-south1 \
  --event-filters="type=google.cloud.storage.object.v1.finalized" \
  --event-filters="bucket=example-landing" \
  --service-account=loader@analytics-prod.iam.gserviceaccount.com

The commands follow the documented forms (Google Cloud: deploy a Cloud Run function; Google Cloud: trigger functions from Cloud Storage). The first builds the code into a container and deploys it. The second creates the trigger: an Eventarc rule that routes one type of event, from one bucket, to the function.

Give the function and the trigger a dedicated service account with only the access they need. See service accounts and workload identity.

Retries and failures

When a function fails, the event can be delivered again. With a trigger created from the command line, retries are on by default, using exponential backoff between 10 and 600 seconds within a 24-hour window. Two consequences:

• A retried function must be safe to rerun, which is the idempotency already described.
• A permanently failing event will be retried until it expires. Configure a dead-letter topic so that such events are kept for inspection instead of vanishing.

Google suggests disabling retries while developing, to avoid runaway loops, and enabling them in production.

Limits that shape what you build

A function is for short work:

• Time. HTTP functions can run for up to 60 minutes. That is a ceiling, not an invitation.
• State. Instances come and go. Anything kept in memory, like the dictionary in the example, is lost when an instance stops and is not shared between instances. Keep state in a database, a bucket or the service you call.
• Cold starts. The first request to a new instance waits for it to start. Keep dependencies light, create clients once in global scope so later requests reuse them, and set a minimum number of instances if latency matters (Google Cloud: development tips).

The pattern that respects all three: let the function decide and delegate. It validates the event and asks BigQuery, Dataflow or a workflow to do the heavy work, as the example does.

What it costs

Functions are billed as Cloud Run: for CPU and memory while handling requests, rounded up to the nearest 100 milliseconds, plus a charge per request. A monthly free tier covers 2 million requests, 180,000 vCPU-seconds and 360,000 GiB-seconds (Google Cloud: Cloud Run pricing). A function that runs briefly a few thousand times a day usually costs nothing. The costs to watch are the services it calls, and a retry loop that calls them repeatedly.

Good uses in data work

• Start a load or a pipeline when a file arrives.
• Validate or route incoming files before anything processes them.
• Receive a webhook from an outside system and publish it to Pub/Sub.
• Send a notification when a pipeline fails.
• Glue between services where no built-in integration exists.

And poor uses: transforming large files inside the function, long chains of functions calling functions, and anything that needs to run for hours. For those, see Cloud Run vs Cloud Functions vs App Engine and the Dataflow tutorial. For the scheduled alternative to an event-driven load, see Cloud Storage to BigQuery.

Event-driven pipelines are covered in the Data Engineering course.

Sources

• Compare Cloud Run functions (Google Cloud Documentation) — Google Cloud
• Write Cloud Run functions (Google Cloud Documentation) — Google Cloud
• Deploy a Cloud Run function (Google Cloud Documentation) — Google Cloud
• Trigger functions from Cloud Storage using Eventarc (Google Cloud Documentation) — Google Cloud
• Retry events (Google Cloud Documentation) — Google Cloud
• General development tips (Google Cloud Documentation) — Google Cloud
• Cloud Run pricing (Google Cloud) — Google Cloud
• Load CSV data from Cloud Storage (Google Cloud Documentation) — Google Cloud