Pub/Sub Explained: Messaging and Streaming on GCP

In a system of any size, the service that knows something happened is rarely the only one that needs to know. An order is placed: billing, stock, analytics and email all care. If the order service calls each of them directly, it must know them all, wait for them all and fail when any of them fails. Google Cloud Pub/Sub removes that knot. The order service publishes one message and moves on. Whoever is interested subscribes. The publisher and the subscribers never meet.

The four terms

Pub/Sub is an asynchronous messaging service that separates the services producing messages from those processing them (Google Cloud: what is Pub/Sub?). Everything in it is built from four things:

• A topic is a named channel that messages are published to.
• A publisher sends messages to a topic and does not know who reads them.
• A subscription is attached to one topic and represents one stream of deliveries. A topic can have many.
• A subscriber receives messages from a subscription and acknowledges them.

Two rules follow from this design and explain most questions about the service (Google Cloud: subscription overview):

• Every subscription gets every message. Three subscriptions on one topic each receive a full copy. Three workers sharing one subscription divide the messages between them.
• A subscription only sees messages published after it was created. Create the subscription before you start publishing.

Three ways to receive

Pull is the default choice for data work. Push suits a service that should wake up only when there is something to do. An export subscription needs no subscriber at all, which is the simplest route when messages only need storing (Google Cloud: choose a subscription type).

Acknowledgment, and why messages repeat

Delivery is a lease. Pub/Sub hands a message to a subscriber and waits for an acknowledgment. If none arrives within the acknowledgment deadline, 10 seconds by default and at most 600, it delivers the message again (Google Cloud: subscription properties).

The guarantee is therefore at-least-once. Every message will be delivered; some will be delivered more than once. Google states it directly: any message can be redelivered, so a subscriber must be idempotent.

The script models those rules for six messages. It assumes no deadline extension, and it is a model, not Pub/Sub.

# A model of Pub/Sub's default delivery rules: at-least-once delivery, an acknowledgment deadline,
# redelivery when it passes or when the subscriber rejects the message, and a dead-letter topic
# after a maximum number of delivery attempts. It is not Pub/Sub and uses no Google service.
ACK_DEADLINE = 10                 # seconds, the default
MAX_DELIVERY_ATTEMPTS = 5         # the default when a dead-letter topic is set

# (message id, seconds the subscriber needs, does processing succeed?)
MESSAGES = [("order-1", 2, True), ("order-2", 4, True), ("order-3", 25, True),
            ("order-4", 3, True), ("order-5", 1, False), ("order-6", 12, True)]


def deliver(message_id, seconds_needed, succeeds):
    """Return how many times the message is delivered and what finally happens to it."""
    if not succeeds:                                   # the subscriber rejects it every time
        return MAX_DELIVERY_ATTEMPTS, "sent to the dead-letter topic"
    # The work finishes, but if it takes longer than the deadline the message is sent again meanwhile.
    deliveries = 1 + (seconds_needed - 1) // ACK_DEADLINE
    return deliveries, "acknowledged"


naive_rows, processed_ids = 0, set()
print(f"{'message':9} {'seconds':>7} {'deliveries':>10}  outcome")
for message_id, seconds_needed, succeeds in MESSAGES:
    deliveries, outcome = deliver(message_id, seconds_needed, succeeds)
    print(f"{message_id:9} {seconds_needed:7} {deliveries:10}  {outcome}")
    if succeeds:
        naive_rows += deliveries                       # a subscriber that inserts on every delivery
        processed_ids.add(message_id)                  # a subscriber that remembers the message ID

good = sum(1 for _, _, succeeds in MESSAGES if succeeds)
print(f"\nmessages that should produce a row: {good}")
print(f"rows written by a subscriber that inserts on every delivery: {naive_rows}")
print(f"rows written by an idempotent subscriber, keyed on message ID: {len(processed_ids)}")

message   seconds deliveries  outcome
order-1         2          1  acknowledged
order-2         4          1  acknowledged
order-3        25          3  acknowledged
order-4         3          1  acknowledged
order-5         1          5  sent to the dead-letter topic
order-6        12          2  acknowledged

messages that should produce a row: 5
rows written by a subscriber that inserts on every delivery: 8
rows written by an idempotent subscriber, keyed on message ID: 5

• Four messages finish inside the deadline and are delivered once.
• order-3 takes 25 seconds. Its lease expires twice while the work is still running, so it is delivered three times. The work succeeded every time.
• order-5 can never be processed. It is retried until the limit of five attempts and then forwarded to a dead-letter topic, where it stops blocking the others.
• A subscriber that inserts a row on every delivery writes 8 rows for 5 good messages. One that remembers the message ID writes 5.

Two defences exist, and you need both. The first is to prevent needless redelivery: the high-level client libraries extend the lease automatically while your code is still working, by default for up to an hour (Google Cloud: lease management). The second is to make processing idempotent, because redelivery can still happen after a crash or a restart. Use the message ID or a business key to detect work already done, or write with a merge instead of an insert.

Stronger guarantees, when you need them

Exactly-once delivery can be enabled on a subscription. Pub/Sub then does not redeliver a message that was successfully acknowledged or that is still outstanding. It is supported only on pull subscriptions and within a region (Google Cloud: exactly-once delivery). It protects delivery, not your side effects: if your code writes to a database and crashes before acknowledging, the message returns.

Ordering is off by default. To receive related messages in order, publish them with the same ordering key, in the same region, and enable ordering on the subscription. Throughput per key is limited to 1 MBps, so choose a key such as a customer ID that spreads the load, not one constant value (Google Cloud: order messages).

Both features cost throughput and simplicity. Use them where the business requires it and nowhere else.

Setting it up

gcloud pubsub topics create orders
gcloud pubsub topics create orders-dead-letter

gcloud pubsub subscriptions create orders-to-warehouse \
  --topic=orders \
  --dead-letter-topic=orders-dead-letter \
  --max-delivery-attempts=5

gcloud pubsub topics publish orders \
  --message='{"order_id": 1001, "amount": 49.5}' \
  --ordering-key=customer-17

gcloud pubsub subscriptions create orders-ordered \
  --topic=orders --enable-message-ordering

gcloud pubsub subscriptions create orders-exactly-once \
  --topic=orders --enable-exactly-once-delivery

The commands follow the documented forms (Google Cloud: dead-letter topics). A dead-letter topic is a property of the subscription, not of the topic, and it needs its own subscription or its messages will sit unread. The attempt limit can be set from 5 to 100 and is approximate.

Retention and replay

Unacknowledged messages are kept for 7 days by default. A subscription that nobody uses expires after 31 days (Google Cloud: quotas and limits).

Acknowledged messages are gone, unless you configured message retention, which on a topic can be up to 31 days. With retention, the seek feature can rewind a subscription to a timestamp or a snapshot and deliver the messages again (Google Cloud: replay with seek). That is how you recover from a subscriber bug that acknowledged messages it mishandled: fix the code, seek back, reprocess. It only works if retention was switched on before the incident.

What it costs

Pub/Sub charges for throughput: bytes published plus bytes delivered. The first 10 GiB each month is free, and the list price beyond that is USD 40 per TiB. BigQuery export subscriptions are USD 50 per TiB with no separate ingestion charge on the BigQuery side. Retained messages cost USD 0.27 per GiB a month (Google Cloud: Pub/Sub pricing).

Two consequences. Each additional subscription adds delivery throughput, so ten subscriptions on a busy topic cost far more than one. And message size matters: publish identifiers and the fields consumers need, not whole documents. A single message can be up to 10 MB, which is a limit, not a target.

How it differs from Kafka

Engineers who know Kafka look for partitions and offsets, and Pub/Sub has neither. It leases individual messages and tracks each one, where Kafka has consumers read partitions in order. The practical effects:

• Scaling consumers needs no repartitioning. Add workers to a subscription and they share the load.
• Ordering is opt-in, through ordering keys.
• Replay is by time or snapshot, not by offset.

Design checklist

1. Create subscriptions before publishing.
2. Make every subscriber idempotent.
3. Give every subscription a dead-letter topic, and monitor it.
4. Turn on topic retention if you may ever need to reprocess.
5. Enable ordering or exactly-once only where required.
6. Keep messages small and include an event timestamp and an ID.

To process the stream, see the Dataflow tutorial. For the full path into the warehouse, see Pub/Sub to Dataflow to BigQuery, and for writing straight to a table, loading data into BigQuery.

Messaging and streaming are covered in the Data Engineering course.

Sources

• What is Pub/Sub? (Google Cloud Documentation) — Google Cloud
• Subscription overview (Google Cloud Documentation) — Google Cloud
• Choose a subscription type (Google Cloud Documentation) — Google Cloud
• Subscription properties (Google Cloud Documentation) — Google Cloud
• Extend ack time with lease management (Google Cloud Documentation) — Google Cloud
• Exactly-once delivery (Google Cloud Documentation) — Google Cloud
• Order messages (Google Cloud Documentation) — Google Cloud
• Dead-letter topics (Google Cloud Documentation) — Google Cloud
• Replay and purge messages with seek (Google Cloud Documentation) — Google Cloud
• Pub/Sub quotas and limits (Google Cloud Documentation) — Google Cloud
• Pub/Sub pricing (Google Cloud) — Google Cloud