GCP Interview Questions for Data Engineers

GCP interview questions for data engineers usually go deepest on streaming, because Dataflow, Google's managed service for Apache Beam, is where the hardest concepts live: event time, watermarks and late data. This guide runs real Beam pipelines locally on a busy hour of New York taxi pickups (NYC TLC: trip record data) to show how windows count events and what happens to an event that arrives late. It then covers Pub/Sub delivery, Cloud Storage classes and IAM from Google's documentation. BigQuery has its own set in BigQuery interview questions.

import glob
import logging
import os
import shutil

import apache_beam as beam
import pandas as pd
from apache_beam.options.pipeline_options import PipelineOptions, StandardOptions
from apache_beam.testing.test_stream import TestStream
from apache_beam.transforms import trigger, window
from apache_beam.utils.timestamp import Timestamp
from apache_beam.utils.windowed_value import PaneInfoTiming

logging.getLogger("apache_beam").setLevel(logging.ERROR)
trips = pd.read_parquet("yellow_tripdata_2025-01.parquet", columns=["tpep_pickup_datetime", "PULocationID"])
hour = trips[(trips.tpep_pickup_datetime >= "2025-01-16 08:00") & (trips.tpep_pickup_datetime < "2025-01-16 09:00")]
events = [(int(z), ts.timestamp()) for z, ts in zip(hour.PULocationID, hour.tpep_pickup_datetime)]   # NYC TLC, a real busy hour
print(f"G1 {len(events):,} real pickups between 08:00 and 09:00 on 16 January 2025")

def read_back(prefix):                                          # pipeline output goes to files, read it back sorted
    lines = [line.strip() for f in glob.glob(prefix + "*") for line in open(f)]
    shutil.rmtree(os.path.dirname(prefix), ignore_errors=True)
    return sorted(lines)


with beam.Pipeline() as p:                                      # runs locally on Beam's built-in runner
    (p | beam.Create(events)
       | beam.Map(lambda e: window.TimestampedValue(1, e[1]))  # use the pickup time as event time
       | beam.WindowInto(window.FixedWindows(15 * 60))
       | beam.CombineGlobally(sum).without_defaults()
       | beam.ParDo(lambda n, w=beam.DoFn.WindowParam: [f"{w.start.to_utc_datetime():%H:%M} {n}"])
       | beam.io.WriteToText("out_windows/part"))
print("   15-minute event-time windows:", read_back("out_windows/part"))

start = Timestamp.of(pd.Timestamp("2025-01-16 08:00", tz="UTC").timestamp())


def late_data_run(allowed_lateness):
    shutil.rmtree(f"out_late_{allowed_lateness}", ignore_errors=True)
    stream = (TestStream()
              .add_elements([window.TimestampedValue("trip", start + 60), window.TimestampedValue("trip", start + 120)])
              .advance_watermark_to(start + 15 * 60 + 30)       # the 08:00-08:15 window's end has passed
              .add_elements([window.TimestampedValue("late trip", start + 300)])   # an 08:05 event arriving late
              .advance_watermark_to_infinity())
    opts = PipelineOptions()
    opts.view_as(StandardOptions).streaming = True
    with beam.Pipeline(options=opts) as p:
        (p | stream
           | beam.WindowInto(window.FixedWindows(15 * 60), allowed_lateness=allowed_lateness,
                             trigger=trigger.AfterWatermark(late=trigger.AfterCount(1)),
                             accumulation_mode=trigger.AccumulationMode.ACCUMULATING)
           | beam.combiners.Count.Globally().without_defaults()
           | beam.ParDo(Record(f"out_late_{allowed_lateness}/part.txt")))
    return read_back(f"out_late_{allowed_lateness}/part")


class Record(beam.DoFn):                                         # one line per firing: count and pane timing
    def __init__(self, path):
        self.path = path

    def process(self, count, pane=beam.DoFn.PaneInfoParam):
        os.makedirs(os.path.dirname(self.path), exist_ok=True)
        with open(self.path, "a") as f:
            print(f"firing {pane.index}: {count} trips ({PaneInfoTiming.to_string(pane.timing)})", file=f)


print("\nG2 an 08:05 event that arrives after the watermark passed 08:15")
print("   allowed lateness 0 s:  ", late_data_run(0))
print("   allowed lateness 600 s:", late_data_run(600))

G1 6,677 real pickups between 08:00 and 09:00 on 16 January 2025
   15-minute event-time windows: ['08:00 1606', '08:15 1632', '08:30 1734', '08:45 1705']

G2 an 08:05 event that arrives after the watermark passed 08:15
   allowed lateness 0 s:   ['firing 0: 2 trips (ON_TIME)']
   allowed lateness 600 s: ['firing 0: 2 trips (ON_TIME)', 'firing 1: 3 trips (LATE)']

G1. What is the difference between event time and processing time?

Event time is when something happened; processing time is when the pipeline sees it. Windowing by event time puts each pickup in the 15-minute window of its own timestamp, whatever order the records arrive in. The busy hour from 08:00 on 16 January 2025 held 6,677 pickups: 1,606, 1,632, 1,734 and 1,705 in its four windows. Processing-time windows would count whatever happened to arrive in each 15 minutes of wall-clock time, which shifts with every delay in the network or pipeline. For business metrics, the answer is almost always event time.

G2. What is a watermark, and what happens to late data?

A watermark is the pipeline's estimate that all events up to a certain time have arrived; when it passes the end of a window, the window's result can be emitted. Data that arrives after that is late. The test stream sent two trips for 08:00 to 08:15, moved the watermark past 08:15, then delivered an event stamped 08:05. With no allowed lateness, the window fired once with 2 trips and the late event was dropped. With 10 minutes of allowed lateness, the window fired on time with 2, then fired again, marked LATE, with 3. Beam's default, a single global window, discards late data (Apache Beam: programming guide).

What interviewers want to hear: you choose allowed lateness from how late real data arrives, decide whether late firings replace or add to earlier results, and make the destination handle updates.

G3. Does Pub/Sub deliver each message exactly once?

By default it delivers at least once, so duplicates are possible and consumers must be idempotent. Exactly-once delivery can be enabled within a region for pull subscriptions, including StreamingPull; push and export subscriptions do not support it (Pub/Sub: exactly-once delivery). Even then, the processing must still be designed to tolerate retries, because acknowledgements can fail.

G4. Which Cloud Storage class would you use?

Standard for data read often, then Nearline, Coldline and Archive for data read less and less. The colder classes cost less to store but add retrieval fees and minimum storage durations: 30 days for Nearline, 90 for Coldline and 365 for Archive (Cloud Storage: storage classes). Deleting or rewriting an object earlier is still billed for the minimum. Lifecycle rules can move objects between classes automatically as they age.

G5. How do you grant access safely on GCP?

Grant the narrowest role that does the job, to a service account for each workload, at the smallest scope. Google's own guidance warns that basic roles such as Editor include thousands of permissions and should not be granted in production (IAM: use IAM securely). A Dataflow job that reads one bucket and writes one dataset needs those permissions only, not project-wide editing.

G6. Dataflow or Dataproc?

Dataflow for Beam pipelines, managed and autoscaled, a natural fit for streaming. Dataproc for existing Spark or Hadoop jobs that you want to run with little change. The choice usually follows the code you already have and how much cluster management you want to own. For new streaming work on Google Cloud, Dataflow is the usual default; for a team migrating hundreds of existing Spark jobs, Dataproc avoids a rewrite.

G7. How do you orchestrate pipelines on Google Cloud?

With Apache Airflow, which Google runs as a managed service. Google's documentation explains that Cloud Composer is evolving into Managed Service for Apache Airflow, a fully managed orchestration service built on the open-source project (Google Cloud: Managed Airflow). Expect to hear both names in interviews. Airflow schedules and coordinates the steps, such as loading files, starting Dataflow jobs and running BigQuery transformations; it should not do the heavy processing itself.

How to answer GCP questions

• Speak in event time and explain watermarks and allowed lateness.
• Assume at-least-once delivery unless exactly-once is configured, and design idempotent sinks.
• Match storage classes and IAM roles to real access patterns.

Streaming is taught in the Dataflow tutorial, Pub/Sub explained and the Pub/Sub to Dataflow to BigQuery pattern. Access is covered in GCP IAM explained, and the other clouds in Azure interview questions for data engineers.

Cloud data pipelines are part of the Data Engineering course.

Sources

• Apache Beam Programming Guide — The Apache Software Foundation
• Exactly-once delivery (Pub/Sub documentation) — Google Cloud
• Storage classes (Cloud Storage documentation) — Google Cloud
• Use IAM securely (IAM documentation) — Google Cloud
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission
• Managed Airflow overview (Google Cloud documentation) — Google Cloud