GCP Networking Basics for Data Engineers

A data engineer can work on Google Cloud for months without touching a network. BigQuery, Cloud Storage and Pub/Sub are reached through APIs, and none of them asks which subnet you are in. Then one day a security review says pipeline workers may not have public addresses, or a job must read a database that has no public endpoint, and GCP networking stops being someone else's subject.

This guide covers the part of it a data engineer needs: enough to read a network diagram, size a subnet and understand why a job cannot reach what it needs.

When the network matters

The dividing line is whether something runs on virtual machines in your project. If it does, it lives in a network.

Networks and subnets

A VPC network is your private network inside Google Cloud. It is a global resource. A subnet is a range of IP addresses in one region, and resources take their addresses from it (Google Cloud: VPC networks).

Every new project starts with a network called default. It is an auto mode network: one subnet in every region, created for you, all taken from the block 10.128.0.0/9. The alternative is a custom mode network, which starts empty and holds only the subnets you create. Google's documentation describes custom mode as better suited to production, and the reasons are practical: you choose the regions and the ranges, and nothing appears that might collide with a network you connect later.

Three rules about ranges save trouble (Google Cloud: subnets):

• A range can be expanded but not shrunk or replaced. Start with room to grow.
• Four addresses in each primary range are unusable. Google Cloud keeps the first two and the last two.
• Ranges must not overlap with anything the network will be connected to: another VPC network, an office network, the auto mode block.

Sizing a subnet

Each Dataflow worker takes an address from the job's subnet, so the size of the subnet caps the number of workers. Google's own example is a /24, which gives 252 usable addresses (Google Cloud: specify a network and subnetwork).

The script checks a five-subnet plan with Python's standard ipaddress module. It counts usable addresses, looks for overlaps with the auto mode block and with an office range that will be connected later, and finds the smallest subnet for a given number of workers. Nothing is created on Google Cloud.

import ipaddress

RESERVED = 4   # Google Cloud keeps the first two and last two addresses of a primary range
AUTO_MODE = ipaddress.ip_network("10.128.0.0/9")   # used by auto mode networks
OFFICE = ipaddress.ip_network("10.20.0.0/16")      # an on-premises range to connect later

PLAN = {"asia-south1  pipelines": "10.10.0.0/24",
        "asia-south1  spark": "10.10.4.0/26",
        "us-central1  pipelines": "10.10.8.0/28",
        "us-central1  notebooks": "10.20.8.0/29",
        "europe-west1 pipelines": "10.130.0.0/24"}

print("subnet                    range            addresses  usable  problem")
for name, cidr in PLAN.items():
    net = ipaddress.ip_network(cidr)
    problems = [label for label, other in (("auto mode block", AUTO_MODE), ("office range", OFFICE))
                if net.overlaps(other)]
    print(f"{name:25} {cidr:16} {net.num_addresses:>9}  {net.num_addresses - RESERVED:>6}  "
          f"{'overlaps ' + ', '.join(problems) if problems else '-'}")

print()
print("smallest subnet for a job that may scale to N workers")
for workers in (10, 50, 252, 253, 1000):
    prefix = 29
    while 2 ** (32 - prefix) - RESERVED < workers:
        prefix -= 1
    print(f"  {workers:>5} workers -> /{prefix}  ({2 ** (32 - prefix) - RESERVED} usable)")

subnet                    range            addresses  usable  problem
asia-south1  pipelines    10.10.0.0/24           256     252  -
asia-south1  spark        10.10.4.0/26            64      60  -
us-central1  pipelines    10.10.8.0/28            16      12  -
us-central1  notebooks    10.20.8.0/29             8       4  overlaps office range
europe-west1 pipelines    10.130.0.0/24          256     252  overlaps auto mode block

smallest subnet for a job that may scale to N workers
     10 workers -> /28  (12 usable)
     50 workers -> /26  (60 usable)
    252 workers -> /24  (252 usable)
    253 workers -> /23  (508 usable)
   1000 workers -> /22  (1020 usable)

• Two of the five ranges are wrong, and neither looks wrong. 10.20.8.0/29 sits inside the office range, and 10.130.0.0/24 sits inside the auto mode block. Both would work until the day the networks are connected.
• A subnet of 16 addresses holds 12 workers. That is the /28 in the plan, and an autoscaling job that wants more simply stops scaling.
• The step from 252 to 253 workers doubles the subnet. Sizes come in powers of two, so plan for the peak, not the average.

Firewall rules

A firewall rule allows or denies connections to or from VMs. Four facts explain most behaviour (Google Cloud: VPC firewall rules):

• Incoming traffic is denied unless a rule allows it. Every network has an implied deny rule for ingress.
• Outgoing traffic is allowed unless a rule denies it. There is an implied allow rule for egress.
• Rules are stateful. If a connection is allowed, the replies are allowed too.
• Priority is a number from 0 to 65535, and the lowest matching number wins.

The default network comes with rules that allow SSH, RDP and ping from any address, and all traffic between its own VMs. That is convenient for a first experiment and one more reason not to run production on it.

In a custom network, workers cannot talk to each other until you allow it. Dataflow workers that do not use Dataflow Shuffle or Streaming Engine need TCP ports 12345 and 12346 open among themselves (Google Cloud: Dataflow internet access and firewall rules). A rule can target VMs by network tag, so that it applies to the workers and nothing else.

Private workers that can still reach their data

By default, Dataflow workers get an external and an internal address. Turning the external address off is a common security requirement, and it raises an obvious problem: BigQuery and Cloud Storage are reached at public API addresses.

Private Google Access solves it. It is a setting on a subnet. With it on, VMs that have only internal addresses can reach Google APIs and services (Google Cloud: Private Google Access). Without it, a private worker cannot read a bucket in its own project.

Cloud NAT covers the rest of the internet. It lets resources without external addresses make outbound connections, for example to download a Python package, and it does not accept unsolicited connections coming in (Google Cloud: Cloud NAT overview).

Private Service Connect covers managed services that live in a network of their own, such as Cloud SQL. It gives the service an internal address inside your network, and the traffic stays within Google Cloud (Google Cloud: Private Service Connect).

The commands for a network set up this way, in the forms Google documents (Google Cloud: configure Private Google Access):

gcloud compute networks create data-net --subnet-mode=custom

gcloud compute networks subnets create pipelines-asia-south1 \
  --network=data-net --region=asia-south1 --range=10.10.0.0/24 \
  --enable-private-ip-google-access

gcloud compute firewall-rules create allow-dataflow-workers \
  --network=data-net --direction=ingress --action=allow \
  --source-tags=dataflow --target-tags=dataflow \
  --rules=tcp:12345-12346

A Python pipeline then runs with --subnetwork set to that subnet and --no_use_public_ips. See the Dataflow tutorial.

Perimeters and shared networks

Two more terms appear in larger organisations.

VPC Service Controls draws a perimeter around the resources of services such as BigQuery and Cloud Storage. Data inside cannot be copied to a resource outside, even by someone whose permissions would allow it. It works independently of IAM, and Google recommends using both (Google Cloud: VPC Service Controls overview). For a data engineer, the visible effect is an error when a job in one project reads a dataset inside a perimeter it does not belong to. The fix is a perimeter change, not a role. See GCP IAM.

Shared VPC puts the network in one host project and lets other projects, called service projects, use its subnets (Google Cloud: Shared VPC). The network team keeps control of ranges and firewall rules, and your project supplies only the workloads. If you are told to "use the shared subnet", this is what is meant, and the Dataflow service account needs the Compute Network User role on that subnet. See service accounts and workload identity.

What moving data costs

Network charges are for data leaving a place. At list prices read on 2 October 2026 (Google Cloud: network pricing):

The practical rule for pipelines follows from the table: keep the compute in the same region as the data. A cluster in one region reading a bucket in another pays for every gigabyte it reads, and waits longer for it. See Google Cloud Storage for bucket locations and cost controls and budgets for watching the bill.

When a job cannot connect

Work through the path in order:

1. Is the worker in the subnet you think? Check the job's network settings and region.
2. Has the subnet run out of addresses? A job that will not scale often has.
3. Does the worker have an external address? If not, is Private Google Access on for that subnet?
4. Does it need the internet? Then it needs Cloud NAT.
5. Is a firewall rule missing? In a custom network, nothing is allowed in by default.
6. Is the target inside a service perimeter?
7. Only then, check permissions. The two usually look different: a network block shows as a timeout or a refused connection, and a permission error is an explicit denial that names the permission.

Common mistakes

• Running production on the default network, with SSH open to every address.
• Choosing ranges without asking what the network will be connected to.
• Sizing a subnet for today's job.
• Turning off external addresses and forgetting Private Google Access.
• Putting compute and data in different regions.
• Treating a perimeter error as a permission problem and granting roles until something else breaks.

Working with cloud platforms, Google Cloud among them, is part of the Data Engineering course.

Sources

• VPC networks (Google Cloud Documentation) — Google Cloud
• Subnets (Google Cloud Documentation) — Google Cloud
• VPC firewall rules (Google Cloud Documentation) — Google Cloud
• Private Google Access (Google Cloud Documentation) — Google Cloud
• Configure Private Google Access (Google Cloud Documentation) — Google Cloud
• Cloud NAT overview (Google Cloud Documentation) — Google Cloud
• Private Service Connect (Google Cloud Documentation) — Google Cloud
• Overview of VPC Service Controls (Google Cloud Documentation) — Google Cloud
• Shared VPC (Google Cloud Documentation) — Google Cloud
• Specify a network and subnetwork (Google Cloud Documentation) — Google Cloud
• Configure internet access and firewall rules (Google Cloud Documentation) — Google Cloud
• Network pricing (Google Cloud) — Google Cloud