VPC Basics for Data Engineers on AWS

A data engineer can go a long time on AWS without thinking about networks. S3, Athena and the Glue Data Catalog are reached over AWS's public endpoints and simply work. Then a job has to read from a database, or a security review says nothing may touch the internet, and suddenly a Glue job hangs and then fails with a timeout that says nothing useful.

Those failures are almost always networking, and almost always one of four things. This guide covers the VPC concepts behind them. It is not a networking course; it is the part a pipeline builder needs.

What a VPC is

Amazon VPC lets you launch resources in a logically isolated virtual network that you define (VPC docs: what is Amazon VPC). The pieces:

• VPC: the network, with a range of private IP addresses.
• Subnet: a slice of that range, living in exactly one Availability Zone.
• Route table: rules deciding where traffic from a subnet goes.
• Gateways and endpoints: the ways out of the VPC.
• Security groups: firewalls around individual resources.

Databases such as RDS and Redshift, EMR clusters, MSK brokers and Glue jobs that use a connection all get network interfaces inside your subnets. S3 does not; it lives outside, which is the cause of the most common surprise.

Address planning

A VPC's address range is written as a CIDR block. 10.20.0.0/16 means the first 16 bits are fixed, leaving 65,536 addresses. Subnets are smaller blocks inside it; AWS allows subnet sizes from /28 to /16, and reserves the first four addresses and the last one in every subnet (VPC docs: subnet CIDR blocks).

This matters to data work because distributed jobs use many addresses. Every node of a cluster and every network interface a job creates takes one. A subnet that is too small produces failures that look like capacity errors.

A small data platform network

The script plans the addresses with Python's ipaddress module, then creates the network with boto3 against moto, a local emulator of the EC2 API. No real network exists; the point is the structure.

import ipaddress, os
import boto3
from moto import mock_aws

os.environ.update(AWS_ACCESS_KEY_ID="testing", AWS_SECRET_ACCESS_KEY="testing",
                  AWS_DEFAULT_REGION="us-east-1")

# --- Plan the address space before creating anything ---
vpc_cidr = ipaddress.ip_network("10.20.0.0/16")
plan = {"public-a": "10.20.0.0/24", "private-a": "10.20.10.0/23", "private-b": "10.20.12.0/23"}
print(f"VPC {vpc_cidr}: {vpc_cidr.num_addresses:,} addresses")
for name, cidr in plan.items():
    net = ipaddress.ip_network(cidr)
    assert net.subnet_of(vpc_cidr)
    print(f"  {name:10} {cidr:15} {net.num_addresses:4} addresses, {net.num_addresses - 5:4} usable (AWS reserves 5)")


@mock_aws
def main():
    ec2 = boto3.client("ec2")
    vpc = ec2.create_vpc(CidrBlock=str(vpc_cidr))["Vpc"]["VpcId"]
    zones = {"public-a": "us-east-1a", "private-a": "us-east-1a", "private-b": "us-east-1b"}
    subnets = {n: ec2.create_subnet(VpcId=vpc, CidrBlock=c, AvailabilityZone=zones[n])["Subnet"]["SubnetId"]
               for n, c in plan.items()}

    # Public subnet: a route to an internet gateway
    igw = ec2.create_internet_gateway()["InternetGateway"]["InternetGatewayId"]
    ec2.attach_internet_gateway(InternetGatewayId=igw, VpcId=vpc)
    public_rt = ec2.create_route_table(VpcId=vpc)["RouteTable"]["RouteTableId"]
    ec2.create_route(RouteTableId=public_rt, DestinationCidrBlock="0.0.0.0/0", GatewayId=igw)
    ec2.associate_route_table(RouteTableId=public_rt, SubnetId=subnets["public-a"])

    # Private subnets: no internet route, but a gateway endpoint so jobs can reach S3
    private_rt = ec2.create_route_table(VpcId=vpc)["RouteTable"]["RouteTableId"]
    for name in ("private-a", "private-b"):
        ec2.associate_route_table(RouteTableId=private_rt, SubnetId=subnets[name])
    ec2.create_vpc_endpoint(VpcId=vpc, ServiceName="com.amazonaws.us-east-1.s3",
                            VpcEndpointType="Gateway", RouteTableIds=[private_rt])

    # Security groups: the database accepts PostgreSQL only from the jobs' group
    jobs = ec2.create_security_group(GroupName="etl-jobs", Description="ETL jobs", VpcId=vpc)["GroupId"]
    db = ec2.create_security_group(GroupName="warehouse-db", Description="database", VpcId=vpc)["GroupId"]
    ec2.authorize_security_group_ingress(GroupId=db, IpPermissions=[
        {"IpProtocol": "tcp", "FromPort": 5432, "ToPort": 5432, "UserIdGroupPairs": [{"GroupId": jobs}]}])

    print()
    for label, rt in (("public", public_rt), ("private", private_rt)):
        routes = ec2.describe_route_tables(RouteTableIds=[rt])["RouteTables"][0]["Routes"]
        targets = [("local" if r.get("GatewayId") == "local" else
                    "internet gateway" if str(r.get("GatewayId", "")).startswith("igw-") else "S3 gateway endpoint")
                   for r in routes]
        print(f"{label:8} route table -> {sorted(set(targets))}")
    rule = ec2.describe_security_groups(GroupIds=[db])["SecurityGroups"][0]["IpPermissions"][0]
    print(f"warehouse-db inbound: tcp {rule['FromPort']} from security group "
          f"{'etl-jobs' if rule['UserIdGroupPairs'][0]['GroupId'] == jobs else '?'}")


main()

VPC 10.20.0.0/16: 65,536 addresses
  public-a   10.20.0.0/24     256 addresses,  251 usable (AWS reserves 5)
  private-a  10.20.10.0/23    512 addresses,  507 usable (AWS reserves 5)
  private-b  10.20.12.0/23    512 addresses,  507 usable (AWS reserves 5)

public   route table -> ['internet gateway', 'local']
private  route table -> ['S3 gateway endpoint', 'local']
warehouse-db inbound: tcp 5432 from security group etl-jobs

The plan gives the two private subnets 507 usable addresses each, in two Availability Zones, and keeps the public subnet small because almost nothing belongs in it.

Public and private: it is only a route

Nothing about a subnet is inherently public. The definition is in the route table: a public subnet has a direct route to an internet gateway; a private subnet does not (VPC docs: subnets). The output shows it: the public route table contains the internet gateway, the private one does not.

Data resources belong in private subnets. A database with a public address and an open security group is exposed to anyone who scans for it.

How private resources reach things

A private subnet has no way out by default. That is the first cause of mysterious failures: a job in a private subnet that needs S3, the Glue API, or a package repository, and cannot get there. There are two remedies.

NAT gateway. It lets instances in a private subnet connect to services outside the VPC while preventing outside services from initiating connections to them (VPC docs: NAT gateways). It sits in a public subnet. It solves general internet access, and it is a billed resource; check VPC pricing for its hourly and data processing charges.

VPC endpoints. An endpoint connects the VPC privately to an AWS service without an internet gateway or NAT device. For S3 there is a gateway endpoint, added as a target in a route table, at no additional charge (VPC docs: gateway endpoints for S3). The private route table in the example has one. Other services are reached through interface endpoints, which are charged.

For a data platform the S3 gateway endpoint is close to mandatory. Pipelines move large volumes to and from S3; sending that through a NAT gateway adds NAT data processing charges for no benefit.

Security groups: who may talk to whom

A security group is a set of rules attached to a resource. Two properties are worth knowing (VPC docs: infrastructure security):

• It is stateful. If a request is allowed out, the reply is allowed back automatically.
• It works at the level of the resource, unlike a network ACL, which works at subnet level and is stateless.

The best habit is in the example's last line. The database's group allows port 5432 not from an IP range but from the jobs' security group. Any resource in etl-jobs can connect; nothing else can; and no rule needs editing when addresses change.

Glue has a specific requirement worth remembering: for Glue's components to communicate, its security group needs a self-referencing inbound rule for all TCP ports, meaning the group allows traffic from itself (Glue docs: VPC setup for JDBC). A missing self-reference is a classic cause of a Glue connection that fails to start.

The four usual faults

When a job in a VPC cannot reach something, check in this order:

1. No route. The subnet's route table has no path to the destination: no endpoint for S3, no NAT for the internet.
2. Security group. The target's inbound rules do not allow the source, or the source's outbound rules were tightened.
3. Wrong subnet or zone. The job runs in a subnet that cannot reach the database's subnet, or an endpoint is not associated with that subnet's route table.
4. No free addresses. The subnet is full.

A fifth, less common: DNS. Private endpoints rely on name resolution inside the VPC; a custom DNS setup can break it.

The symptom of the first two is a timeout, not an error message, because packets are silently dropped. A job that fails fast with "access denied" is an IAM problem. A job that hangs and then times out is a network problem. That distinction saves hours; see AWS IAM explained for the other side.

What to ask for, if a platform team owns the network

Often you will not build the VPC yourself. Know what to request:

• Two or more private subnets in different Availability Zones, with enough free addresses for your largest job.
• An S3 gateway endpoint on their route tables.
• Interface endpoints, or NAT, for the other AWS APIs your jobs call.
• A security group for your jobs, and rules on each data store allowing that group.
• VPC Flow Logs enabled, so that dropped connections can be seen.

Networking is one of three layers a request must pass, with permissions and encryption; securing data on AWS with encryption and KMS covers the third. For services that live in these subnets, see Amazon Redshift explained and Amazon EMR explained.

AWS data services, and the cloud foundations they run on, are covered in the Data Engineering course.

Sources

• What is Amazon VPC? (Amazon VPC User Guide) — AWS Documentation
• Subnet CIDR blocks (Amazon VPC User Guide) — AWS Documentation
• Subnets for your VPC (Amazon VPC User Guide) — AWS Documentation
• NAT gateways (Amazon VPC User Guide) — AWS Documentation
• Gateway endpoints for Amazon S3 (AWS PrivateLink) — AWS Documentation
• Infrastructure security in Amazon VPC (Amazon VPC User Guide) — AWS Documentation
• Setting up Amazon VPC for JDBC connections to Amazon RDS data stores from AWS Glue (AWS Glue User Guide) — AWS Documentation