AWS IAM Explained: Users, Roles and Policies

Every request to AWS, whether it comes from a person clicking in the console, a script, or one AWS service calling another, is checked against the same system before anything happens. That system is AWS Identity and Access Management. AWS describes IAM as the service that controls who is authenticated, signed in, and authorised, has permissions, to use resources (IAM docs: what is IAM).

IAM is where access is granted, so it is where access mistakes are made. It is also where most "it does not work" problems in data pipelines turn out to live. Understanding four nouns and one rule covers the bulk of it.

The four nouns

User. An identity for one person or application, with long-term credentials: a password, or access keys.

Group. A collection of users. Permissions attached to a group apply to its members. Groups exist to avoid attaching policies to individuals.

Role. An identity with permissions that is not tied to one person. The guide's wording: a role is intended to be assumable by anyone who needs it, has no standard long-term credentials, and provides temporary security credentials when assumed (IAM docs: roles).

Policy. A document that says what is allowed or denied. Most policies are JSON.

There is also the root user, the identity an account starts with, which has complete access to everything. AWS strongly recommends not using it for everyday tasks.

Reading a policy

A policy is a list of statements. Each statement has the same parts:

• Effect: Allow or Deny.
• Action: the API operations, such as s3:GetObject.
• Resource: the things the statement applies to, written as ARNs.
• Condition (optional): circumstances in which it applies, such as a source IP range or a required tag.
• Principal: who the statement applies to. This appears only in resource-based policies.

Policies come in two main kinds (IAM docs: policies and permissions). Identity-based policies are attached to a user, group or role and say what that identity may do. Resource-based policies are attached to a resource, such as an S3 bucket, and say who may do what to it. A role's trust policy, which states who is allowed to assume the role, is a resource-based policy too.

The one rule

Within an account, a request is allowed if a policy allows it, and an explicit deny in any applicable policy overrides the allow (IAM docs: policy evaluation). In practice:

1. Nothing is allowed unless some policy allows it.
2. An explicit Deny always wins.
3. Otherwise an Allow permits the request.

So permissions are built by adding allows, and an explicit deny is the tool for a rule that must hold whatever else is granted.

Least privilege, demonstrated

The policy below gives an analyst exactly two things: read objects under one prefix of a bucket, and list the bucket. The script creates the user, attaches the policy, and then makes five requests using that user's own credentials. It runs against moto, a local emulator, with IAM enforcement switched on, so the decisions come from the emulator's simplified evaluation and not from AWS itself.

import json, os
import boto3
from botocore.exceptions import ClientError
from moto import mock_aws
from moto.core import set_initial_no_auth_action_count

os.environ.update(AWS_ACCESS_KEY_ID="admin", AWS_SECRET_ACCESS_KEY="admin", AWS_DEFAULT_REGION="us-east-1")

POLICY = {
    "Version": "2012-10-17",
    "Statement": [
        {"Sid": "ReadCurated", "Effect": "Allow",
         "Action": ["s3:GetObject"], "Resource": "arn:aws:s3:::demo-lake/curated/*"},
        {"Sid": "ListBucket", "Effect": "Allow",
         "Action": ["s3:ListBucket"], "Resource": "arn:aws:s3:::demo-lake"},
    ],
}


@mock_aws
@set_initial_no_auth_action_count(8)          # the first 8 calls below are setup, made as an administrator
def main():
    admin_s3, iam = boto3.client("s3"), boto3.client("iam")
    admin_s3.create_bucket(Bucket="demo-lake")                                           # 1
    admin_s3.put_object(Bucket="demo-lake", Key="curated/trips.csv", Body=b"a,b\n1,2\n")  # 2
    admin_s3.put_object(Bucket="demo-lake", Key="raw/customers.csv", Body=b"name\nx\n")   # 3
    iam.create_user(UserName="analyst")                                                  # 4
    iam.create_policy(PolicyName="read-curated", PolicyDocument=json.dumps(POLICY))      # 5
    iam.attach_user_policy(UserName="analyst",
                           PolicyArn="arn:aws:iam::123456789012:policy/read-curated")    # 6
    key = iam.create_access_key(UserName="analyst")["AccessKey"]                         # 7
    iam.get_user(UserName="analyst")                                                     # 8

    # From here on, requests are checked against IAM policies
    s3 = boto3.client("s3", aws_access_key_id=key["AccessKeyId"], aws_secret_access_key=key["SecretAccessKey"])

    def attempt(label, call):
        try:
            call()
            print(f"{label:38} allowed")
        except ClientError as e:
            print(f"{label:38} {e.response['Error']['Code']}")

    attempt("GetObject  curated/trips.csv", lambda: s3.get_object(Bucket="demo-lake", Key="curated/trips.csv"))
    attempt("GetObject  raw/customers.csv", lambda: s3.get_object(Bucket="demo-lake", Key="raw/customers.csv"))
    attempt("PutObject  curated/new.csv", lambda: s3.put_object(Bucket="demo-lake", Key="curated/new.csv", Body=b"x"))
    attempt("DeleteObject curated/trips.csv", lambda: s3.delete_object(Bucket="demo-lake", Key="curated/trips.csv"))
    attempt("ListBucket demo-lake", lambda: s3.list_objects_v2(Bucket="demo-lake"))


main()

GetObject  curated/trips.csv           allowed
GetObject  raw/customers.csv           AccessDenied
PutObject  curated/new.csv             AccessDenied
DeleteObject curated/trips.csv         AccessDenied
ListBucket demo-lake                   allowed

Two of the five requests succeed, the two the policy names. The other three are refused without any Deny statement being written:

• Reading raw/customers.csv fails because the allowed resource is curated/* only.
• Writing and deleting fail because only s3:GetObject and s3:ListBucket were allowed.

One detail in the policy is worth copying. s3:GetObject applies to objects, so its resource ends in /curated/*. s3:ListBucket applies to the bucket itself, so its resource is the bucket ARN with no path. Putting both actions against one resource is among the most common reasons a policy that "should work" does not.

Why roles are preferred to users

Access keys for a user are long-lived secrets. They get committed to repositories, copied into configuration files and forgotten. A role has no long-term secret. Whoever assumes it receives credentials that expire.

AWS's best practices make this the default in both directions (IAM docs: security best practices):

• Human users should sign in through federation with an identity provider and use temporary credentials. AWS recommends IAM Identity Center for managing that centrally.
• Workloads should use temporary credentials through IAM roles.

For data engineering, the second point is everything. A Glue job, a Lambda function, an EMR cluster and a Redshift workgroup each run with a role, often called an execution or service role. A role involves two policies:

When a pipeline fails with AccessDenied, the cause is nearly always that the role's permissions policy lacks an action or resource, or that the trust policy does not name the service trying to assume it. AWS Lambda explained covers execution roles for functions.

Practices AWS recommends

From the same best-practices page:

• Protect the root user and do not use it for daily work.
• Require multi-factor authentication.
• Apply least-privilege permissions: grant only what a task needs.
• Start with AWS managed policies and move toward least privilege. Managed policies are broad; narrow them as you learn what the workload uses.
• Use IAM Access Analyzer to generate policies from actual access activity and to validate policies.
• Use conditions to restrict access further.
• Review regularly and remove unused users, roles, permissions and credentials.
• Rotate access keys where long-term credentials are unavoidable.

Mistakes that cause incidents

• Wildcards everywhere. "Action": "*" on "Resource": "*" is administrator access, whatever the policy is called.
• One role shared by many jobs. A bug in any of them can then reach everything.
• Access keys in code or in a repository.
• Granting broad access "to make it work" and never narrowing it.
• Forgetting the resource side. A bucket policy or a KMS key policy can refuse a request that the identity's policy allows.
• Confusing who may assume a role with what the role may do.

Where IAM stops

IAM controls access to AWS actions and resources. It does not understand a table's columns or rows; that is the job of AWS Lake Formation, which adds database-style grants on top. Reading encrypted data needs permission on the key as well as on the data; see securing data on AWS with encryption and KMS. And network reachability is a separate layer again; see VPC basics for data engineers. A request has to pass all of them.

AWS data services, with the access controls that pipelines depend on, are covered in the Data Engineering course.

Sources

• What is IAM? (IAM User Guide) — AWS Documentation
• IAM roles (IAM User Guide) — AWS Documentation
• Policies and permissions in AWS Identity and Access Management (IAM User Guide) — AWS Documentation
• Policy evaluation for requests within a single account (IAM User Guide) — AWS Documentation
• Security best practices in IAM (IAM User Guide) — AWS Documentation