GCP IAM Explained: Roles, Policies and Security

Every action on Google Cloud, reading a table, creating a bucket, starting a job, is checked first. The system that does the checking is Identity and Access Management, and GCP IAM reduces to one question: who can do what on which resource. Most access problems, including the frustrating ones where a user can see a table and still cannot query it, are explained by the few rules below.

The three parts

An access grant has three components (Google Cloud: IAM overview):

• A principal: the identity being given access. A person's Google Account, a Google group, or a service account for a workload.
• A role: a collection of permissions.
• A resource: the thing access is granted on. A project, a bucket, a dataset.

You grant a role to a principal on a resource. You cannot grant a single permission directly; permissions only ever arrive bundled in roles.

Permissions and roles

A permission names one operation, in the form service, resource, verb: bigquery.tables.getData, storage.objects.create. They usually correspond one to one with API methods.

A role is a named set of them. There are three kinds (Google Cloud: roles overview):

Google's warning about basic roles is blunt: they include thousands of permissions across all services, and in production they should not be granted unless there is no alternative. A pipeline's service account with the Editor role on a project can modify nearly everything in it. That is the most common serious mistake on the platform.

Allow policies

Roles are granted through an allow policy, a document attached to a resource. It holds a list of role bindings; each binding ties one role to one or more principals (Google Cloud: allow policies).

{
  "bindings": [
    {
      "role": "roles/bigquery.jobUser",
      "members": [ "group:analysts@example.com" ]
    },
    {
      "role": "roles/bigquery.dataViewer",
      "members": [ "user:auditor@example.com" ],
      "condition": {
        "title": "Until end of audit",
        "expression": "request.time < timestamp('2026-12-31T00:00:00Z')"
      }
    }
  ]
}

In daily work you rarely edit the document. You add a binding with a command:

gcloud projects add-iam-policy-binding analytics-prod \
  --member="group:analysts@example.com" \
  --role="roles/bigquery.jobUser"

Two properties of allow policies explain a lot of surprises:

• One policy per resource, which can hold up to 1,500 principals. Granting to groups, not individuals, keeps it readable and well inside the limit.
• Changes are eventually consistent. A new grant can take a little time to take effect everywhere.

Inheritance

Resources are arranged in a hierarchy: an organization contains folders, folders contain projects, projects contain the resources of each service. An allow policy set on a container applies to everything inside it.

The consequence is that access is additive downwards. A role granted on a folder is held on every project and dataset below. To know who can reach a resource, you must read its own policy and every ancestor's. Google calls the union the effective allow policy.

It also means an allow policy lower down cannot take away what was granted higher up. There is no "except this dataset" in allow policies. Removing access means removing the higher grant, or using a deny policy.

Deny policies

A deny policy states that certain principals cannot use certain permissions, whatever roles they hold. It works because of the order of evaluation: IAM always checks deny policies before allow policies (Google Cloud: deny policies). Deny policies are attached to an organization, folder or project and inherited below.

They are a guardrail for exceptions, such as ensuring that no external contractor account can read data in a sensitive folder however roles are later assigned. They are not a replacement for granting less in the first place.

A worked model

The script below models those rules on a small hierarchy: a folder containing a project with two BigQuery datasets. Auditors are granted the Data Viewer role on the folder. Analysts are granted Job User on the project and Data Viewer on one dataset. A deny rule on the folder stops a contractors group reading table data. It is a model written for this article, not Google Cloud IAM, and the role contents are a subset taken from the BigQuery documentation (Google Cloud: BigQuery roles).

# A model of IAM evaluation: deny policies first, then the union of allow policies up the resource hierarchy.
ROLES = {                                                  # a few permissions from each predefined role
    "roles/bigquery.dataViewer": {"bigquery.tables.getData", "bigquery.tables.list"},
    "roles/bigquery.dataEditor": {"bigquery.tables.getData", "bigquery.tables.list", "bigquery.tables.updateData"},
    "roles/bigquery.jobUser":    {"bigquery.jobs.create"},
}
PARENT = {"dataset:sales": "project:analytics", "dataset:hr": "project:analytics",
          "project:analytics": "folder:data", "folder:data": "organization"}
GROUPS = {"user:asha": {"group:analysts"}, "user:ben": {"group:auditors"},
          "user:carl": {"group:analysts", "group:contractors"}, "serviceAccount:loader": set()}

ALLOW = {                                                  # resource -> role bindings
    "folder:data":       [("roles/bigquery.dataViewer", {"group:auditors"})],
    "project:analytics": [("roles/bigquery.jobUser", {"group:analysts", "serviceAccount:loader"})],
    "dataset:sales":     [("roles/bigquery.dataViewer", {"group:analysts"}),
                          ("roles/bigquery.dataEditor", {"serviceAccount:loader"})],
}
DENY = {"folder:data": [({"group:contractors"}, {"bigquery.tables.getData"})]}


def ancestors(resource):
    while resource:
        yield resource
        resource = PARENT.get(resource)


def check(principal, permission, resource):
    identities = {principal} | GROUPS[principal]
    for level in ancestors(resource):                      # deny policies are checked before allow policies
        for denied, permissions in DENY.get(level, []):
            if identities & denied and permission in permissions:
                return f"DENIED  by a deny rule on {level}"
    for level in ancestors(resource):                      # allow policies are inherited from every ancestor
        for role, members in ALLOW.get(level, []):
            if identities & members and permission in ROLES[role]:
                return f"allowed by {role} on {level}"
    return "DENIED  no role grants this permission"


for principal, permission, resource in [
        ("user:asha", "bigquery.jobs.create",       "project:analytics"),
        ("user:asha", "bigquery.tables.getData",    "dataset:sales"),
        ("user:asha", "bigquery.tables.getData",    "dataset:hr"),
        ("user:ben",  "bigquery.tables.getData",    "dataset:hr"),
        ("user:ben",  "bigquery.jobs.create",       "project:analytics"),
        ("serviceAccount:loader", "bigquery.tables.updateData", "dataset:sales"),
        ("user:carl", "bigquery.tables.getData",    "dataset:sales")]:
    print(f"{principal:22} {permission:27} {resource:18} {check(principal, permission, resource)}")

user:asha              bigquery.jobs.create        project:analytics  allowed by roles/bigquery.jobUser on project:analytics
user:asha              bigquery.tables.getData     dataset:sales      allowed by roles/bigquery.dataViewer on dataset:sales
user:asha              bigquery.tables.getData     dataset:hr         DENIED  no role grants this permission
user:ben               bigquery.tables.getData     dataset:hr         allowed by roles/bigquery.dataViewer on folder:data
user:ben               bigquery.jobs.create        project:analytics  DENIED  no role grants this permission
serviceAccount:loader  bigquery.tables.updateData  dataset:sales      allowed by roles/bigquery.dataEditor on dataset:sales
user:carl              bigquery.tables.getData     dataset:sales      DENIED  by a deny rule on folder:data

Each line teaches something:

• Asha, an analyst, can read the sales dataset and not the HR dataset. The role was granted on one dataset, not on the project. This is least privilege working.
• Ben, an auditor, can read HR data through a role granted two levels up. Nothing on the dataset mentions him. Inheritance is why "I checked the dataset's permissions" is never the whole answer.
• Ben cannot run a query. He has permission to read table data and no permission to create a job, and in BigQuery a query is a job. The Job User role, which carries that permission, can only be granted on a project, folder or organization. This is the cause of the classic complaint that a user "has access to the table but gets an error running a query".
• Carl is denied, though he is an analyst. He is also in the contractors group, and the deny rule is checked first.

Conditions

A role binding can carry a condition, an expression that must be true for the binding to apply. The example policy above grants a role until a date. Conditions can also test the resource's name or type (Google Cloud: IAM Conditions).

Two limits to remember: conditions cannot be used with the legacy basic roles, and a conditional binding does not restrict an unconditional one. If a principal already holds the same role without a condition, adding a conditional binding changes nothing.

Temporary, conditional access is the right answer to "I need production access to investigate this", in place of a permanent grant that nobody remembers to remove.

Least privilege in practice

• Grant to groups. People join and leave groups; policies stay unchanged.
• Grant at the lowest level that works. A dataset, not a project. A bucket, not a project.
• Use predefined roles, and the narrowest one. A reader does not need an editor role.
• Separate data access from job access. In BigQuery, most users need a job role on the project and a data role on specific datasets.
• Never grant basic roles to service accounts. Service accounts and workload identity covers identities for pipelines.
• Separate projects by environment. Development, test and production as different projects makes "who can touch production" a short list.
• Review grants on a schedule, starting at the top of the hierarchy, where one binding reaches furthest.

Diagnosing "permission denied"

1. Read the error. It names the missing permission, which is more precise than a role name.
2. Find which role contains it. The role pages list permissions.
3. Check the right level. Some roles can only be granted on a project; some permissions apply to a job, not to the data.
4. Look up the hierarchy for where access was, or was not, granted.
5. Consider a deny policy if the roles look right.
6. Wait briefly. A grant made a minute ago may not have propagated.

For the same ideas on another cloud, see AWS IAM explained. On this one, the resources you will secure first are covered in Google Cloud Storage and BigQuery explained.

Google Cloud and data governance are covered in the Data Engineering course.

Sources

• IAM overview (Google Cloud Documentation) — Google Cloud
• Roles overview (Google Cloud Documentation) — Google Cloud
• Understanding allow policies (Google Cloud Documentation) — Google Cloud
• Deny policies (Google Cloud Documentation) — Google Cloud
• Overview of IAM Conditions (Google Cloud Documentation) — Google Cloud
• BigQuery IAM roles and permissions (Google Cloud Documentation) — Google Cloud