AWS Lake Formation: Governance for Data Lakes

A data lake starts as one bucket and two people. A year later it is hundreds of tables read by analysts, pipelines, dashboards and a machine learning team, and the question "who can see the salary column?" has no clear answer. On AWS, access to lake data is by default a matter of S3 bucket policies and IAM policies, which speak in terms of buckets and prefixes, not tables and columns.

AWS Lake Formation is the service that closes that gap. It manages fine-grained access to data in S3 and its metadata in the Glue Data Catalog, using a grant and revoke model much like a relational database's (Lake Formation docs: what is Lake Formation).

The problem with S3 and IAM alone

IAM and bucket policies can say "this role may read objects under s3://lake/sales/". They cannot say "this role may read the sales table except the customer email column" or "only rows where region is EU". The file either can be read or cannot.

Teams work around this by copying data into restricted and unrestricted versions, or by writing ever longer policies per prefix. Both approaches drift. Lake Formation's own documentation describes its permissions as replacing complex S3 bucket policies and the corresponding IAM policies.

How a query is authorised

Lake Formation works at two levels: metadata permissions on Data Catalog databases, tables and columns, and storage access to the underlying S3 locations that are registered with it. The flow for a query, as the guide describes it (Lake Formation docs: how it works):

1. Get metadata. A user submits a query to an integrated engine such as Athena, Glue, EMR or Redshift Spectrum. The engine asks the Data Catalog for the table.
2. Check permissions. The catalog checks the user's Lake Formation permissions and returns only the metadata the user may see.
3. Get credentials. If the table's location is registered with Lake Formation, the engine asks Lake Formation for access and receives temporary credentials.
4. Get data. The engine reads S3 with those credentials, applies any column, row or cell filtering, and returns the result.

AWS calls this credential vending. The consequence that matters is that the user needs no S3 permission of their own on the data. Access comes through the engine, with the filter applied. If users keep direct S3 read access to the same bucket, they can bypass every column and row rule by reading the files, so removing that direct access is part of adopting Lake Formation.

What you can grant

Permissions are granted to IAM users and roles on catalog resources: a database, a table, or selected columns. The documented CLI form for a column-level grant, adapted with placeholder names and not run:

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/analyst \
  --permissions "SELECT" \
  --resource '{ "TableWithColumns": {
      "DatabaseName": "sales",
      "Name": "customers",
      "ColumnNames": ["customer_id", "country", "segment"] } }'

After this, a query by the analyst role that selects email from the table fails or omits the column, in every integrated engine, without a second copy of the data.

Data filters go further (Lake Formation docs: data filtering):

Tags for scale

Granting on named tables works for ten tables and becomes unmanageable for a thousand. LF-Tags are the answer the guide recommends for large catalogs. A tag is a key and value, such as classification=restricted or department=sales, attached to databases, tables or columns. Principals are granted access to tag values, not to individual resources (Lake Formation docs: tag-based access control).

The effect is that policy follows classification. Tag a new table's columns correctly and the right people can read them with no new grant. Note that LF-Tags are separate from IAM tags; the two are not interchangeable.

The default that trips people up

Many teams turn on Lake Formation, grant a few permissions, and find that nothing changes: everyone can still read everything. The cause is a deliberate default. To stay backward compatible with Glue, Lake Formation initially grants the Super permission to a group called IAMAllowedPrincipals on existing catalog resources, and enables "use only IAM access control" for new ones. The guide is explicit about the effect: access is controlled solely by IAM policies, and individual Lake Formation permissions are not in effect (Lake Formation docs: default settings).

Lake Formation governs a table only when all of these hold:

• The S3 location is registered with Lake Formation.
• The IAMAllowedPrincipals grant has been removed from the database and table.
• Principals have Lake Formation grants, and no direct S3 access that bypasses them.

Adopting it without breaking things

Changing those defaults for a whole catalog at once is how production pipelines stop on a Monday morning. Hybrid access mode exists for this: it lets Lake Formation permissions apply to chosen databases, tables and principals while existing IAM-based access continues for everyone else. The guide describes it as a way to onboard one data lake use case at a time.

A cautious sequence:

1. Appoint a data lake administrator and inventory who reads what today.
2. Classify data: decide the tags and which columns are sensitive.
3. Choose one database with sensitive data. Register its location in hybrid access mode.
4. Grant Lake Formation permissions to one group of users, and confirm their queries in Athena behave as intended.
5. Remove those users' direct S3 access to that location.
6. Extend table by table. Move pipelines' roles last, after confirming the grants they need.

What it is not

• Not a replacement for IAM. It augments IAM. Users still need IAM permission to call Athena or Glue.
• Not enforcement for tools that read S3 directly. A script with S3 credentials is outside its control, which is why direct access must be withdrawn.
• Not a data catalogue or a quality tool. It governs access. Describing data and testing it are separate jobs; data governance basics covers the wider picture.

It also supports sharing catalog resources with other AWS accounts, which is how multi-account data platforms expose one team's tables to another without copying.

For the lake that these permissions sit on, see building a data lake on AWS with S3 and Glue; for the engine most users will query it with, Amazon Athena explained; and for the catalog itself, the Glue Data Catalog and crawlers.

Data lakes on AWS and data governance are covered in the Data Engineering course.

Sources

• What is AWS Lake Formation? (AWS Lake Formation Developer Guide) — AWS Documentation
• AWS Lake Formation: How it works (AWS Lake Formation Developer Guide) — AWS Documentation
• Data filtering and cell-level security in Lake Formation (AWS Lake Formation Developer Guide) — AWS Documentation
• Lake Formation tag-based access control (AWS Lake Formation Developer Guide) — AWS Documentation
• Changing the default settings for your data lake (AWS Lake Formation Developer Guide) — AWS Documentation
• Granting table permissions using the named resource method (AWS Lake Formation Developer Guide) — AWS Documentation