AWS Glue Data Catalog and Crawlers Explained

Files in S3 have no names for their columns that a query engine can rely on, no declared types, and no list of which folders belong together. The AWS Glue Data Catalog supplies all three. AWS describes it as a centralised metadata repository that acts as an index to the location, schema and properties of your data (Glue docs: Data Catalog). Athena, Redshift external tables, EMR and Glue jobs all read table definitions from it.

Crawlers are the automatic way to fill it. They are convenient, and they make guesses. This article explains what the catalog holds, how crawlers decide what they decide, and how to keep the result under control.

What the catalog stores

The catalog is organised like a relational catalog:

• Database: a logical group of table definitions.
• Table: the schema, the S3 location, the format and its properties.
• Partition: a subfolder of the table's location, with the values of the partition keys.

One point the concepts page stresses: tables and databases in the catalog contain metadata; they do not contain data (Glue docs: concepts). Deleting a catalog table does not delete a file, and adding a file does not change the catalog. The two are kept in step only by whatever process you choose. Each account has one Data Catalog per Region.

Names are lowercased. Athena's naming guidance says names in Glue should be lower case and that Athena lowers upper-case names in DDL, so do not rely on case to tell columns apart (Athena docs: naming).

How a crawler works

A crawler does three things when it runs (Glue docs: crawlers):

1. Classifies the data to determine format, schema and properties.
2. Groups the data into tables or partitions.
3. Writes table and partition definitions to the catalog.

Classification is done by classifiers. Each reads the data and, if it recognises the format, generates a schema and a certainty number. Custom classifiers you supply run first, in order; then the built-in ones for formats such as CSV, JSON, Avro and XML. If nothing recognises the data, the classification is UNKNOWN (Glue docs: classifiers).

Grouping is where surprises come from. Given a folder tree, the crawler must decide which level is a table and which levels below it are partitions. The rule in the guide: when the majority of schemas at a folder level are similar, the crawler creates partitions of one table instead of separate tables. Two consequences follow:

• Point a crawler at s3://bucket/folder1/ containing table1/ and table2/ with similar schemas, and you get one table with table1 and table2 as partition values, not two tables.
• Put files with dissimilar schemas under one prefix, and you can get several tables from a single folder, which the guide warns can make Athena queries return zero results.

The fix in both cases is the one the guide gives: define a separate include path for each table.

What inference has to cope with

To see why schema inference is not trivial, compare three files that are supposedly the same dataset: New York yellow taxi trips for January 2022, 2024 and 2025. The script reads each file's own Parquet schema and lists only the columns that disagree. It is a local comparison, not a crawler run.

import pyarrow.parquet as pq

FILES = {"2022-01": "yellow_tripdata_2022-01.parquet",
         "2024-01": "yellow_tripdata_2024-01.parquet",
         "2025-01": "yellow_tripdata_2025-01.parquet"}

schemas = {}
for month, path in FILES.items():
    schema = pq.read_schema(path)
    schemas[month] = {field.name: str(field.type) for field in schema}
    print(f"{month}: {len(schema)} columns, {pq.read_metadata(path).num_rows:,} rows")

names = []
for columns in schemas.values():
    names += [n for n in columns if n not in names]

print()
print(f"{'column':24}" + "".join(f"{m:16}" for m in schemas))
for name in names:
    types = [schemas[m].get(name, "-") for m in schemas]
    if len(set(types)) > 1:                       # show only the columns that disagree
        print(f"{name:24}" + "".join(f"{t:16}" for t in types))

2022-01: 19 columns, 2,463,931 rows
2024-01: 19 columns, 2,964,624 rows
2025-01: 20 columns, 3,475,226 rows

column                  2022-01         2024-01         2025-01         
VendorID                int64           int32           int32           
passenger_count         double          int64           int64           
RatecodeID              double          int64           int64           
store_and_fwd_flag      string          large_string    large_string    
PULocationID            int64           int32           int32           
DOLocationID            int64           int32           int32           
airport_fee             double          -               -               
Airport_fee             -               double          double          
cbd_congestion_fee      -               -               double          

Nine columns differ across three years of one published dataset. Integer widths changed. passenger_count and RatecodeID moved from double to integer. A fee column was renamed by a change of case, airport_fee to Airport_fee, which in a catalog that lowercases names is the same name with a different history. A new column appeared in 2025.

Put those three files under one table's prefix and something has to reconcile them. A crawler will produce a schema; whether it is the schema you want, and whether every engine can then read every file with it, is something to verify, not assume. Schema evolution covers handling such changes deliberately.

Keeping a crawler predictable

By default a crawler updates the catalog to match whatever it finds. That default is risky for tables other people depend on. The controls (Glue docs: schema change settings):

The documented CLI form for "add new columns only", not run here:

aws glue update-crawler \
  --name myCrawler \
  --configuration '{ "Version": 1.0, "CrawlerOutput": { "Tables": { "AddOrUpdateBehavior": "MergeNewColumns" } } }'

Other settings worth knowing:

• Incremental crawls. After a first full crawl, the crawler adds only new partitions, which is faster. The guide notes they do not detect modified or deleted partitions and suit stable schemas (Glue docs: incremental crawls).
• Schedules. Crawlers run on demand or on a cron schedule.
• Table threshold. A maximum number of tables the crawler may create, a useful guard against a mis-aimed crawl creating hundreds.

Crawler or explicit definition?

A crawler is not the only way in. Tables can be created through the console, the CreateTable API, CloudFormation, or DDL in Athena.

A common arrangement is a crawler for discovery in the raw zone and explicit, version-controlled definitions for curated tables. The pipeline that writes a curated partition also registers it, so data and catalog cannot drift apart. Building a data lake on AWS shows that registration step.

Mistakes that cause trouble

• One crawler aimed at the bucket root. Table boundaries are then left to heuristics.
• Mixed schemas under one prefix.
• Trusting inferred types for CSV, where a column of codes may be read as numbers.
• Never reviewing what was created. Check the table list and column types after the first crawl.
• Crawling on a schedule when nothing changes, which costs crawler time for no benefit.
• Forgetting permissions. The catalog says what exists; who may read it is a separate matter, handled with IAM and Lake Formation.

Once tables are in the catalog, they are immediately usable from Amazon Athena and as sources and targets for Glue jobs; the AWS Glue tutorial continues from there.

AWS Glue and data lake cataloguing on AWS are covered in the Data Engineering course.

Sources

• Data discovery and cataloging in AWS Glue (AWS Glue User Guide) — AWS Documentation
• AWS Glue concepts (AWS Glue User Guide) — AWS Documentation
• Using crawlers to populate the Data Catalog (AWS Glue User Guide) — AWS Documentation
• Defining and managing classifiers (AWS Glue User Guide) — AWS Documentation
• Preventing a crawler from changing an existing schema (AWS Glue User Guide) — AWS Documentation
• Name databases, tables, and columns (Amazon Athena User Guide) — AWS Documentation
• Scheduling incremental crawls for adding new partitions (AWS Glue User Guide) — AWS Documentation