Parquet vs ORC vs Avro: File Formats Compared

Choosing a file format sounds like a detail, but it decides how much storage a data lake uses, how fast queries run and how painful schema changes are. Three open formats dominate: Parquet and ORC, which store data by column, and Avro, which stores it by row. This guide explains the difference, measures it on real data, and gives a simple rule for choosing.

Rows versus columns

A row-based format writes one complete record after another. Writing a new record is cheap, and reading whole records is natural, which suits streaming events and message queues. Avro is row-based: its container files hold the writer's schema in the header, then blocks of serialized records (Avro specification).

A columnar format groups values by column. Parquet is described by its project as a column-oriented format designed for efficient storage and retrieval (Parquet docs: overview): a file holds row groups, each row group holds one contiguous column chunk per column, and chunks are divided into pages, the unit of compression and encoding (Parquet docs: concepts). ORC is similar, organising data into stripes of roughly 64 MB with indexes that let readers skip stripes and 10,000-row groups that cannot match a filter (ORC docs).

Columnar storage wins for analytics for two reasons. A query that needs three columns out of twenty reads only those three. And values of one column are alike, dates next to dates and small integers next to small integers, so they compress far better than mixed records.

Measured on real data

To compare fairly, the same 500,000 New York yellow taxi trips (NYC TLC: trip record data), nine columns, are written in each format:

import os
import urllib.request
from pathlib import Path
import pyarrow as pa
import pyarrow.csv as pcsv
import pyarrow.orc as orc
import pyarrow.parquet as pq
from fastavro import parse_schema, writer

source = "yellow_tripdata_2024-01.parquet"
if not Path(source).exists():
    urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{source}", source)

cols = ["tpep_pickup_datetime", "PULocationID", "DOLocationID", "passenger_count", "trip_distance",
        "payment_type", "fare_amount", "tip_amount", "total_amount"]
table = pq.read_table(source, columns=cols).slice(0, 500_000)
print(f"{table.num_rows} rows, {table.num_columns} columns")

pcsv.write_csv(table, "trips.csv")
pq.write_table(table, "trips_snappy.parquet", compression="snappy")
pq.write_table(table, "trips_zstd.parquet", compression="zstd")
orc.write_table(table, "trips_zstd.orc", compression="zstd")

avro_type = {pa.int32(): "int", pa.int64(): "long", pa.float64(): "double"}
schema = parse_schema({"type": "record", "name": "Trip", "fields": [
    {"name": f.name, "type": ["null", {"type": "long", "logicalType": "timestamp-micros"} if pa.types.is_timestamp(f.type) else avro_type[f.type]]}
    for f in table.schema]})
records = table.to_pylist()
for codec in ("null", "deflate"):
    with open(f"trips_{codec}.avro", "wb") as out:
        writer(out, schema, records, codec=codec)

for name in ["trips.csv", "trips_null.avro", "trips_deflate.avro", "trips_snappy.parquet", "trips_zstd.parquet", "trips_zstd.orc"]:
    print(f"{name:<22} {os.path.getsize(name) / 1e6:6.2f} MB")

meta = pq.ParquetFile("trips_zstd.parquet").metadata
sizes = {meta.row_group(0).column(i).path_in_schema: sum(meta.row_group(g).column(i).total_compressed_size for g in range(meta.num_row_groups)) for i in range(meta.num_columns)}
total = sum(sizes.values())
print("row groups:", meta.num_row_groups)
for c in ["fare_amount", "payment_type"]:
    print(f"{c}: {sizes[c] / 1e6:.2f} MB, {sizes[c] / total:.1%} of column data")

500000 rows, 9 columns
trips.csv               28.36 MB
trips_null.avro         27.45 MB
trips_deflate.avro       9.45 MB
trips_snappy.parquet     7.04 MB
trips_zstd.parquet       5.82 MB
trips_zstd.orc           7.13 MB
row groups: 1
fare_amount: 0.53 MB, 9.1% of column data
payment_type: 0.07 MB, 1.3% of column data

What the numbers show:

• CSV and uncompressed Avro are about the same size, roughly 28 MB. Avro's advantage over CSV is not size but types and an embedded schema.
• Compression matters more than format for row files: deflate cuts Avro to 9.45 MB.
• Columnar formats are smallest, Parquet with zstd at 5.82 MB, about a fifth of the CSV. Parquet with snappy is larger but faster to decompress, which is why it is a common default.
• ORC and Parquet are close. On this data Parquet with zstd was smaller than ORC with zstd; on other data the order can flip, so measure on your own.
• A single column is a small slice. The fare column is 9.1% of the Parquet file's column data and payment type 1.3%, so a query on those columns reads a small fraction of the file. An Avro reader must decode every record to get the same values.

Schema evolution

Data changes shape: columns are added, types widen. Avro was designed for this. A reader supplies its own schema, and fields are matched to the writer's schema by name; fields missing from the data take the reader's default, and promotions such as int to long are allowed. That is why Avro, usually with a schema registry, is the common choice for Kafka messages; Apache Kafka explained covers that setting.

Parquet and ORC files also carry their schemas, and engines can merge compatible schemas across files, but evolution is usually managed a level up, by table formats such as Delta Lake and Apache Iceberg that record schema changes in their metadata. Schema evolution goes deeper, and Delta Lake explained shows a table format built on Parquet.

Splitting and parallel reads

Big-data engines split large files so several tasks can read one file in parallel. All three formats support it: Avro through sync markers between blocks, Parquet through row groups, ORC through stripes. Compressed CSV, such as a gzip file, cannot be split, so one task must read it end to end, a quiet cause of slow jobs.

Getting the most from columnar files

The format alone does not guarantee fast reads; how files are written matters as much:

• Avoid many small files. Thousands of tiny Parquet files, common from streaming jobs, cost more in opening and metadata than in reading data. Compact them into files of tens to hundreds of megabytes.
• Sort or cluster before writing. Parquet and ORC store minimum and maximum values per row group or stripe, so data sorted by a commonly filtered column, such as date, lets readers skip whole blocks.
• Partition folders by a low-cardinality column, such as month, so queries on one month never open the others.
• Keep types precise: dates as dates and numbers as numbers, not text, so encodings and statistics work.

Which to choose

For most new analytical work the answer is Parquet, usually managed by a table format; Iceberg vs Delta Lake vs Hudi compares those. Keep Avro for the streaming edge and convert to Parquet as data lands in the lake.

Big data fundamentals, Spark and a data lake implementation project are part of the Data Engineering course.

Sources

• Apache Parquet overview (Apache Parquet documentation) — Apache Parquet
• Parquet concepts (Apache Parquet documentation) — Apache Parquet
• ORC documentation (Apache ORC) — Apache ORC
• Apache Avro 1.12.0 specification (Apache Avro) — Apache Avro
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission