Python Iterators vs Generators vs Comprehensions

Python gives you four ways to produce a sequence of values: a class implementing the iterator protocol, a generator function with yield, a generator expression in round brackets, and a list comprehension in square brackets. They overlap enough that choosing can feel like taste. It is not: they differ in memory, reusability, flexibility and readability, and those differences are easy to measure. Here is one task written all four ways, followed by the numbers and a rule for choosing.

The task, four ways

Sum the squares of the first million integers. Each version below produces the same values; only the mechanism differs.

import time
import tracemalloc

N = 1_000_000

class Squares:                                  # 1. iterator class
    def __init__(self, n):
        self.n, self.i = n, 0
    def __iter__(self):
        return self
    def __next__(self):
        if self.i >= self.n:
            raise StopIteration
        value = self.i * self.i
        self.i += 1
        return value

def squares(n):                                 # 2. generator function
    for i in range(n):
        yield i * i

versions = {
    "iterator class": lambda: sum(Squares(N)),
    "generator function": lambda: sum(squares(N)),
    "generator expression": lambda: sum(n * n for n in range(N)),   # 3.
    "list comprehension": lambda: sum([n * n for n in range(N)]),   # 4.
}

def best_time(fn, repeat=3):
    times = []
    for _ in range(repeat):
        start = time.perf_counter()
        fn()
        times.append(time.perf_counter() - start)
    return min(times)

def peak_mb(fn):
    tracemalloc.start()
    fn()
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return peak / 1e6

assert len({fn() for fn in versions.values()}) == 1      # all four agree
for name, fn in versions.items():
    print(f"{name:<22}{best_time(fn):6.3f}s   peak {peak_mb(fn):7.3f} MB")

iterator class         0.096s   peak   0.001 MB
generator function     0.050s   peak   0.000 MB
generator expression   0.050s   peak   0.001 MB
list comprehension     0.064s   peak  40.448 MB

tracemalloc records memory allocated by Python while each version runs and reports the peak (Python docs: tracemalloc). Timings vary between machines and runs; the pattern does not.

What the numbers say

Memory is the real divide. The list comprehension builds a list of a million integers before sum sees any of them, so its peak is tens of megabytes. The other three produce one value at a time, and their peak memory is effectively zero, whatever N is. Scale the task to a billion values and the list version fails for lack of memory, while the others simply take longer.

Speed is close, with one exception. The two generator forms are the fastest here. The list comprehension is not far behind: building a list has a cost, but it runs in tight interpreter code. The iterator class is clearly slowest, because every value goes through a Python-level __next__ call with attribute lookups on self. The Python tutorial makes the design point directly: generators create __iter__ and __next__ automatically and keep local state between calls, which is what makes them more compact than a class (Python tutorial: generators).

Beyond the benchmark

Two rows decide most real choices. Reuse: a generator expression is spent after one pass, so if the values are needed twice, as in computing a mean and then a standard deviation, a list is simpler. Logic: comprehensions and generator expressions hold a single expression; the moment you need a try, several steps, or state carried from one item to the next, write a generator function.

When each one is the right choice

• List comprehension: the result is small or modest, and you need to use it more than once, index into it, sort it, or know its length.
• Generator expression: you are feeding a single consumer, especially sum, max, any, all, "".join or a for loop, and the data may be large. It is the default for "transform and aggregate" (Python reference: generator expressions).
• Generator function: producing each value takes more than one expression, the sequence is unbounded, or you are building a pipeline of stages that read, parse, filter and aggregate.
• Iterator class: the object is more than a sequence. It has configuration, methods or state that callers use directly, such as a paginated API client with a reset() method or statistics about what it has fetched. Even then, its __iter__ can simply be a generator method that uses yield.

A realistic case: a large CSV export

Suppose you must total the refunds in a 3 GB export of orders. Reading every row into a list and then filtering needs memory for the whole file. A generator function that opens the file and yields one parsed row at a time, followed by a generator expression that keeps refunds and a single sum, reads the file once in constant memory. The code is barely longer than the list version, and it keeps working when next year's export is twice the size. If the same rows were also needed for a second report, the better design is still one pass: compute both totals in the same loop, rather than materialising the rows to iterate over them twice.

A rule of thumb

Start with the least machinery that works: a list comprehension for small, reused results; a generator expression for one-pass aggregation; a generator function when the logic outgrows one expression; a class only when the iterator needs an identity of its own. Switching from a list comprehension to a generator expression is a matter of changing two brackets, so there is no need to agonise at the start. Measure when data grows.

How the protocol and yield work underneath is explained in Python generators and iterators made simple; the full comprehension syntax, including dict and set comprehensions and the walrus operator, is in the list comprehensions guide. Where memory actually goes in a Python program is covered in Python memory management. Comprehensions, iterators and generators are taught in the advanced concepts module of the Python Programming course.

Sources

• tracemalloc — Trace memory allocations — Python documentation
• Generator expressions — Python documentation
• Generators (tutorial) — Python documentation