Python Generators and Iterators Made Simple

Every for loop in Python runs on the same small protocol, and once you see it, generators stop looking like magic. They become the simplest way to write code that produces values one at a time, which is how you process a 10 GB log file with a few kilobytes of memory. This guide builds up from the protocol to generator pipelines and the standard library tools around them.

What a for loop actually does

A for loop asks its object for an iterator with iter(), then calls next() on it until it raises StopIteration (Python tutorial: iterators):

cities = ["Hyderabad", "Pune"]
it = iter(cities)
print(next(it), next(it))
try:
    next(it)
except StopIteration:
    print("StopIteration: the loop ends here")

Hyderabad Pune
StopIteration: the loop ends here

The list is an iterable: something you can get an iterator from. The iterator is the object that remembers the position. Files, dictionaries, ranges and strings are all iterables, and they all support for through this same pair of calls.

Writing an iterator the long way

To make your own object iterable, give it __iter__, returning the iterator, and __next__, returning the next value or raising StopIteration:

class Countdown:
    def __init__(self, start: int):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self) -> int:
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

print(list(Countdown(3)))

[3, 2, 1]

It works, but it is a lot of ceremony: state stored on self, and the stopping condition written as an exception.

The same thing with a generator

A generator function contains yield. Calling it does not run the body; it returns a generator, an iterator whose __next__ runs the function until the next yield, hands out that value, and pauses with all its local variables intact (Python glossary: generator):

def countdown(start: int):
    while start > 0:
        yield start
        start -= 1

gen = countdown(3)
print(gen)
print(next(gen), list(gen))

<generator object countdown at 0x...>
3 [2, 1]

(The address in the first line varies.) Five lines replace the class, and the logic reads top to bottom. When the function returns, the generator raises StopIteration for you. Notice also that next(gen) took the 3, so list(gen) got only what was left: a generator keeps its position.

Why it matters: lazy pipelines

A generator produces each value only when asked. Chain several and each item flows through the whole pipeline before the next one is read, so memory use stays constant however large the input. Here is a log analysis written as four small stages:

import tempfile
from collections import Counter
from pathlib import Path

log = Path(tempfile.mkdtemp()) / "app.log"
log.write_text("\n".join([
    "2026-09-25 10:00:01 INFO user login id=17",
    "2026-09-25 10:00:03 ERROR payment timeout id=42",
    "2026-09-25 10:00:04 INFO page view id=17",
    "2026-09-25 10:00:09 ERROR payment timeout id=43",
    "2026-09-25 10:00:12 ERROR disk full",
]), encoding="utf-8")

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

def parse(lines):
    for line in lines:
        date, time, level, message = line.split(" ", 3)
        yield {"time": time, "level": level, "message": message}

def only(level, records):
    return (r for r in records if r["level"] == level)

errors = only("ERROR", parse(read_lines(log)))
print(Counter(r["message"].split(" id=")[0] for r in errors))

Counter({'payment timeout': 2, 'disk full': 1})

Nothing happens when errors is created; the pipeline is just wired up. Counter pulls records through one at a time, so the file is read line by line and no stage ever holds the whole file. Each stage is also a small, testable function: parse can be tested with a list of strings, no file needed. The only stage uses a generator expression, the parenthesised cousin of a list comprehension covered in the list comprehensions guide.

yield from: delegating to another iterable

yield from hands out every value of another iterable, which makes recursive generators short (PEP 380):

def flatten(items):
    for item in items:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

print(list(flatten([1, [2, [3, 4]], 5])))

[1, 2, 3, 4, 5]

itertools: generators you do not have to write

The standard library's itertools module is a toolbox of fast, lazy building blocks (Python docs: itertools):

from itertools import accumulate, chain, count, islice, pairwise

print(list(islice(count(start=100, step=5), 4)))          # first 4 of an infinite sequence
print(list(chain(["a", "b"], ("c",), "de")))                # several iterables as one
print(list(pairwise([10, 12, 9, 15])))                       # consecutive pairs (Python 3.10+)
print(list(accumulate([120, 80, 45])))                       # running totals

[100, 105, 110, 115]
['a', 'b', 'c', 'd', 'e']
[(10, 12), (12, 9), (9, 15)]
[120, 200, 245]

count never ends, which is fine because islice asks for only four values; infinite generators are safe as long as something limits how many you take. pairwise makes day-over-day changes a one-liner: [b - a for a, b in pairwise(values)]. groupby groups consecutive items and needs its input sorted by the same key first.

The one-shot rule

An iterator is used up once it has been consumed, and it stays empty after that (Python glossary: iterator):

squares = (n * n for n in range(4))
print(sum(squares), sum(squares))

14 0

The second sum sees an exhausted generator and returns 0, with no error. This is the most common generator bug: passing a generator to two consumers, or looping over it twice. If you need the values more than once, store them in a list, or make a function that returns a fresh generator each time it is called.

Debugging and testing generators

Because a generator does nothing until it is consumed, its bugs appear later than you expect: an exception inside parse surfaces only when Counter pulls the faulty line through, and the traceback points into the generator, not at the line that created it. Three habits make generators easy to work with:

• Peek without consuming everything. list(islice(gen, 5)) shows the first five items of any pipeline stage while you develop.
• Test stages with plain lists. Each stage takes an iterable, so a test can pass a short list of strings and compare list(stage(items)) with the expected output.
• Keep side effects out of the middle. Stages that only transform values can be recombined freely; put writing to files or databases in the final consumer, where it runs exactly once per item.

When to use which

• A list when you need the values more than once, random access, len(), or the data is small.
• A generator when data is large or unbounded, when you want to stop early, or when you are building a pipeline of processing stages.
• An iterator class when the object has other behaviour besides being iterated, such as methods or configuration, and even then its __iter__ can simply be a generator method.

Generators also power context managers written with @contextmanager, as shown in decorators vs context managers. For a side-by-side comparison with comprehensions, see iterators vs generators vs comprehensions; for the memory side of the story, Python memory management. Iterators and generators, together with decorators, closures and context managers, form the advanced concepts module of the Python Programming course.

Sources

• Iterators and Generators (tutorial) — Python documentation
• Glossary: generator and iterator — Python documentation
• itertools — Functions creating iterators for efficient looping — Python documentation
• PEP 380 – Syntax for Delegating to a Subgenerator — Python Enhancement Proposals