Generator Expressions

Updated

September 7, 2026

Generator Expressions

After reading this chapter, you will construct lazy, memory-efficient data pipelines using generator expressions, measure and contrast their heap allocation footprint against eager collections, stream data directly into built-in aggregators like sum(), max(), and any(), and avoid generator exhaustion and late-binding pitfalls.

Mental model

A list comprehension is eager: it computes every element immediately and allocates memory for the entire collection on the heap. A generator expression is lazy: it produces values one at a time, on demand, via Python’s iterator protocol.

Eager Evaluation (List Comprehension: [...]):
  [x * 2 for x in range(10_000_000)]
  │
  ▼
  [ 0, 2, 4, 6, 8, ... ]  ──▶ Allocates ~80 Megabytes of RAM immediately!
                              High allocation latency; blocks until full.

Lazy Evaluation (Generator Expression: (...)):
  (x * 2 for x in range(10_000_000))
  │
  ▼
  ┌───────────────────────────────┐
  │ Generator State Machine (~200B)│
  │ Current: x=0, next: x=1       │
  └───────────────┬───────────────┘
                  │ next()
                  ▼
              Yields: 0 ──▶ Yields: 2 ──▶ Yields: 4 (O(1) memory footprint!)

Generator expressions also chain together into pipelines where values are pulled through on demand like items on an assembly line:

Source Data ──▶ ( parse ) ──▶ ( filter ) ──▶ ( transform ) ──▶ Consumer (sum/any/for)
                  ▲              ▲                 ▲
                  └──────────────┴─────────────────┴── Only one item processed at a time!

Minimal example

Save as gen_expr_overview.py:

# gen_expr_overview.py
import sys

def main() -> None:
    count = 1_000_000

    # 1. Eager List Comprehension: Allocates full list in memory
    eager_list = [n * 2 for n in range(count)]
    list_mem_bytes = sys.getsizeof(eager_list)

    # 2. Lazy Generator Expression: Allocates lightweight iterator object
    lazy_gen = (n * 2 for n in range(count))
    gen_mem_bytes = sys.getsizeof(lazy_gen)

    print(f"Items processed:         {count:,}")
    print(f"Eager list memory:       {list_mem_bytes:,} bytes (~{list_mem_bytes / (1024 * 1024):.2f} MB)")
    print(f"Lazy generator memory:   {gen_mem_bytes:,} bytes")
    print(f"Memory ratio:            {list_mem_bytes // gen_mem_bytes:,}x less memory!")

    # 3. Consuming on demand with next()
    print(f"\nFirst three values pulled on demand:")
    print(f"  Item 1: {next(lazy_gen)}")
    print(f"  Item 2: {next(lazy_gen)}")
    print(f"  Item 3: {next(lazy_gen)}")

if __name__ == "__main__":
    main()

Run via uv run python gen_expr_overview.py:

Items processed:         1,000,000
Eager list memory:       8,448,728 bytes (~8.06 MB)
Lazy generator memory:   208 bytes
Memory ratio:            40,618x less memory!

First three values pulled on demand:
  Item 1: 0
  Item 2: 2
  Item 3: 4

Worked examples

Case 1: Streaming Gigabyte Log Parsing with Constant O(1) Memory

In production environments, log files can span tens of gigabytes. Reading an entire log file into a list causes MemoryError and triggers Linux Out-Of-Memory (OOM) kills. Chaining generator expressions processes files of arbitrary size with constant memory:

# streaming_log_parser.py
from collections.abc import Iterator

def generate_mock_log_stream() -> Iterator[str]:
    """Simulate reading lines from a high-throughput server log."""
    logs = [
        '192.168.1.10 - - [2026-09-07] "GET /api/v1/health" 200 45',
        '192.168.1.11 - - [2026-09-07] "POST /api/v1/login" 401 120',
        '# SRE maintenance comment - ignore',
        '10.0.0.50 - - [2026-09-07] "GET /api/v1/orders" 200 890',
        '',
        '172.16.0.4 - - [2026-09-07] "GET /api/v1/checkout" 500 240',
        '192.168.1.12 - - [2026-09-07] "GET /api/v1/status" 200 62',
    ]
    yield from logs

def main() -> None:
    raw_lines = generate_mock_log_stream()

    # Stage 1: Strip whitespace and drop comments/blank lines
    clean_lines = (line.strip() for line in raw_lines if line.strip() and not line.startswith("#"))

    # Stage 2: Tokenize and extract (status_code, response_bytes)
    def parse_entry(line: str) -> tuple[int, int]:
        parts = line.split()
        return int(parts[-2]), int(parts[-1])

    records = (parse_entry(line) for line in clean_lines)

    # Stage 3: Filter for successful requests (HTTP 200)
    success_bytes = (resp_bytes for status, resp_bytes in records if status == 200)

    # Stage 4: Consume directly into built-in aggregator sum()
    total_payload_bytes = sum(success_bytes)
    print(f"Total payload transferred for HTTP 200 responses: {total_payload_bytes} bytes")

if __name__ == "__main__":
    main()

Run:

uv run python streaming_log_parser.py

Output:

Total payload transferred for HTTP 200 responses: 997 bytes

Why this matters: Notice that sum(resp_bytes for status, resp_bytes in ...) does not require double parentheses ((...)). When a generator expression is the sole argument to a function call, outer parentheses are optional.

Case 2: Short-Circuiting Aggregations with `any()` and `all()`

Built-in boolean predicates any() and all() short-circuit: they terminate evaluation immediately upon encountering the first definitive boolean value (True for any, False for all). Using a generator expression prevents running expensive operations on remaining elements:

# short_circuit_eval.py
def check_node_health(node_id: str) -> bool:
    """Simulate an expensive remote node probe."""
    print(f"  [Probe] Checking node: {node_id}...")
    # Simulate node-03 being unhealthy
    return node_id != "node-03"

def main() -> None:
    cluster_nodes = [f"node-{i:02d}" for i in range(1, 10)]

    print("--- Testing with Generator Expression (Lazy, Short-Circuits) ---")
    # Generator expression: evaluation stops immediately at node-03!
    all_healthy_gen = all(check_node_health(node) for node in cluster_nodes)
    print(f"Cluster all healthy result: {all_healthy_gen}")

    print("\n--- Contrast: List Comprehension (Eager, Evaluates All 9 Nodes!) ---")
    # List comprehension: MUST evaluate all 9 nodes before calling all()
    probes_run = 0
    def logged_probe(node: str) -> bool:
        nonlocal probes_run
        probes_run += 1
        return node != "node-03"

    all_healthy_list = all([logged_probe(node) for node in cluster_nodes])
    print(f"List comprehension ran {probes_run} probes even though node-03 failed!")

if __name__ == "__main__":
    main()

Run:

uv run python short_circuit_eval.py

Output:

--- Testing with Generator Expression (Lazy, Short-Circuits) ---
  [Probe] Checking node: node-01...
  [Probe] Checking node: node-02...
  [Probe] Checking node: node-03...
Cluster all healthy result: False

--- Contrast: List Comprehension (Eager, Evaluates All 9 Nodes!) ---
List comprehension ran 9 probes even though node-03 failed!

Case 3: Composing Multi-Stage Data Pipelines

Generator expressions can be chained linearly to construct readable Extract-Transform-Load (ETL) pipelines without third-party frameworks:

# pipeline_composition.py
def main() -> None:
    raw_telemetry = [
        {"host": "srv-1", "cpu": 45.2, "mem_mb": 4096},
        {"host": "srv-2", "cpu": 92.1, "mem_mb": 16384},
        {"host": "srv-3", "cpu": 78.4, "mem_mb": 8192},
        {"host": "srv-4", "cpu": 95.8, "mem_mb": 32768},
        {"host": "srv-5", "cpu": 12.0, "mem_mb": 2048},
    ]

    # Stage 1: Filter hosts breaching CPU threshold (> 75%)
    overloaded = (rec for rec in raw_telemetry if rec["cpu"] > 75.0)

    # Stage 2: Convert memory to gigabytes
    with_gb = (
        {"host": rec["host"], "cpu": rec["cpu"], "mem_gb": rec["mem_mb"] / 1024}
        for rec in overloaded
    )

    # Stage 3: Format alert payload strings
    alerts = (
        f"[ALERT] {item['host']}: CPU={item['cpu']:.1f}% Mem={item['mem_gb']:.1f}GB"
        for item in with_gb
    )

    # Terminal Consumer: Iterate and dispatch
    print("Dispatched Infrastructure Alerts:")
    for alert in alerts:
        print(f"  {alert}")

if __name__ == "__main__":
    main()

Run:

uv run python pipeline_composition.py

Output:

Dispatched Infrastructure Alerts:
  [ALERT] srv-2: CPU=92.1% Mem=16.0GB
  [ALERT] srv-3: CPU=78.4% Mem=8.0GB
  [ALERT] srv-4: CPU=95.8% Mem=32.0GB

Pitfalls

Pitfall 1: Generator Exhaustion (The “Read-Once” Trap)

Unlike lists or tuples, generators are stateful, single-pass iterators. Once consumed to completion, they are exhausted and produce no further items:

# THE TRAP: Re-iterating an exhausted generator
data = (x * 10 for x in range(1, 5))

total = sum(data)      # Consumes all items: 10 + 20 + 30 + 40 = 100
maximum = max(data)    # BUG: ValueError: max() arg is an empty sequence!

# THE FIX: Materialize as list if multiple passes are strictly required,
# or recreate the generator expression.
numbers = [x * 10 for x in range(1, 5)]  # If dataset fits in memory
total = sum(numbers)
maximum = max(numbers)
print(f"Total: {total}, Max: {maximum}")

Pitfall 2: Late-Binding Closures in Loop Expressions

When a generator expression references a variable from an outer loop, the variable is looked up when the generator is evaluated, NOT when it was defined:

# THE TRAP: Late binding
multipliers = []
for factor in [2, 3, 5]:
    # The generator captures the variable name `factor`, not its current value!
    multipliers.append((x * factor for x in range(3)))

# When consumed, `factor` has its final loop value (5) for ALL generators!
for gen in multipliers:
    print(list(gen))
# Prints:
# [0, 5, 10]
# [0, 5, 10]
# [0, 5, 10]

# THE FIX: Bind the current value eagerly using a function scope or default parameter
def make_multiplier_gen(f):
    return (x * f for x in range(3))

multipliers = [make_multiplier_gen(factor) for factor in [2, 3, 5]]
for gen in multipliers:
    print(list(gen))
# Prints correctly:
# [0, 2, 4]
# [0, 3, 6]
# [0, 5, 10]

Pitfall 3: Indexing and Length Queries (`gen[0]` or `len(gen)`)

Because generator expressions compute elements on the fly, they do not know their future length and cannot jump to arbitrary indices:

gen = (x ** 2 for x in range(10))

# TypeError: 'generator' object is not subscriptable
# first = gen[0]

# TypeError: object of type 'generator' has no len()
# count = len(gen)

# THE FIX:
# Use next() for the first item:
first = next(gen)

# Use itertools.islice for slice windows:
import itertools
next_three = list(itertools.islice(gen, 3))
print(f"First: {first}, Next three: {next_three}")

Exercises

Given a large range range(1, 10_000_001), use a generator expression with sum() to calculate the sum of all odd integers. Verify how quickly it executes without consuming significant memory.
Given a list of server hostnames, write an all() expression using a generator to verify that every hostname starts with "node-" and ends with ".internal". Test that it terminates early when the first hostname fails.
Write a generator pipeline that reads a stream of numbers, squares each number, filters out numbers not divisible by 3, and formats the output into strings "Num: <val>".
Demonstrate generator exhaustion: create a generator expression, verify list(gen) yields 5 elements, and verify a second list(gen) call returns [].
Write a memory benchmark script comparing sys.getsizeof for a list comprehension vs a generator expression for 5 million elements.

Generator Expressions

Mental model

Minimal example

Worked examples

Case 1: Streaming Gigabyte Log Parsing with Constant O(1) Memory

Case 2: Short-Circuiting Aggregations with any() and all()

Case 3: Composing Multi-Stage Data Pipelines

Pitfalls

Pitfall 1: Generator Exhaustion (The “Read-Once” Trap)

Pitfall 2: Late-Binding Closures in Loop Expressions

Pitfall 3: Indexing and Length Queries (gen[0] or len(gen))

Exercises

Further reading

Case 2: Short-Circuiting Aggregations with `any()` and `all()`

Pitfall 3: Indexing and Length Queries (`gen[0]` or `len(gen)`)