Generator Functions and Subgenerator Delegation

Updated

September 7, 2026

Generator Functions and Subgenerator Delegation

After reading this chapter, you will master the mechanics of generator functions (yield), establish bi-directional communication pipelines using .send(), inject exceptions and trigger cleanup via .throw() and .close(), and delegate execution to nested subgenerators using yield from (PEP 380).

Mental model

An ordinary Python function begins execution, computes a result, and exits, destroying its stack frame. A generator function (any function containing the yield keyword) returns a generator object whose execution state is preserved on the heap across calls:

Standard Function:
  call() ──▶ [ Execute Frame ] ──▶ return ──▶ Frame destroyed, state lost.

Generator Function (Suspension & Resumption):
  gen = my_gen() ──▶ Allocates generator frame on heap (GEN_CREATED)
                          │
  next(gen)      ──▶ Runs to 'yield X' ──▶ Suspends (GEN_SUSPENDED), yields X
                          │
  gen.send(Y)    ──▶ Resumes at 'yield', receives Y, runs to next yield...
                          │
  return         ──▶ Raises StopIteration (GEN_CLOSED), frame cleaned up.

The `yield from` Bi-Directional Pipe (PEP 380)

yield from is not merely a shorthand for for item in subgen: yield item. It opens an unimpeded transparent bi-directional channel between the outer caller and the inner subgenerator:

Caller ◄════════════════════════════════════════════════════► Subgenerator
                    Direct Bi-Directional Channel
            (Values, exceptions, and returns flow directly;
             the delegating generator is completely bypassed!)

Minimal example

Save as generator_delegation.py:

# generator_delegation.py
from collections.abc import Generator, Iterator

# 1. Producer subgenerator
def counter_subgen(start: int, stop: int) -> Iterator[int]:
    """Yield integers from start up to stop."""
    current = start
    while current < stop:
        yield current
        current += 1

# 2. Delegating generator using 'yield from'
def compound_pipeline() -> Iterator[int | str]:
    print("  [Delegator] Starting pipeline...")
    # yield from establishes direct pipe to counter_subgen
    yield from counter_subgen(1, 4)
    yield "MIDPOINT"
    yield from counter_subgen(10, 13)
    print("  [Delegator] Pipeline finished.")

def main() -> None:
    for item in compound_pipeline():
        print(f"Pulled: {item}")

if __name__ == "__main__":
    main()

Run via uv run python generator_delegation.py:

  [Delegator] Starting pipeline...
Pulled: 1
Pulled: 2
Pulled: 3
Pulled: MIDPOINT
Pulled: 10
Pulled: 11
Pulled: 12
  [Delegator] Pipeline finished.

Worked examples

Case 1: Two-Way Coroutines for Rolling Averages with `.send()`

When yield appears on the right-hand side of an assignment (received = yield output), the generator can consume data sent by the caller via gen.send(value):

# rolling_average.py
from collections.abc import Generator

def rolling_averager() -> Generator[float, float, None]:
    """Continuously receive incoming float samples and yield the cumulative average."""
    total = 0.0
    count = 0
    average = 0.0

    while True:
        # Suspends yielding current average; receives next incoming sample via .send()
        sample = yield average
        total += sample
        count += 1
        average = total / count

def main() -> None:
    calc = rolling_averager()

    # Priming the generator: Must advance to the first yield before sending data!
    next(calc)  # Returns initial 0.0

    samples = [10.0, 20.0, 60.0, 10.0]
    print("Streaming Telemetry Samples to Coroutine:")
    for sample in samples:
        curr_avg = calc.send(sample)
        print(f"  Sent sample: {sample:4.1f} -> Current Rolling Average: {curr_avg:.2f}")

    # Cleanly terminate the coroutine
    calc.close()

if __name__ == "__main__":
    main()

Run:

uv run python rolling_average.py

Output:

Streaming Telemetry Samples to Coroutine:
  Sent sample: 10.0 -> Current Rolling Average: 10.00
  Sent sample: 20.0 -> Current Rolling Average: 15.00
  Sent sample: 60.0 -> Current Rolling Average: 30.00
  Sent sample: 10.0 -> Current Rolling Average: 25.00

Case 2: Deep Tree Traversal and Recursive Flattening with `yield from`

Nested data structures (such as nested JSON documents, file hierarchies, or AST nodes) can be flattened recursively without managing manual stack arrays:

# tree_flattener.py
from collections.abc import Iterator
from typing import Any

def flatten_tree(element: Any) -> Iterator[Any]:
    """Recursively flatten arbitrarily nested lists and primitives into a flat stream."""
    if isinstance(element, list):
        for item in element:
            # Recursively delegate to subgenerator
            yield from flatten_tree(item)
    else:
        yield element

def main() -> None:
    nested_topology = [
        "gateway-01",
        ["switch-a", ["node-01", "node-02"]],
        ["switch-b", ["node-03", ["storage-01", "storage-02"]]],
        "firewall-01",
    ]

    print("Flattened Infrastructure Inventory:")
    for device in flatten_tree(nested_topology):
        print(f"  Device: {device}")

if __name__ == "__main__":
    main()

Run:

uv run python tree_flattener.py

Output:

Flattened Infrastructure Inventory:
  Device: gateway-01
  Device: switch-a
  Device: node-01
  Device: node-02
  Device: switch-b
  Device: node-03
  Device: storage-01
  Device: storage-02
  Device: firewall-01

Case 3: Capturing Return Values from Subgenerators via `yield from`

In Python generators, a return value statement does not return to a caller like a regular function; it raises StopIteration(value). When consumed by yield from, the subgenerator’s return value is automatically assigned to the target variable:

# subgen_returns.py
from collections.abc import Generator

def accumulate_metrics() -> Generator[None, int, tuple[int, int]]:
    """Accumulate integers sent via .send(); return (count, total) when finished."""
    total = 0
    count = 0
    while True:
        val = yield
        if val is None:  # Sentinel value signaling end of batch
            break
        total += val
        count += 1
    # The return value is embedded inside StopIteration
    return (count, total)

def batch_collector() -> Generator[None, int, str]:
    print("  [Collector] Awaiting batch...")
    # yield from captures the subgenerator's RETURN value directly!
    count, total = yield from accumulate_metrics()
    return f"Batch summary: Processed {count} items with sum {total}"

def main() -> None:
    collector = batch_collector()
    # Prime the generator
    collector.send(None)

    # Stream items into the subgenerator
    for num in [15, 25, 60]:
        collector.send(num)

    # Send sentinel None to trigger return
    try:
        collector.send(None)
    except StopIteration as exit_info:
        summary_result = exit_info.value
        print(f"\nFinal Result: {summary_result}")

if __name__ == "__main__":
    main()

Run:

uv run python subgen_returns.py

Output:

  [Collector] Awaiting batch...

Final Result: Batch summary: Processed 3 items with sum 100

Pitfalls

Pitfall 1: Sending Data to an Unprimed Generator

Attempting to call .send(value) on a generator that has not yet been advanced to its first yield raises TypeError:

def my_coro():
    val = yield

c = my_coro()

# THE TRAP:
try:
    c.send("first_item")  # TypeError!
except TypeError as err:
    print(f"Caught: {err}")

Output:

Caught: can't send non-None value to a just-started generator

# THE FIX: Prime with next() or .send(None)
c = my_coro()
next(c)          # Advanced to initial yield (ready to receive)
c.send("first_item")

Pitfall 2: Re-iterating an Exhausted Generator

Just like generator expressions, generator functions produce single-pass iterators. Once execution exits or returns, the generator is closed:

def generate_ids():
    yield 101
    yield 102

ids = generate_ids()
first_pass = list(ids)  # [101, 102]
second_pass = list(ids) # []  <-- EMPTY! Generator is exhausted!

print(f"First: {first_pass}, Second: {second_pass}")

Exercises

Implement a generator function tail_f(lines: list[str]) that yields lines one by one, simulating continuous streaming.
Build a coroutine running_median() using yield and bisect.insort that receives numbers via .send() and yields the current median.
Write a recursive generator function using yield from that traverses a directory structure on disk and yields all files ending in .log.
Demonstrate how .throw(RuntimeError("Abort")) can be caught inside a generator function’s try ... except block to execute cleanup logic.
Create a generator that yields Fibonacci numbers up to max_val, and measure its memory usage compared to generating a full list of Fibonacci numbers.

Generator Functions and Subgenerator Delegation

Mental model

The yield from Bi-Directional Pipe (PEP 380)

Minimal example

Worked examples

Case 1: Two-Way Coroutines for Rolling Averages with .send()

Case 2: Deep Tree Traversal and Recursive Flattening with yield from

Case 3: Capturing Return Values from Subgenerators via yield from

Pitfalls

Pitfall 1: Sending Data to an Unprimed Generator

Pitfall 2: Re-iterating an Exhausted Generator

Exercises

Further reading

The `yield from` Bi-Directional Pipe (PEP 380)

Case 1: Two-Way Coroutines for Rolling Averages with `.send()`

Case 2: Deep Tree Traversal and Recursive Flattening with `yield from`

Case 3: Capturing Return Values from Subgenerators via `yield from`