multiprocessing

Updated

September 8, 2026

multiprocessing

A process is a separate interpreter with its own memory. ProcessPoolExecutor runs a function in those interpreters and brings the return value back. Use it for CPU work the GIL would serialise in threads. The boring default is a module-level worker, a with pool, and a mandatory if __name__ == "__main__": guard.

Mental model

On Python 3.14 the default start method on Linux is forkserver (not fork). Child processes import your module and look up the worker by name. The worker must be a top-level function. A lambda or a nested def will not unpickle.

The main guard stops the child import from starting another pool. Without it, you can spawn processes until the machine weeps. Always write the guard. Always put pool construction inside main.

Arguments and return values must pickle. Lists of ints are fine. Open sockets and generators are not.

Worked examples

Case 1: ProcessPoolExecutor and the guard

Save as table_totals.py. Each table’s orders are summed in a child. pool.map preserves input order. The with block shuts the pool down.

# table_totals.py
from concurrent.futures import ProcessPoolExecutor


def total_for_table(orders):
    return sum(orders)


def main():
    tables = [
        [1200, 800, 400],
        [2000],
        [500, 500, 500],
    ]
    with ProcessPoolExecutor(max_workers=3) as pool:
        totals = list(pool.map(total_for_table, tables))
    print(totals)
    print(sum(totals))


if __name__ == "__main__":
    main()

Run:

uv run python table_totals.py

Output:

[2400, 2000, 1500]
5900

total_for_table sits at module level on purpose. The children import table_totals and call that name.

Case 2: CPU work that is actually worth a process

Save as cover_hot.py. A tight Python loop is GIL-bound in threads. Processes run it on more than one core. The result is still a number you check.

# cover_hot.py
from concurrent.futures import ProcessPoolExecutor


def count_covers(n):
    total = 0
    for i in range(n):
        total += i % 7
    return total


def main():
    jobs = [200_000, 200_000, 200_000]
    with ProcessPoolExecutor(max_workers=3) as pool:
        parts = list(pool.map(count_covers, jobs))
    print(parts)
    print(sum(parts))


if __name__ == "__main__":
    main()

Run:

uv run python cover_hot.py

Output:

[599994, 599994, 599994]
1799982

Three tiny jobs like this may lose to sequential overhead. That is fine: the pattern is what you copy when a profiler says the loop is hot. Bound max_workers to something near the core count, not to the number of tickets in a year.

Case 3: What you send across the boundary

Save as ticket_payload.py. Send plain data. Get plain data back. Do not send a live TicketWindow object unless you are ready to pickle its guts.

# ticket_payload.py
from concurrent.futures import ProcessPoolExecutor


def label(ticket):
    return f"{ticket['id']} table {ticket['table']}"


def main():
    tickets = [
        {"id": "T-11", "table": 4},
        {"id": "T-12", "table": 2},
    ]
    with ProcessPoolExecutor(max_workers=2) as pool:
        for line in pool.map(label, tickets):
            print(line)


if __name__ == "__main__":
    main()

Run:

uv run python ticket_payload.py

Output:

T-11 table 4
T-12 table 2

A dict of strings and ints pickles. A generator of tickets does not travel as a live cursor; you would send a list (or a path the child reopens).

The trap

Save as nested_worker.py. The worker is nested inside main. forkserver cannot look it up on the child.

# nested_worker.py
from concurrent.futures import ProcessPoolExecutor


def main():
    def total_for_table(orders):
        return sum(orders)

    tables = [[1200, 800], [2000]]
    with ProcessPoolExecutor(max_workers=2) as pool:
        print(list(pool.map(total_for_table, tables)))


if __name__ == "__main__":
    main()

Run:

uv run python nested_worker.py

Output (traceback shortened; the process exits non-zero):

_pickle.PicklingError: Can't pickle local object <function main.<locals>.total_for_table at 0x...>

The child cannot look up a nested function. Lift total_for_table to module level. Keep the guard. Do not wrap the whole file in main without leaving the worker outside.

A second trap: calling ProcessPoolExecutor at import time, above the guard. Children import the module, start more children, and you have a fork bomb. Construction stays inside main.

The boring rule

  • if __name__ == "__main__": on every process-pool script. No exceptions.
  • Worker functions at module level. No nested def, no lambda.
  • Send lists, dicts, numbers, strings. Not sockets, not generators, not the executor itself.
  • with ProcessPoolExecutor(max_workers=...) so shutdown always runs.
  • Threads for I/O, processes for CPU. Do not open a process pool to print three tickets.

Try this

  1. Move total_for_table in Case 1 to nested-inside-main and confirm it breaks. Move it back.
  2. Add a fourth table to table_totals.py. Keep max_workers=3.
  3. Change label to reject a missing table key with ValueError. See how the parent surfaces that error from .map.
  4. Run cover_hot.py with max_workers=1 and with max_workers=3. Feel the wall-clock difference on your machine; do not expect a textbook speedup on tiny jobs.