List, Dict, and Set Comprehensions

List, Dict, and Set Comprehensions

After reading this chapter, you will construct concise, declarative data transformations using list, dictionary, and set comprehensions, apply conditional filters, flatten multi-dimensional structures with nested loops, leverage assignment expressions (:=) to eliminate duplicate calculations, and identify when to refactor overly complex comprehensions into explicit loops.

Mental model

In imperative programming, transforming a collection requires creating an empty accumulator, iterating with a for loop, applying if guards, and appending items one by one. Comprehensions compress this workflow into a single declarative expression that is faster, less error-prone, and visually matches set-builder notation from mathematics:

Imperative Accumulation (4 statements):
  result = []
  for item in source:
      if passes_filter(item):
          result.append(transform(item))

Declarative Comprehension (1 expression):
  result = [ transform(item)  for item in source  if passes_filter(item) ]
             └──────┬──────┘  └───────┬────────┘  └──────────┬─────────┘
                 Expression         Iteration             Condition

Python provides three comprehension syntaxes based on surrounding delimiters:

┌────────────────────┬───────────────────────────────┬───────────────────────────┐
│ Container Type     │ Syntax                        │ Result Type               │
├────────────────────┼───────────────────────────────┼───────────────────────────┤
│ List Comprehension │ [expr for item in iterable]   │ list (ordered, mutable)   │
│ Set Comprehension  │ {expr for item in iterable}   │ set (unique, unordered)   │
│ Dict Comprehension │ {k: v for item in iterable}   │ dict (key-value mapping)  │
└────────────────────┴───────────────────────────────┴───────────────────────────┘

For nested loops, the comprehension syntax preserves standard left-to-right nesting order:

[ x for row in matrix for x in row ]
  │ └───────┬───────┘ └─────┬────┘
  │     Outer Loop      Inner Loop
  ▼
Equivalent to:
  for row in matrix:
      for x in row:
          yield x

---

Minimal example

Save as comprehensions_overview.py:

# comprehensions_overview.py
def main() -> None:
    raw_nodes = [" worker-01 ", "Worker-02", "  api-01", "WORKER-01", "db-01 "]

    # 1. List Comprehension: Normalize names and filter worker nodes
    worker_nodes = [
        node.strip().lower()
        for node in raw_nodes
        if "worker" in node.lower()
    ]
    print(f"Worker List: {worker_nodes}")

    # 2. Set Comprehension: Deduplicate normalized names
    unique_workers = {
        node.strip().lower()
        for node in raw_nodes
        if "worker" in node.lower()
    }
    print(f"Unique Worker Set: {sorted(unique_workers)}")

    # 3. Dict Comprehension: Build node-to-role lookup map
    node_roles = {
        node.strip().lower(): ("database" if "db" in node else "compute")
        for node in raw_nodes
    }
    print("Node Roles:")
    for name, role in sorted(node_roles.items()):
        print(f"  {name:10} -> {role}")

if __name__ == "__main__":
    main()

Run via uv run python comprehensions_overview.py:

Worker List: ['worker-01', 'worker-02', 'worker-01']
Unique Worker Set: ['worker-01', 'worker-02']
Node Roles:
  api-01     -> compute
  db-01      -> database
  worker-01  -> compute
  worker-02  -> compute

---

Worked examples

Case 1: Ingesting and Sanitizing Raw Metric Records (List Comprehensions)

When ingesting metrics from HTTP log lines or sensors, raw records frequently contain malformed entries or values outside acceptable thresholds. A list comprehension combines parsing and filtering cleanly:

# metric_sanitizer.py
def sanitize_latencies(raw_samples: list[str]) -> list[float]:
    """Parse comma-separated millisecond strings, filter errors (-1.0), and convert to seconds."""
    return [
        float(val.strip()) / 1000.0
        for val in raw_samples
        if val.strip() != "" and float(val.strip()) >= 0.0
    ]

def main() -> None:
    samples = ["120.5", " 45.2 ", "-1.0", "", "850.0", "312.4", "-99.0", "12.0"]
    cleaned_seconds = sanitize_latencies(samples)

    print(f"Raw sample count:     {len(samples)}")
    print(f"Valid sample count:   {len(cleaned_seconds)}")
    print("Cleaned values (seconds):")
    for sec in cleaned_seconds:
        print(f"  {sec:.4f}s")

if __name__ == "__main__":
    main()

Run:

uv run python metric_sanitizer.py

Output:

Raw sample count:     8
Valid sample count:   5
Cleaned values (seconds):
  0.1205s
  0.0452s
  0.8500s
  0.3124s
  0.0120s

Why this matters: Notice how [f(x) for x in xs if cond(x)] replaces chained map() and filter() calls. In Python, comprehensions are more readable than list(map(..., filter(...))) and avoid the overhead of lambda function call frames.

---

Case 2: Inverting and Indexing Service Registries (Dict Comprehensions)

In infrastructure management, service registries often map hostnames to IP addresses. SRE tooling frequently requires reverse lookups (IP-to-hostname) or normalized sub-mappings:

# registry_indexing.py
def main() -> None:
    # Forward map: Hostname -> IPv4
    dns_records = {
        "gateway.internal": "10.0.1.1",
        "auth.internal":    "10.0.1.15",
        "db-pri.internal":  "10.0.2.100",
        "db-sec.internal":  "10.0.2.101",
        "cache.internal":   "10.0.3.50",
    }

    # 1. Reverse Map: IP -> Hostname
    reverse_dns = {ip: host for host, ip in dns_records.items()}

    # 2. Filtered Subnet Map: Database subnet (10.0.2.x) only
    db_subnet = {
        host: ip
        for host, ip in dns_records.items()
        if ip.startswith("10.0.2.")
    }

    print("--- Reverse DNS Lookup (IP -> Host) ---")
    for ip, host in sorted(reverse_dns.items()):
        print(f"  {ip:12} -> {host}")

    print("\n--- DB Subnet Only ---")
    for host, ip in sorted(db_subnet.items()):
        print(f"  {host:16} -> {ip}")

if __name__ == "__main__":
    main()

Run:

uv run python registry_indexing.py

Output:

--- Reverse DNS Lookup (IP -> Host) ---
  10.0.1.1     -> gateway.internal
  10.0.1.15    -> auth.internal
  10.0.2.100   -> db-pri.internal
  10.0.2.101   -> db-sec.internal
  10.0.3.50    -> cache.internal

--- DB Subnet Only ---
  db-pri.internal  -> 10.0.2.100
  db-sec.internal  -> 10.0.2.101

---

Case 3: Matrix Flattening and Cartesian Grids (Nested Loops)

When managing multi-zone cloud deployments or flattening nested tabular datasets, nested comprehensions process multi-dimensional data without requiring manual stack allocations:

# cluster_topology.py
def main() -> None:
    regions = ["us-east", "eu-central"]
    zones = ["a", "b"]
    tiers = ["web", "api"]

    # Cartesian Product: Region x Zone x Tier
    deploy_targets = [
        f"{region}-{zone}-{tier}"
        for region in regions
        for zone in zones
        for tier in tiers
    ]

    print(f"Total Deployment Targets: {len(deploy_targets)}")
    for target in deploy_targets:
        print(f"  Target: {target}")

    # 2D Grid Flattening
    matrix = [
        [10, 20, 30],
        [40, 50, 60],
        [70, 80, 90],
    ]
    flattened = [val for row in matrix for val in row if val > 25]
    print(f"\nFiltered Flattened Matrix (values > 25): {flattened}")

if __name__ == "__main__":
    main()

Run:

uv run python cluster_topology.py

Output:

Total Deployment Targets: 8
  Target: us-east-a-web
  Target: us-east-a-api
  Target: us-east-b-web
  Target: us-east-b-api
  Target: eu-central-a-web
  Target: eu-central-a-api
  Target: eu-central-b-web
  Target: eu-central-b-api

Filtered Flattened Matrix (values > 25): [30, 40, 50, 60, 70, 80, 90]

Rule for Nested Comprehensions: Read the clauses from left to right. The first for clause is the outermost loop, and subsequent for clauses are nested inside it.

---

Case 4: Avoiding Duplicate Computation with the Walrus Operator (:=)

A common anti-pattern in comprehensions is calling an expensive function or parsing routine twice: once in the if clause to filter, and again in the expression clause to transform:

# walrus_comprehension.py
import json

def parse_payload(raw_text: str) -> dict[str, str] | None:
    """Simulate parsing an incoming JSON string payload."""
    try:
        data = json.loads(raw_text)
        return data if isinstance(data, dict) else None
    except json.JSONDecodeError:
        return None

def main() -> None:
    incoming_logs = [
        '{"service": "auth", "status": "200"}',
        'MALFORMED_HEADER',
        '{"service": "payment", "status": "503"}',
        '{}',
        '{"service": "cart", "status": "404"}',
    ]

    # Use the walrus operator (:=) in the condition to bind parsed_obj
    # and reuse it directly in the output expression without re-parsing!
    valid_services = [
        parsed["service"]
        for log in incoming_logs
        if (parsed := parse_payload(log)) is not None and "service" in parsed
    ]

    print("Parsed Active Services:")
    for svc in valid_services:
        print(f"  Service: {svc}")

if __name__ == "__main__":
    main()

Run:

uv run python walrus_comprehension.py

Output:

Parsed Active Services:
  Service: auth
  Service: payment
  Service: cart

---

Pitfalls

Pitfall 1: Misusing Comprehensions for Side Effects

Comprehensions are expressions designed to produce collections. Using them purely for their side effects (such as printing or writing to files) needlessly allocates in-memory lists that are immediately discarded:

# THE TRAP: Wasted memory and obscure intent
tasks = ["build", "test", "package", "deploy"]
[print(f"Executing: {task}") for task in tasks]  # Allocates [None, None, None, None]!

# THE FIX: Use a clean, standard for loop
for task in tasks:
    print(f"Executing: {task}")

---

Pitfall 2: Over-Nesting (Write-Only Code)

When a comprehension contains more than two loops or multiple complex conditionals, readability collapses:

# THE TRAP: Write-only comprehension
bad = [
    f(x, y, z)
    for x in dataset
    if x.is_valid()
    for y in x.sub_items
    if y.is_ready()
    for z in y.tokens
    if z.is_active()
]

# THE FIX: Refactor into an explicit generator or loop with clear variable names
def extract_active_tokens(dataset):
    for x in dataset:
        if not x.is_valid():
            continue
        for y in x.sub_items:
            if not y.is_ready():
                continue
            for z in y.tokens:
                if z.is_active():
                    yield f(x, y, z)

good = list(extract_active_tokens(dataset))

---

Pitfall 3: Scope Isolation (Python 3 Comprehension Scope)

In Python 2, the loop variable in a list comprehension leaked into the enclosing scope and overwrote existing variables. In modern Python (3.x and 3.14), comprehensions execute in their own implicit function scope:

# comprehension_scope.py
def main() -> None:
    x = "original_outer_value"
    squares = [x * x for x in range(5)]

    print(f"Squares: {squares}")
    # In Python 3+, x remains unmodified!
    print(f"Outer x is preserved: {x}")

if __name__ == "__main__":
    main()

Output:

Squares: [0, 1, 4, 9, 16]
Outer x is preserved: original_outer_value

---

Exercises

1. Given a list of filenames ["app.py", "README.md", "server.py", "config.json", "utils.py"], write a list comprehension that extracts only the stems of Python files (.py), producing ["app", "server", "utils"].
2. Given a dictionary mapping usernames to email addresses, write a dict comprehension that extracts only accounts belonging to the @company.com domain, lowercasing both keys and values.
3. Given a list of sentences, write a set comprehension that extracts all unique words that are 5 letters or longer, stripping punctuation and whitespace.
4. Using nested comprehensions, generate a 5x5 identity matrix (a list of 5 lists, each containing 5 numbers, where element (i, j) is 1 if i == j and 0 otherwise).
5. Given a list of HTTP status code strings ["200", "invalid", "404", "500", "abc", "301"], use an assignment expression (:=) inside a comprehension to filter and convert valid integer status codes that are >= 400.

---

Further reading

• PEP 202: List Comprehensions.
• PEP 274: Dict Comprehensions.
• PEP 572: Syntax for Assignment Expressions (the := walrus operator).
• Python Documentation: Data Structures – List Comprehensions.