Profiling, Optimization, and Native C/Rust Interoperability

Updated

September 7, 2026

Profiling, Optimization, and Native C/Rust Interoperability

After reading this chapter, you will master performance diagnostics using cProfile and pstats, track heap allocations and memory spikes with tracemalloc, interface with native C libraries using ctypes, and architect high-speed compiled extensions with PyO3 and Rust.

Mental model

Performance engineering follows an evidence-based feedback loop: never optimize without profiling data.

Performance Engineering Pipeline:
  [ Macro Measurement ] ──▶ time.perf_counter() ──▶ Identifies slow high-level subsystems
           │
           ▼
  [ Deterministic Profiling ] ──▶ cProfile / pstats ──▶ Identifies exact hotspot functions
           │
           ▼
  [ Memory Profiling ]  ──▶ tracemalloc ──▶ Locates exact lines allocating heap memory
           │
           ▼
  [ Optimization Paths ]:
    1. Algorithmic Fix (O(N^2) -> O(N log N) using sets/dicts)
    2. Built-in C-Accelerators (itertools, collections.deque)
    3. Native FFI Binding (ctypes or PyO3 Rust extension)

Minimal example

Save as profiling_and_ctypes.py:

# profiling_and_ctypes.py
import cProfile
import ctypes
import pstats
import tracemalloc

def cpu_heavy_function() -> list[int]:
    """Function with an intentional list-growing bottleneck."""
    data = []
    for i in range(200_000):
        data.append(i * 2)
    return data

def demonstrate_profiling() -> None:
    # 1. Deterministic Profiling with cProfile
    profiler = cProfile.Profile()
    profiler.enable()
    
    cpu_heavy_function()
    
    profiler.disable()
    stats = pstats.Stats(profiler).sort_stats("tottime")
    print("--- Top Profiler Entries ---")
    stats.print_stats(3)

def demonstrate_ctypes() -> None:
    # 2. Native C interop via ctypes (Calling libc directly)
    libc = ctypes.CDLL(None)  # Accesses standard C library
    print("\n--- Native C Interop with ctypes ---")
    # Call standard C 'puts' function
    libc.puts(b"Hello from native C library puts()!")

def main() -> None:
    demonstrate_profiling()
    demonstrate_ctypes()

if __name__ == "__main__":
    main()

Run via uv run python profiling_and_ctypes.py:

--- Top Profiler Entries ---
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.012    0.012    0.012    0.012 ...:cpu_heavy_function
        ...
--- Native C Interop with ctypes ---
Hello from native C library puts()!

Worked examples

Case 1: Locating Memory Leaks with tracemalloc

tracemalloc tracks every memory block allocated by CPython, recording the exact Python source file and line number:

# memory_leak_tracer.py
import tracemalloc

def simulate_memory_allocation() -> list[str]:
    # Allocate 100,000 strings
    return [f"cluster_telemetry_event_node_{i}" for i in range(100_000)]

def main() -> None:
    # Start tracking memory allocations
    tracemalloc.start()

    snapshot_before = tracemalloc.take_snapshot()
    data = simulate_memory_allocation()
    snapshot_after = tracemalloc.take_snapshot()

    # Compare snapshots to identify top allocating lines
    top_diffs = snapshot_after.compare_to(snapshot_before, "lineno")

    print("--- Top Memory Allocating Lines ---")
    for stat in top_diffs[:3]:
        print(f"  {stat}")

    current, peak = tracemalloc.get_traced_memory()
    print(f"\nCurrent memory: {current / (1024*1024):.2f} MB | Peak memory: {peak / (1024*1024):.2f} MB")
    tracemalloc.stop()

if __name__ == "__main__":
    main()

Run:

uv run python memory_leak_tracer.py

Output:

--- Top Memory Allocating Lines ---
  ...:simulate_memory_allocation: size=7524 KiB (+7524 KiB), count=100000 (+100000)
Current memory: 7.37 MB | Peak memory: 7.37 MB

Case 2: Calling Native C Functions with ctypes.Structure

You can define C-compatible memory structs in Python and pass them directly across native library boundaries:

# c_struct_ffi.py
import ctypes

class Point2D(ctypes.Structure):
    _fields_ = [
        ("x", ctypes.c_double),
        ("y", ctypes.c_double),
    ]

def main() -> None:
    pt = Point2D(10.5, 20.25)
    print(f"Python C-compatible struct: x={pt.x}, y={pt.y}")
    print(f"Size of struct in C memory: {ctypes.sizeof(pt)} bytes (2 x 8-byte doubles)")

if __name__ == "__main__":
    main()

Run:

uv run python c_struct_ffi.py

Output:

Python C-compatible struct: x=10.5, y=20.25
Size of struct in C memory: 16 bytes (2 x 8-byte doubles)

Case 3: Modern High-Performance Extensions: PyO3 and Rust

When pure Python algorithms cannot meet latency budgets (e.g. cryptography, parsers, simulation), modern Python projects use PyO3 to author extensions in Rust.

Benefits of PyO3 over legacy C-extensions: - Memory safety guarantees without segmentation faults. - Seamless release of the GIL using Python::allow_threads. - Zero-cost conversion between Python types and Rust standard types.

Pitfalls

Pitfall 1: Premature Micro-Optimization

Bikeshedding string concatenations or swapping operators before profiling is wasted effort. In 95% of applications, performance bottlenecks are concentrated in I/O wait times or poorly chosen algorithmic complexity (\(O(N^2)\) lookups in lists).

Pitfall 2: Memory Hazards in ctypes

ctypes bypasses Python’s memory protections. Passing an incorrect pointer, writing past an array boundary, or calling a function with the wrong argument types will immediately trigger a hard crash (Segmentation Fault) that terminates the entire Python process.

Exercises

  1. Profile an algorithm that searches for items in a list vs a set using cProfile and compare their execution profiles.
  2. Use tracemalloc to identify how much RAM is consumed by 50,000 instances of a standard class versus a class using __slots__.
  3. Call the C standard library qsort function using ctypes to sort an array of C integers.
  4. Export profiling results to a .prof file using cProfile.run(..., filename="out.prof") and inspect it using pstats.

Further reading

  • Python Standard Library: cProfile, pstats, tracemalloc, and ctypes modules.
  • PyO3 Project: Rust bindings for the Python interpreter.
  • PEP 659: Zero-cost exception handling.