High-Performance File I/O with Memory-Mapped Files (mmap)

Updated

September 7, 2026

High-Performance File I/O with Memory-Mapped Files (mmap)

After reading this chapter, you will master operating-system-level virtual memory mapping using Python’s mmap module, inspect multi-gigabyte files with a near-zero memory footprint, execute in-place binary substitutions directly on disk, search disk contents using regular expressions without loading strings into RAM, and establish shared memory IPC.

Mental model

Traditional file I/O requires copying data multiple times: from disk storage into the OS kernel buffer cache, and then from the kernel buffer into a Python heap-allocated bytes or str object via the read() system call:

Standard File I/O (Two Copies):
  Disk File ──▶ [ OS Kernel Page Cache ] ──▶ read() copies ──▶ Python Heap Object
                                                               (High RAM usage!)

A memory-mapped file (mmap) asks the operating system kernel to map a file’s disk blocks directly into your process’s virtual address space. Accessing the mmap object is as fast as accessing a memory array, with the OS page cache transparently paging blocks in and out on demand:

Memory-Mapped File (Zero User-Space Buffer Copies):
┌────────────────────────────────────────────────────────┐
│ Disk Storage (e.g. 50 GB log file or database tables)   │
└────────────────────────┬───────────────────────────────┘
                         │ Direct Virtual Memory Mapping (mmap)
                         ▼
┌────────────────────────────────────────────────────────┐
│ Process Virtual Address Space                          │
│   mm[0:1024] reads directly from kernel page cache     │
│   mm[offset] = b'X' writes directly to page cache      │
│   (Instant random access, O(1) Python heap footprint)  │
└────────────────────────────────────────────────────────┘

Minimal example

Save as mmap_overview.py:

# mmap_overview.py
import mmap
import tempfile

def main() -> None:
    # 1. Create a temporary file with sample log data
    with tempfile.NamedTemporaryFile(delete=False) as f:
        f.write(b"2026-09-07 12:00:00 [INFO] Cluster initialized\n")
        f.write(b"2026-09-07 12:00:05 [WARN] Disk watermark 85%\n")
        f.write(b"2026-09-07 12:00:10 [ERR ] Failed node: srv-04\n")
        filepath = f.name

    # 2. Map the file into virtual memory
    with open(filepath, "r+b") as file_obj:
        # mmap(fileno, length): 0 means map the entire file
        with mmap.mmap(file_obj.fileno(), 0) as mm:
            print(f"Memory-mapped file size: {mm.size()} bytes")

            # Search without loading file into Python RAM
            target = b"[ERR ]"
            offset = mm.find(target)
            print(f"Found '{target.decode()}' at byte offset: {offset}")

            # Read a specific slice directly from disk memory
            mm.seek(offset)
            error_line = mm.readline()
            print(f"Read line: {error_line.decode().strip()}")

            # In-place disk mutation: overwrite [ERR ] with [CRIT]
            mm.seek(offset)
            mm.write(b"[CRIT]")
            mm.flush()  # Ensure changes are written back to physical disk

    # Verify mutation persisted in physical file
    with open(filepath, "rb") as verify_f:
        print(f"\nPersisted file content:\n{verify_f.read().decode()}")

if __name__ == "__main__":
    main()

Run via uv run python mmap_overview.py:

Memory-mapped file size: 140 bytes
Found '[ERR ]' at byte offset: 113
Read line: [ERR ] Failed node: srv-04

Persisted file content:
2026-09-07 12:00:00 [INFO] Cluster initialized
2026-09-07 12:00:05 [WARN] Disk watermark 85%
2026-09-07 12:00:10 [CRIT] Failed node: srv-04

Worked examples

Case 1: Regex Searching Gigabyte Logs with Zero RAM Allocation

When auditing 20 GB web server logs, calling file.read() triggers an Out-Of-Memory error. Because mmap supports Python’s buffer protocol, standard re expressions can search directly over the memory map:

# mmap_regex_search.py
import mmap
import re
import tempfile

def generate_large_log(filepath: str, line_count: int = 10_000) -> None:
    with open(filepath, "wb") as f:
        for i in range(line_count):
            if i == 4200:
                f.write(b'192.168.1.100 - [2026-09-07] "GET /admin/secret" 403 128\n')
            elif i == 8900:
                f.write(b'10.0.0.50 - [2026-09-07] "POST /api/v1/auth" 500 256\n')
            else:
                f.write(b'127.0.0.1 - [2026-09-07] "GET /health" 200 32\n')

def main() -> None:
    with tempfile.NamedTemporaryFile(delete=False) as tmp:
        filepath = tmp.name

    generate_large_log(filepath)

    pattern = re.compile(rb'(\d+\.\d+\.\d+\.\d+) - .* "([A-Z]+ [^"]+)" ([45]\d\d)')

    with open(filepath, "rb") as f:
        with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
            print("Scanning log file with zero heap allocation:")
            for match in pattern.finditer(mm):
                ip, endpoint, status = match.groups()
                print(f"  Anomaly detected at offset {match.start():6d}: IP={ip.decode()} Status={status.decode()} ({endpoint.decode()})")

if __name__ == "__main__":
    main()

Run:

uv run python mmap_regex_search.py

Output:

Scanning log file with zero heap allocation:
  Anomaly detected at offset 193200: IP=192.168.1.100 Status=403 (GET /admin/secret)
  Anomaly detected at offset 409411: IP=10.0.0.50 Status=500 (POST /api/v1/auth)

Case 2: In-Place Binary Patching and Header Updates

In firmware flashing and database file management, updating records in place avoids rewriting the entire file:

# binary_header_patcher.py
import mmap
import struct
import tempfile

# Binary Header Format: 4-byte Magic, 4-byte Version, 8-byte Checksum
HEADER_FORMAT = "!4sIQ"

def main() -> None:
    with tempfile.NamedTemporaryFile(delete=False) as f:
        # Write initial header
        initial_bytes = struct.pack(HEADER_FORMAT, b"BLOB", 1, 0x1122334455667788)
        f.write(initial_bytes + (b"\x00" * 128))
        filepath = f.name

    with open(filepath, "r+b") as f:
        with mmap.mmap(f.fileno(), 0) as mm:
            magic, version, checksum = struct.unpack(HEADER_FORMAT, mm[:16])
            print("Initial Header:")
            print(f"  Magic:    {magic}")
            print(f"  Version:  {version}")
            print(f"  Checksum: 0x{checksum:X}")

            # Bump version to 2 and update checksum in place!
            new_header = struct.pack(HEADER_FORMAT, b"BLOB", 2, 0xAABBCCDDEEFF0011)
            mm[:16] = new_header
            mm.flush()

    with open(filepath, "rb") as f:
        magic, version, checksum = struct.unpack(HEADER_FORMAT, f.read(16))
        print("\nVerified Header on Disk:")
        print(f"  Magic:    {magic}")
        print(f"  Version:  {version}")
        print(f"  Checksum: 0x{checksum:X}")

if __name__ == "__main__":
    main()

Run:

uv run python binary_header_patcher.py

Output:

Initial Header:
  Magic:    b'BLOB'
  Version:  1
  Checksum: 0x1122334455667788

Verified Header on Disk:
  Magic:    b'BLOB'
  Version:  2
  Checksum: 0xAABBCCDDEEFF0011

Case 3: Anonymous Shared Memory IPC Between Processes

By specifying -1 as the file descriptor, mmap allocates an anonymous shared memory segment backed by RAM rather than a physical file, enabling zero-disk IPC:

# anonymous_shared_memory.py
import mmap
import os
import time

def main() -> None:
    # fileno = -1 creates an anonymous memory map
    shm = mmap.mmap(-1, 1024)

    # Fork child process
    pid = os.fork()

    if pid == 0:
        # Child process: Wait for parent to write data
        time.sleep(0.02)
        shm.seek(0)
        message = shm.readline().decode("utf-8").strip()
        print(f"[Child PID {os.getpid()}] Read from shared memory: '{message}'")
        shm.close()
        os._exit(0)
    else:
        # Parent process: Write data to shared memory
        shm.write(b"SYNC_TOKEN: 0x8FA2BC99\n")
        print(f"[Parent PID {os.getpid()}] Wrote token to anonymous shared memory.")
        os.waitpid(pid, 0)
        shm.close()

if __name__ == "__main__":
    main()

Run:

uv run python anonymous_shared_memory.py

Output:

[Parent PID 12345] Wrote token to anonymous shared memory.
[Child PID 12346] Read from shared memory: 'SYNC_TOKEN: 0x8FA2BC99'

Pitfalls

Pitfall 1: Attempting to Map an Empty (0-Byte) File

The operating system cannot map a file with a length of 0 bytes. Attempting to do so raises ValueError:

# THE TRAP:
import tempfile
import mmap

with tempfile.NamedTemporaryFile() as empty_file:
    # empty_file has size 0
    try:
        mm = mmap.mmap(empty_file.fileno(), 0)
    except ValueError as err:
        print(f"Caught: {err}")

Output:

Caught: cannot mmap an empty file

The Fix: Ensure the file contains at least 1 byte before mapping, or check os.path.getsize(filepath) > 0.

Pitfall 2: Modifying File Size Outside mmap

If another process or thread truncates the underlying file while a memoryview or mmap is open, reading beyond the new end of file triggers an uncatchable OS SIGBUS (Bus Error) signal, crashing the Python process immediately. Always coordinate file resizing with process-wide locks.

Exercises

  1. Create a 10 MB file filled with random ASCII bytes. Use mmap.find() to locate all occurrences of the word "TARGET".
  2. Write a script that counts line breaks (\n) in a large text file using mmap and compare its execution time and memory usage against file.readlines().
  3. Implement a circular ring buffer inside an anonymous memory map (mmap(-1, size)) shared between two forked processes.
  4. Open a read-only binary file using access=mmap.ACCESS_READ and verify that attempting to write to the memory map raises TypeError.
  5. Build an in-place string replacement function that searches for a 4-letter token and replaces it with a new 4-letter token across a 50 MB log file.

Further reading

  • Python Documentation: mmap — Memory-mapped file support.
  • Linux Programmer’s Manual: mmap(2) — Map files or devices into memory.
  • David Beazley: Python Cookbook (Recipe 5.18: Memory Mapping an I/O File).