Memory-Mapped Files

Updated

September 8, 2026

Memory-Mapped Files

Overview

mmap maps a file (or anonymous memory) into the process address space so you read and write with normal memory operations while the kernel pages data to/from disk. Go exposes this via golang.org/x/sys/unix (Unix) or careful use of platform APIs—not as a one-liner in the core stdlib.

When to use mmap vs os.Read / bufio:

Prefer read/stream Prefer mmap
Sequential scans of huge files Random access into large read-mostly files
Simple code, small files Shared memory between processes (advanced)
Portable stdlib only OS-specific performance work

Deep dive context: 251 mmap vs pread.

Mental model

  process virtual memory
  +---------------------------+
  |  mapped region (pages)    |  <-->  file on disk
  +---------------------------+
         page fault loads

Writes to a shared map may update the file (depending on flags). Always understand PROT_* and MAP_* flags.

Read-only map (Unix sketch)

//go:build unix

package mmapfile

import (
    "fmt"
    "os"

    "golang.org/x/sys/unix"
)

type RO struct {
    data []byte
}

func OpenRO(path string) (*RO, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    defer f.Close()

    st, err := f.Stat()
    if err != nil {
        return nil, err
    }
    size := int(st.Size())
    if size == 0 {
        return &RO{data: []byte{}}, nil
    }

    data, err := unix.Mmap(int(f.Fd()), 0, size, unix.PROT_READ, unix.MAP_SHARED)
    if err != nil {
        return nil, err
    }
    return &RO{data: data}, nil
}

func (m *RO) Bytes() []byte { return m.data }

func (m *RO) Close() error {
    if len(m.data) == 0 {
        return nil
    }
    return unix.Munmap(m.data)
}
m, err := mmapfile.OpenRO("large.bin")
if err != nil {
    log.Fatal(err)
}
defer m.Close()
fmt.Printf("first 16: %x\n", m.Bytes()[:min(16, len(m.Bytes()))])

Critical: after Munmap, the slice is invalid—do not retain sub-slices.

Write map (careful)

data, err := unix.Mmap(int(f.Fd()), 0, size,
    unix.PROT_READ|unix.PROT_WRITE, unix.MAP_SHARED)
// mutate data[i] = ...
// unix.Msync(data, unix.MS_SYNC) // durability

Crash consistency is harder than write-temp-rename. For config files, prefer atomic replace (chapter 142). Use writable mmap for specialized stores/databases, not casual state.

Anonymous maps

data, err := unix.Mmap(-1, 0, size, unix.PROT_READ|unix.PROT_WRITE,
    unix.MAP_ANON|unix.MAP_PRIVATE)

Useful for large scratch buffers; still process-private memory.

Safety with Go

  1. No GC pointers into foreign maps incorrectly — mmap slices are plain []byte; fine for bytes, not for storing Go pointers.
  2. Bounds — map length is fixed at map time; file growth needs remap.
  3. Concurrent access — concurrent reads OK for RO; writers need sync.
  4. Windows — different API (golang.org/x/sys/windows); isolate with build tags.

When mmap loses

  • One-pass line scan of a log → bufio.Scanner is simpler and fine
  • Network filesystems with weird consistency
  • Tiny files → overhead dominates

Minimal CLI: mmapsum

// sha256 of mmapped file without ReadAll
h := sha256.New()
_, _ = h.Write(m.Bytes())
fmt.Printf("%x\n", h.Sum(nil))

Compare memory profile vs io.Copy(h, f) for multi-GB inputs.

Rules of thumb

Do Don’t
Munmap exactly once Use slice after unmap
Prefer RO maps for inspection tools mmap for every config write
Document OS build tags Assume mmap code is portable without tags
Measure vs pread/Read Assume mmap is always faster

Try next

  1. Map /etc/hosts read-only and print line count without ReadFile.
  2. Benchmark mmap vs io.Copy SHA-256 on a 500MB file.
  3. Read the deep-dive chapter on mmap vs pread for allocator interaction.