Nix performance

Updated

July 30, 2026

Nix performance

Goal: Measure where time goes (eval vs download vs build), tune max-jobs / cores / remote builders / caches, and record a before/after rebuild timing with settings you can justify.

Why this chapter exists

NixOS rebuilds feel “slow” for different reasons. Cranking --max-jobs 128 on a laptop with 8GB RAM and a spinning disk can thrash harder than serial builds. Capstone CI budgets and lab iteration speed both require measurement first.

Theory 1 — The rebuild time budget

A flake host rebuild roughly pays:

eval (modules + nixpkgs)  →  realize (substitute or build)  →  activate
Phase Dominated by Knobs
Eval CPU single-thread-ish, memory, module count, IFD eval-cache, avoid IFD, slim modules
Substitute Network, cache hit rate, compression binary caches, closer mirrors
Build CPU cores, jobs, IO, remote builders max-jobs, cores, builders
Activate Service restarts, switch strategy switch vs boot vs test 
# rough wall clock
/usr/bin/time -v nixos-rebuild build --flake .#<host>
# or
time nix build .#nixosConfigurations.<host>.config.system.build.toplevel

Separate cold (empty download cache / first eval) vs warm timings or you will lie to yourself.

Elitebook 16 GB reality

Workload Expectation
Leaf nix eval hostname Seconds
Warm toplevel after small edit Often substitute-heavy; minutes if many new builds
Cold after lock bump Long; plan coffee + cache
Parallel max-jobs=8 link storms OOM risk; prefer 2–4 jobs

Theory 2 — Jobs vs cores

Definitions

Setting Meaning
max-jobs How many derivations realized in parallel
cores How many cores advertised to each derivation (NIX_BUILD_CORES / enableParallelBuilding) 
nix.settings = {
  max-jobs = 4;    # or "auto"
  cores = 2;       # per-job parallelism
};

CLI overrides:

nix build -j 4 --cores 2 .#…
nixos-rebuild build --flake .#host --max-jobs 4 --cores 2

Choosing numbers (rules of thumb)

Hardware Starting point
4c/8t laptop, 16GB max-jobs = 2–4, cores = 2
16c desktop, 64GB max-jobs = 8, cores = 2–4
Many small builds higher max-jobs, lower cores
Few huge compiles (gcc/llvm) lower max-jobs, higher cores
Tiny rootfs / low RAM lower both; watch OOM

Constraint: max-jobs * cores should not massively exceed physical cores + RAM headroom, especially with linking-heavy packages.

Remote builders

nix.distributedBuilds = true;
nix.buildMachines = [ {
  hostName = "builder";
  system = "x86_64-linux";
  maxJobs = 8;
  speedFactor = 2;
  supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
  mandatoryFeatures = [ ];
} ];

Offload large builds; keep eval local unless you know your setup.

Feature Why list it
kvm nixosTest VMs
big-parallel large compiles allowed
nixos-test test driver builds

Theory 3 — Eval performance

Eval cache

Modern Nix can cache evaluation results for flakes when inputs unchanged.

nix config show | grep -i eval
# features / options evolve; check your version docs for eval-cache

Practical wins:

  • Stable flake.lock → better cache reuse
  • Avoid unnecessary --impure
  • Avoid Import From Derivation (IFD) when possible

IFD cost

# expensive pattern: build something at eval time
let pkg = import (pkgs.fetchFromGitHub { … }) { };
in pkg.something

IFD forces realization during eval—serial and painful in CI. Prefer flake inputs and generated files committed or produced in build phase.

Module count

Every module imported has a cost. Shared modules should be focused, not a megamodule imported everywhere with giant mkIf trees if you can split roles.

Memory

Large evals can exceed default stack/heap. Symptoms: evaluator OOM, thrashing. Fix structure before only raising limits.

# rough: watch RSS while evaluating toplevel
/usr/bin/time -v nix eval .#nixosConfigurations.lab.config.system.build.toplevel --raw >/dev/null

Theory 4 — Download & store IO

Lever Effect
Hit cache.nixos.org + project cache Avoid local compiles
http2 / modern curl stack Throughput (usually already fine)
Keep disk free Avoid GC mid-build
min-free / max-free auto-GC Prevent disk full mid-CI
SSD vs HDD Link farms and unpack dominate on HDD 
nix.settings = {
  min-free = 1073741824;   # 1 GiB
  max-free = 5368709120;   # 5 GiB
};

On Elitebook labs with small disks, aggressive auto-GC can surprise you mid-ISO build—know the knobs.

Theory 5 — What not to do

Anti-pattern Why
sandbox = false for speed Trust/purity regression
Disable binary caches “to force builds” in daily driver Wastes hours
Infinite max-jobs on laptop OOM + thermal throttle
Micro-optimizing before cache hits Measure substitute rate first
Blindly copying someone else’s nix.conf Hardware differs
Timing only after switch Includes service downtime noise

Theory 6 — CI budget design

Goal Technique
PR feedback < N minutes Cache push from main; build only changed hosts if matrixed
Avoid rebuild world Pin lock; share /nix/store cache between jobs carefully
nixosTest cost Nightly checks.slow-*; PR runs fast subset
Fail fast Eval/checks before ISO builds

Write target budgets in PERF.md even for homelab CI.

Worked example — Measurement protocol

# 1) Note config
nix config show | egrep 'max-jobs|cores|substituters' | tee before-config.txt

# 2) Warm caches optionally with a cheap build
nix build nixpkgs#hello

# 3) Wall-clock system build (no switch)
/usr/bin/time -p nix build \
  .#nixosConfigurations.lab.config.system.build.toplevel \
  2>&1 | tee time-before.txt

# 4) Change nix.settings via configuration.nix / flake; rebuild once to apply daemon settings

# 5) Repeat timing → time-after.txt

Daemon-level settings change may need:

sudo systemctl restart nix-daemon

(On NixOS after a config switch that changes nix.settings.)

Interpreting results

Pattern Interpretation
Eval leaf fast, toplevel slow, lots of building Real compile cost; builders/cores
Many copying path Substitute/network; cache placement
Eval alone dominates IFD/modules; not max-jobs
After setting change wall clock worse Thrash; lower parallelism

Lab 0 — Workspace

mkdir -p ~/lab/nixos/internals/perf
cd ~/lab/nixos/internals/perf

Use your Capstone/lab flake host.

Lab 1 — Baseline timings

Record in PERF.md:

Run Command Wall seconds Notes (cold/warm)
A eval hostname only
B nix build toplevel
C nixos-rebuild build 
time nix eval .#nixosConfigurations.<host>.config.networking.hostName
time nix build .#nixosConfigurations.<host>.config.system.build.toplevel --print-build-logs

Capture nix config show snapshot as config-before.txt.

Acceptance: Three timed runs with cold/warm labels.

Lab 2 — Tune jobs/cores

Pick one hypothesis, e.g. “too many parallel link jobs thrash RAM.”

Apply via flake:

nix.settings.max-jobs = 2;
nix.settings.cores = 4;

Switch config (or set user nix.conf on non-NixOS carefully), restart daemon if needed, re-time the same target as Lab 1.

Document winner settings for this machine only.

Lab 3 — Cache hit awareness

# After a toplevel build, approximate download vs build by reading logs (-L)
# Count "copying path" vs "building '" lines from a log snippet

Optional: temporarily misconfigure a wrong extra substituter (unreachable) and observe timeout cost—then remove it. Note CI implications.

Lab 4 — Eval hygiene checklist

In PERF.md, audit your flake for:

  • IFD sites (import of derivation results)
  • Accidental --impure scripts
  • Oversized imports on every host
  • Dev shells pulled into system eval unnecessarily

Fix one real issue if found (or document “none found” with evidence).

Lab 5 — Remote builder decision (write-only if no hardware)

Write BUILDERS.md: when you would add a remote builder vs buy a better cache. Include supportedFeatures needed for nixosTest (kvm).

Lab 6 — Memory pressure drill (lab only)

While a parallel build runs:

# another terminal
free -h
# watch -n2 free -h

If swap thrash appears, lower max-jobs and re-time. Record decision.

Acceptance (Elitebook): Documented settings that do not OOM a 16 GB host during a typical host toplevel build.

Common gotchas

Symptom What to do
Settings in nix.conf ignored Multi-user: daemon config; restart daemon
Faster jobs, slower wall clock RAM thrash / disk queue; lower max-jobs
Eval dominates always Module/IFD issue, not cores
CI OOM on matrix Cap jobs per runner
auto max-jobs surprises Pin explicit numbers for servers
Timing after switch includes downtime Use build for fair comparison
Auto-GC mid-ISO Raise free space; schedule GC windows

Checkpoint

  • Before/after timings in PERF.md
  • Final max-jobs / cores recorded with rationale
  • Cache vs build awareness notes
  • One eval-hygiene audit completed
  • No sandbox disabled “for speed”
  • Three personal gotchas

Commit

git add .
git commit -m "lab(nixos): rebuild performance measurement and tuning"

Write three personal gotchas before continuing.

Next

→ Content-addressed derivations (awareness) — IA vs CA without migrating Capstone.