Day 74 — Performance

Day 74 — Performance

Stage VII · ~4h
Goal: Measure where time goes (eval vs download vs build), tune max-jobs / cores / remote builders / caches, and record a before/after rebuild timing with settings you can justify.

Why this day exists

NixOS rebuilds feel “slow” for different reasons. Cranking --max-jobs 128 on a laptop with 8GB RAM and a spinning disk can thrash harder than serial builds. Capstone CI budgets and lab iteration speed both require measurement first.

---

Theory 1 — The rebuild time budget

A flake host rebuild roughly pays:

eval (modules + nixpkgs)  →  realize (substitute or build)  →  activate

Phase	Dominated by	Knobs
Eval	CPU single-thread-ish, memory, module count, IFD	eval-cache, avoid IFD, slim modules
Substitute	Network, cache hit rate, compression	binary caches, closer mirrors
Build	CPU cores, jobs, IO, remote builders	max-jobs, cores, builders
Activate	Service restarts, switch strategy	switch vs boot vs test

# rough wall clock
/usr/bin/time -v nixos-rebuild build --flake .#<host>
# or
time nix build .#nixosConfigurations.<host>.config.system.build.toplevel

Separate cold (empty download cache / first eval) vs warm timings or you will lie to yourself.

---

Theory 2 — Jobs vs cores

Definitions

Setting	Meaning
max-jobs	How many derivations realized in parallel
cores	How many cores advertised to each derivation (NIX_BUILD_CORES / enableParallelBuilding)

nix.settings = {
  max-jobs = 4;    # or "auto"
  cores = 2;       # per-job parallelism
};

CLI overrides:

nix build -j 4 --cores 2 .#…
nixos-rebuild build --flake .#host --max-jobs 4 --cores 2

Choosing numbers (rules of thumb)

Hardware	Starting point
4c/8t laptop, 16GB	max-jobs = 2–4, cores = 2
16c desktop, 64GB	max-jobs = 8, cores = 2–4
Many small builds	higher max-jobs, lower cores
Few huge compiles (gcc/llvm)	lower max-jobs, higher cores
Tiny rootfs / low RAM	lower both; watch OOM

Constraint: max-jobs * cores should not massively exceed physical cores + RAM headroom, especially with linking-heavy packages.

Remote builders

nix.distributedBuilds = true;
nix.buildMachines = [ {
  hostName = "builder";
  system = "x86_64-linux";
  maxJobs = 8;
  speedFactor = 2;
  supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
  mandatoryFeatures = [ ];
} ];

Offload large builds; keep eval local unless you know your setup.

---

Theory 3 — Eval performance

Eval cache

Modern Nix can cache evaluation results for flakes when inputs unchanged.

nix config show | grep -i eval
# features / options evolve; check your version docs for eval-cache

Practical wins:

• Stable flake.lock → better cache reuse
  
• Avoid unnecessary --impure
  
• Avoid Import From Derivation (IFD) when possible

IFD cost

# expensive pattern: build something at eval time
let pkg = import (pkgs.fetchFromGitHub { … }) { };
in pkg.something

IFD forces realization during eval—serial and painful in CI. Prefer flake inputs and generated files committed or produced in build phase.

Module count

Every module imported has a cost. Shared modules (Day 61) should be focused, not a megamodule imported everywhere with giant mkIf trees if you can split roles.

Memory

Large evals can exceed default stack/heap. Symptoms: evaluator OOM, thrashing. Fix structure before only raising limits.

---

Theory 4 — Download & store IO

Lever	Effect
Hit cache.nixos.org + project cache	Avoid local compiles
http2 / modern curl stack	Throughput (usually already fine)
Keep disk free	Avoid GC mid-build
min-free / max-free auto-GC	Prevent disk full mid-CI
SSD vs HDD	Link farms and unpack dominate on HDD

nix.settings = {
  min-free = 1073741824;   # 1 GiB
  max-free = 5368709120;   # 5 GiB
};

---

Theory 5 — What not to do

Anti-pattern	Why
sandbox = false for speed	Trust/purity regression
Disable binary caches “to force builds” in daily driver	Wastes hours
Infinite max-jobs on laptop	OOM + thermal throttle
Micro-optimizing before cache hits	Measure substitute rate first
Blindly copying someone else’s nix.conf	Hardware differs

---

Worked example — Measurement protocol

# 1) Note config
nix config show | egrep 'max-jobs|cores|substituters' | tee before-config.txt

# 2) Warm caches optionally with a cheap build
nix build nixpkgs#hello

# 3) Wall-clock system build (no switch)
/usr/bin/time -p nix build \
  .#nixosConfigurations.lab.config.system.build.toplevel \
  2>&1 | tee time-before.txt

# 4) Change nix.settings via configuration.nix / flake; rebuild once to apply daemon settings

# 5) Repeat timing → time-after.txt

Daemon-level settings change may need:

sudo systemctl restart nix-daemon

(On NixOS after a config switch that changes nix.settings.)

---

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day74
cd ~/lab/90daysofx/02-nixos/day74

Use your Capstone/lab flake host.

---

Lab 1 — Baseline timings

Record in PERF.md:

Run	Command	Wall seconds	Notes (cold/warm)
A	eval hostname only
B	nix build toplevel
C	nixos-rebuild build

time nix eval .#nixosConfigurations.<host>.config.networking.hostName
time nix build .#nixosConfigurations.<host>.config.system.build.toplevel --print-build-logs

Capture nix config show snapshot as config-before.txt.

---

Lab 2 — Tune jobs/cores

Pick one hypothesis, e.g. “too many parallel link jobs thrash RAM.”

Apply via flake:

nix.settings.max-jobs = 2;
nix.settings.cores = 4;

Switch config (or set user nix.conf on non-NixOS carefully), restart daemon if needed, re-time the same target as Lab 1.

Document winner settings for this machine only.

---

Lab 3 — Cache hit awareness

# After a toplevel build, approximate download vs build by reading logs (-L)
# Count "copying path" vs "building '" lines from a log snippet

Optional: temporarily misconfigure a wrong extra substituter (unreachable) and observe timeout cost—then remove it. Note CI implications.

---

Lab 4 — Eval hygiene checklist

In PERF.md, audit your flake for:

• IFD sites (import of derivation results)
  
• Accidental --impure scripts
  
• Oversized imports on every host
  
• Dev shells pulled into system eval unnecessarily

Fix one real issue if found (or document “none found” with evidence).

---

Lab 5 — Remote builder decision (write-only if no hardware)

Write BUILDERS.md: when you would add a remote builder vs buy a better cache. Include supportedFeatures needed for nixosTest (kvm).

---

Common gotchas

Symptom	What to do
Settings in nix.conf ignored	Multi-user: daemon config; restart daemon
Faster jobs, slower wall clock	RAM thrash / disk queue; lower max-jobs
Eval dominates always	Module/IFD issue, not cores
CI OOM on matrix	Cap jobs per runner
auto max-jobs surprises	Pin explicit numbers for servers
Timing after switch includes downtime	Use build for fair comparison

---

Checkpoint

• Before/after timings in PERF.md
  
• Final max-jobs / cores recorded with rationale
  
• Cache vs build awareness notes
  
• One eval-hygiene audit completed
  
• No sandbox disabled “for speed”
  
• Three personal gotchas

Commit

git add .
git commit -m "day74: nix rebuild performance measurement and tuning"

Write three personal gotchas before continuing.

Tomorrow

Day 75 — CA derivations: experimental content-addressed derivations—awareness, status, optional tiny experiment.