Day 25 — Performance, CA Derivations & Advanced Testing

Updated

July 30, 2025

Day 25 — Performance, CA Derivations & Advanced Testing

Day 74 — Performance

Stage VII · ~4h
Goal: Measure where time goes (eval vs download vs build), tune max-jobs / cores / remote builders / caches, and record a before/after rebuild timing with settings you can justify.

Why this day exists

NixOS rebuilds feel “slow” for different reasons. Cranking --max-jobs 128 on a laptop with 8GB RAM and a spinning disk can thrash harder than serial builds. Capstone CI budgets and lab iteration speed both require measurement first.

Theory 1 — The rebuild time budget

A flake host rebuild roughly pays:

eval (modules + nixpkgs)  →  realize (substitute or build)  →  activate
Phase Dominated by Knobs
Eval CPU single-thread-ish, memory, module count, IFD eval-cache, avoid IFD, slim modules
Substitute Network, cache hit rate, compression binary caches, closer mirrors
Build CPU cores, jobs, IO, remote builders max-jobs, cores, builders
Activate Service restarts, switch strategy switch vs boot vs test 
# rough wall clock
/usr/bin/time -v nixos-rebuild build --flake .#<host>
# or
time nix build .#nixosConfigurations.<host>.config.system.build.toplevel

Separate cold (empty download cache / first eval) vs warm timings or you will lie to yourself.

Theory 2 — Jobs vs cores

Definitions

Setting Meaning
max-jobs How many derivations realized in parallel
cores How many cores advertised to each derivation (NIX_BUILD_CORES / enableParallelBuilding) 
nix.settings = {
  max-jobs = 4;    # or "auto"
  cores = 2;       # per-job parallelism
};

CLI overrides:

nix build -j 4 --cores 2 .#…
nixos-rebuild build --flake .#host --max-jobs 4 --cores 2

Choosing numbers (rules of thumb)

Hardware Starting point
4c/8t laptop, 16GB max-jobs = 2–4, cores = 2
16c desktop, 64GB max-jobs = 8, cores = 2–4
Many small builds higher max-jobs, lower cores
Few huge compiles (gcc/llvm) lower max-jobs, higher cores
Tiny rootfs / low RAM lower both; watch OOM

Constraint: max-jobs * cores should not massively exceed physical cores + RAM headroom, especially with linking-heavy packages.

Remote builders

nix.distributedBuilds = true;
nix.buildMachines = [ {
  hostName = "builder";
  system = "x86_64-linux";
  maxJobs = 8;
  speedFactor = 2;
  supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
  mandatoryFeatures = [ ];
} ];

Offload large builds; keep eval local unless you know your setup.

Theory 3 — Eval performance

Eval cache

Modern Nix can cache evaluation results for flakes when inputs unchanged.

nix config show | grep -i eval
# features / options evolve; check your version docs for eval-cache

Practical wins:

  • Stable flake.lock → better cache reuse
  • Avoid unnecessary --impure
  • Avoid Import From Derivation (IFD) when possible

IFD cost

# expensive pattern: build something at eval time
let pkg = import (pkgs.fetchFromGitHub { … }) { };
in pkg.something

IFD forces realization during eval—serial and painful in CI. Prefer flake inputs and generated files committed or produced in build phase.

Module count

Every module imported has a cost. Shared modules (Day 61) should be focused, not a megamodule imported everywhere with giant mkIf trees if you can split roles.

Memory

Large evals can exceed default stack/heap. Symptoms: evaluator OOM, thrashing. Fix structure before only raising limits.

Theory 4 — Download & store IO

Lever Effect
Hit cache.nixos.org + project cache Avoid local compiles
http2 / modern curl stack Throughput (usually already fine)
Keep disk free Avoid GC mid-build
min-free / max-free auto-GC Prevent disk full mid-CI
SSD vs HDD Link farms and unpack dominate on HDD 
nix.settings = {
  min-free = 1073741824;   # 1 GiB
  max-free = 5368709120;   # 5 GiB
};

Theory 5 — What not to do

Anti-pattern Why
sandbox = false for speed Trust/purity regression
Disable binary caches “to force builds” in daily driver Wastes hours
Infinite max-jobs on laptop OOM + thermal throttle
Micro-optimizing before cache hits Measure substitute rate first
Blindly copying someone else’s nix.conf Hardware differs

Worked example — Measurement protocol

# 1) Note config
nix config show | egrep 'max-jobs|cores|substituters' | tee before-config.txt

# 2) Warm caches optionally with a cheap build
nix build nixpkgs#hello

# 3) Wall-clock system build (no switch)
/usr/bin/time -p nix build \
  .#nixosConfigurations.lab.config.system.build.toplevel \
  2>&1 | tee time-before.txt

# 4) Change nix.settings via configuration.nix / flake; rebuild once to apply daemon settings

# 5) Repeat timing → time-after.txt

Daemon-level settings change may need:

sudo systemctl restart nix-daemon

(On NixOS after a config switch that changes nix.settings.)

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day74
cd ~/lab/90daysofx/02-nixos/day74

Use your Capstone/lab flake host.

Lab 1 — Baseline timings

Record in PERF.md:

Run Command Wall seconds Notes (cold/warm)
A eval hostname only
B nix build toplevel
C nixos-rebuild build 
time nix eval .#nixosConfigurations.<host>.config.networking.hostName
time nix build .#nixosConfigurations.<host>.config.system.build.toplevel --print-build-logs

Capture nix config show snapshot as config-before.txt.

Lab 2 — Tune jobs/cores

Pick one hypothesis, e.g. “too many parallel link jobs thrash RAM.”

Apply via flake:

nix.settings.max-jobs = 2;
nix.settings.cores = 4;

Switch config (or set user nix.conf on non-NixOS carefully), restart daemon if needed, re-time the same target as Lab 1.

Document winner settings for this machine only.

Lab 3 — Cache hit awareness

# After a toplevel build, approximate download vs build by reading logs (-L)
# Count "copying path" vs "building '" lines from a log snippet

Optional: temporarily misconfigure a wrong extra substituter (unreachable) and observe timeout cost—then remove it. Note CI implications.

Lab 4 — Eval hygiene checklist

In PERF.md, audit your flake for:

  • IFD sites (import of derivation results)
  • Accidental --impure scripts
  • Oversized imports on every host
  • Dev shells pulled into system eval unnecessarily

Fix one real issue if found (or document “none found” with evidence).

Lab 5 — Remote builder decision (write-only if no hardware)

Write BUILDERS.md: when you would add a remote builder vs buy a better cache. Include supportedFeatures needed for nixosTest (kvm).

Common gotchas

Symptom What to do
Settings in nix.conf ignored Multi-user: daemon config; restart daemon
Faster jobs, slower wall clock RAM thrash / disk queue; lower max-jobs
Eval dominates always Module/IFD issue, not cores
CI OOM on matrix Cap jobs per runner
auto max-jobs surprises Pin explicit numbers for servers
Timing after switch includes downtime Use build for fair comparison

Checkpoint

  • Before/after timings in PERF.md
  • Final max-jobs / cores recorded with rationale
  • Cache vs build awareness notes
  • One eval-hygiene audit completed
  • No sandbox disabled “for speed”
  • Three personal gotchas

Commit

git add .
git commit -m "day74: nix rebuild performance measurement and tuning"

Write three personal gotchas before continuing.

Tomorrow

Day 75 — CA derivations: experimental content-addressed derivations—awareness, status, optional tiny experiment.

Day 75 — CA derivations awareness

Stage VII · ~4h
Goal: Explain input-addressed vs content-addressed derivations, know the experimental status and feature flags, run a tiny optional experiment or produce a documented status memo with primary-source links—without putting production Capstone (or 26.05 lab hosts) on CA.

Important

Content-addressed (CA) derivations remain an experimental path. This day is literacy and a sandbox experiment—not a mandate to migrate your host. Capstone A stays on default store semantics.

Why this day exists

Default Nix paths are input-addressed: the store hash fingerprints the build plan (derivations + inputs), not only the output bytes. Two builds that produce identical bytes can still get different paths if input metadata differs. CA aims at output-hash identity—powerful for early cutoff / better sharing—but the feature surface, UX, and nixpkgs readiness evolve. Experts need an honest mental model before blog hype.

You will hear CA in RFCs, release notes, and conference talks. Literacy prevents cargo-cult experimental-features on prod builders.

Theory 1 — Input-addressed world (today’s default)

inputs + build recipe  →  hash  →  /nix/store/<hash>-name

Consequences you already rely on:

Property Implication
Rebuild with same inputs Same path (reproducible address)
Different compiler flags Different path even if binary identical
FODs Special: output hash fixed; may network
Substitutes Trust signatures of builders/caches

Trust is “did we run this recipe (or accept a signed substitute for it)?”

FODs vs CA (do not conflate)

Fixed-output derivation CA derivations (experimental)
Classic use fetches, vendoring general builds
Hash of declared output content-addressed store model
Maturity Everyday Experimental

FODs already teach “hash the result” for a class of builds. CA generalizes the idea; it is not “FODs for everything tomorrow.”

Theory 2 — Content-addressed idea

build  →  output bytes  →  hash  →  /nix/store/<hash>-name

Aspirations:

Goal Intuition
Early cutoff If output already exists with same content hash, skip dependent rebuilds more aggressively
Better dedup Same bytes converge to same path across “recipes” that don’t matter
FODs generalized More builds behave like “hash the result”

Challenges:

  • Self-references in outputs (files embedding their own store path) need rewriting schemes
  • Granularity (per-output CA, etc.)
  • Cache/protocol/nixpkgs compatibility
  • Debugging mental model shift
  • Tooling and UX still moving

Theory 3 — Experimental feature surface

Exact names and defaults change across Nix versions. Always check:

nix --version
nix config show | grep -i experimental
man nix.conf | less   # search ca-derivations / content-address

Historically you will see flags in the family of:

  • ca-derivations
  • related: dynamic-derivations, cgroups, etc. (orthogonal but often discussed together)

Enable only on a lab profile:

# ephemeral lab — not Capstone default
nix.settings.experimental-features = [
  "nix-command"
  "flakes"
  "ca-derivations"  # if supported by your Nix
];

Or local:

# illustrative — confirm flag name for your version
nix build --experimental-features 'nix-command flakes ca-derivations' ...

Reading status (primary sources)

Prefer in this order:

  1. Nix release notes for your exact version
  2. Official Nix manual sections on store / experimental features
  3. nixpkgs issues/PRs tagged CA (for package readiness)
  4. RFCs / discourse RFCs (design intent)

Blogs are optional commentary after primary sources. Version-skew blogs are how people enable the wrong flag.

Theory 4 — What CA does not magically fix

Claim Reality
“CA = perfect reproducibility” Reproducibility is about determinism of builds; CA is about addressing
“CA replaces sandbox” Isolation still required
“Turn on CA, everything faster tomorrow” Needs package + toolchain readiness; may slow or break
“Signatures no longer matter” You still decide trust for substitutes
“Capstone should adopt CA” No for this volume’s Capstone A

Theory 5 — Relationship to Capstone and 26.05

Capstone A (Days 83–88) should stay on stable, default derivation mode unless you explicitly document a lab-only branch. DR wipe/rebuild proofs should not depend on experimental store semantics.

Your nixos-26.05 pin is about module and package stability. CA is a Nix daemon / store feature layer—orthogonal but coupled in operational risk if enabled fleet-wide.

Optional: track CA as a 26.11 roadmap item (Day 90)—revisit when your Nix pin upgrades and primary sources look greener.

Theory 6 — Trust and substitutes under CA thinking

Even with CA aspirations, operations still answer:

Question Still true?
Who is allowed to write to my store? Yes
Which substituters do I trust? Yes
Do I verify signatures? Yes (policy)
Can a malicious builder hurt me? Yes

CA is not a substitute for Day 53 cache trust discipline or Day 54 builder trust.

Worked example — Conceptual comparison card

Dimension Input-addressed Content-addressed (experimental)
Path hash from Recipe/inputs Output content (design)
Same bytes, different flags Different paths May converge
Ecosystem maturity Default everywhere Experimental / partial
Debugging familiarity High (this book) Lower; fewer blog runbooks
Prod recommendation (this volume) Yes Not yet for Capstone

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day75
cd ~/lab/90daysofx/02-nixos/day75

Lab 1 — Version & feature inventory (required)

nix --version | tee nix-version.txt
nix config show > nix-config.txt
grep -iE 'experimental|ca-|content' nix-config.txt || true

Open official docs for your version; save URLs in SOURCES.md (at least 2 primary links).

Write a 10–15 line status summary: what CA means, whether your Nix binary advertises the feature, and any warnings in the manual.

Lab 2 — Choose a track

Track A — Optional tiny experiment (if feature available)

  1. Enable ca-derivations (or current equivalent) only in a disposable VM or user nix.conf overlay you can revert.
  2. Build a trivial derivation twice with intentional non-semantic input noise if docs show how CA modes are requested for derivations (API differs by version—follow current manual).
  3. Compare store paths and nix path-info output.
  4. Capture logs in experiment/.
  5. Disable experimental CA for daily lab work if it complicates Capstone.

If the manual’s API is unclear or broken on your pin, stop and switch to Track B—do not thrash for hours.

Track B — Status memo without enabling (always acceptable)

Write CA-MEMO.md (≥1 page) covering:

  1. Input- vs content-addressed definitions
  2. Problems CA wants to solve
  3. Current experimental flags for your Nix version
  4. Known limitations (self-refs, ecosystem)
  5. Decision: not enabling on Capstone host, with revisit date (e.g. after 26.11 notes)
  6. Primary source links
  7. Difference from FODs in your own words

Lab 3 — Teach-back diagram

In NOTES.md, draw (ASCII is fine):

[user config] → eval → derivations → (IA path | CA path?) → substitute/build → store

Annotate where trust/signatures apply.

Lab 4 — Nixpkgs awareness scan

Search your local nixpkgs (or GitHub) for mentions:

# if you have nixpkgs checkout
rg -n "contentAddressed|__contentAddressed|ca-derivations" -g '*.nix' | head

Note whether anything in your critical path packages advertises CA—likely little for a lab host. Record findings.

Lab 5 — Explicit non-adoption

Add to Capstone docs (or day75 notes):

## Store semantics
- Mode: input-addressed (default)
- CA: deferred until <date/criteria>
- Owner: <you>

This prevents a well-meaning future PR from flipping experimental flags mid-freeze.

Lab 6 — Analogy card (teach others)

Write three sentences max for a teammate:

  1. Input-addressed: “The address is the recipe’s fingerprint.”
  2. Content-addressed: “The address wants to be the loaf’s fingerprint.”
  3. Why Capstone waits: “Experimental + ecosystem partial; DR should not depend on it.”

If you cannot explain without jargon, revise CA-MEMO.md until you can.

Interaction with other Stage VII days

Day Link to CA literacy
71 evaluator Eval still produces plans; addressing is store-side
73 store/daemon Feature flags live with daemon policy
74 performance Early cutoff is the performance story—measure before enabling
79 release notes Watch Nix + nixpkgs notes for CA maturity signals

Do not enable CA “for speed” without a benchmark and a rollback plan for the builder.

Common gotchas

Symptom / mistake What to do
Blog for Nix 2.X vs your 2.Y Trust nix --version + matching manual
Enabled CA on shared prod builder Revert; isolate experiments
Assuming path equality ⇒ bit equality always Still verify; trust model differs
Spending the whole day debugging experimental UX Fall back to Track B memo
Mixing CA talk with FOD confusion FODs already hash outputs for fetch class
Capstone depends on CA Redesign; default store only

Checkpoint

  • nix-version.txt + feature inventory
  • SOURCES.md with primary links
  • Track A experiment notes or Track B CA-MEMO.md
  • Explicit Capstone decision: stay input-addressed
  • Three personal gotchas

Commit

git add .
git commit -m "day75: CA derivations experimental awareness"

Write three personal gotchas before continuing.

Tomorrow

Day 76 — nixosTest mastery: harden flaky VM tests, replace sleep-only waits, multi-node interaction patterns.

Day 76 — nixosTest mastery

Stage VII · ~4h
Goal: Harden a real nixosTest (or flake checks) suite: replace brittle sleep waits, use waitForUnit / port / file helpers correctly, handle multi-node networking, and leave CI-ready tests for Capstone.

Why this day exists

Days 64–65 introduced nixosTest. Capstone A requires at least one test that catches a real regression. Flaky tests train you to ignore CI—worse than no tests. Today is production hygiene for the test layer.

Theory 1 — What nixosTest really runs

pkgs.nixosTest / testers.runNixOSTest (names vary by nixpkgs era) builds QEMU VMs from NixOS configs, wires a Python test driver, and runs scripted interactions.

# sketch — adjust to your nixpkgs helpers
pkgs.nixosTest {
  name = "lab-proxy";
  nodes.machine = { config, pkgs, ... }: {
    imports = [ ../modules/proxy.nix ];
    my.proxy.enable = true;
    # …
  };
  testScript = ''
    machine.start()
    machine.wait_for_unit("multi-user.target")
    machine.wait_for_unit("nginx.service")
    machine.succeed("curl -f http://127.0.0.1/")
  '';
}

Wire into flakes:

checks.x86_64-linux.lab-proxy = self.nixosConfigurations… # or pkgs.nixosTest { … }
# common pattern:
checks = {
  lab-proxy = pkgs.nixosTest (import ./tests/lab-proxy.nix { inherit pkgs; });
};
nix build .#checks.x86_64-linux.lab-proxy -L
nix flake check

Theory 2 — Flake (flaky) sources

Cause Symptom Fix
Fixed sleep 10 Passes on fast hosts, fails on CI Wait on unit/port/log
Race before network DNS/curl fail intermittently wait_for_open_port, wait_until_succeeds
Wall-clock timeouts too tight CI under load fails Budget timeouts; retry patterns in driver
Shared mutable state Order-dependent Independent nodes; pure configs
Missing kvm Slow TCG or fail Ensure nested virt / supportedFeatures
Non-determinism in app Intermittent HTTP 500 Fix app or assert eventually

Theory 3 — Driver API patterns (Python DSL)

Exact methods evolve; concepts stable. Prefer:

machine.start()
machine.wait_for_unit("network-online.target")  # if appropriate
machine.wait_for_unit("myapp.service")
machine.wait_for_open_port(8080)
machine.wait_until_succeeds("curl -f http://127.0.0.1:8080/health")
machine.succeed("systemctl is-active myapp.service")
machine.fail("curl -f http://127.0.0.1:8080/admin")  # expect fail
machine.copy_from_vm("/var/log/myapp.log", "out")
print(machine.succeed("journalctl -u myapp -n 50 --no-pager"))

Anti-pattern

machine.succeed("sleep 30")
machine.succeed("curl …")  # still racy if app slower than 30s under load

sleep as a backoff inside wait_until_succeeds is different from a single blind sleep—prefer the wait helpers.

Multi-node

nodes = {
  server = { … };
  client = { … };
};
server.wait_for_unit("myapp.service")
client.wait_until_succeeds("curl -f http://server/health")

Hostname resolution between nodes is provided by the test network—use node names as hostnames per driver docs.

Theory 4 — Systemd-aware testing

Good services for tests:

  • Type=notify when possible (ready = really ready)
  • Explicit After=/Wants= for network
  • Health endpoint or ExecStartPost readiness

If a unit is active but still binding ports, wait on port not only unit.

machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)

Theory 5 — Secrets and tests

Never require production sops keys for unit tests. Patterns:

Pattern Use
Test-only plaintext in VM OK for non-sensitive lab secrets
sops-nix with age key generated in test Heavier; sometimes worth it
Mock module mkIf config.my.testMode Injects dummy secrets
Skip TLS in test, test TLS in one dedicated check Split concerns

Capstone should have at least one test without human interactive decryption.

Theory 6 — CI integration

# sketch GitHub Actions
- uses: cachix/install-nix-action@…
- run: nix build .#checks.x86_64-linux.lab-proxy -L

Needs:

  • KVM available or accept TCG slowness with longer timeouts
  • Binary cache for nixpkgs to avoid rebuilding QEMU world every time
  • Artifacts: fail logs on failure
nix log .#checks.x86_64-linux.lab-proxy

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day76/tests
cd ~/lab/90daysofx/02-nixos/day76

Prefer extending your real flake checks; if missing, create a minimal test module today.

Lab 1 — Inventory existing tests

nix flake show
# list checks
rg -n "nixosTest|runNixOSTest|testScript" -g '*.nix' /path/to/flake

In SUITE.md, list each check: purpose, flake attribute, known flakes.

Lab 2 — Eliminate sleep-only waits

Find sleep in test scripts:

rg -n "sleep" -g '*test*' /path/to/flake

Replace with wait_for_unit / wait_for_open_port / wait_until_succeeds. Re-run:

nix build .#checks.<system>.<name> -L

Run twice consecutively—flakes that pass once and fail once are not done.

Lab 3 — Add one regression that would have bitten you

Pick a real past bug (wrong firewall port, typo in upstream, module mkIf inverted). Write a test that fails if that bug returns.

Example idea:

# ensure proxy port open, backend not world-open if that's your design
machine.succeed("curl -f http://127.0.0.1/")
machine.fail("curl -f http://127.0.0.1:8080/")  # if backend bound localhost only—adjust to design

Lab 4 — Multi-node or client/server (stretch if time)

If single-node only so far, add a second node that curls the first or document why single-node is enough for Capstone A and add a second assertion class instead (journal + HTTP).

Lab 5 — Failure UX

Break the test on purpose (wrong unit name). Confirm the log shows journal excerpts. Add machine.succeed("journalctl -u …") on failure paths if helpful. Record debugging time in SUITE.md.

Theory 7 — Machine objects and multi-node patterns

# conceptual multi-node
start_all()
server.wait_for_unit("nginx.service")
client.wait_until_succeeds("ping -c1 server")
client.succeed("curl -sf http://server | grep -q ok")

Node names become hostnames in the test network. Use them deliberately; avoid hard-coded IPs unless the test network docs say so.

Theory 8 — Snapshotting vs rebuild cost

Long tests that rebuild the world each run train you to skip them. Techniques:

Technique Effect
Smaller module under test Faster eval/build
Shared fixture attrs Less duplication
Separate checks.slow-* CI nightly vs PR

Mastery is green and runnable.

Worked example — systemd journal assertion

machine.wait_for_unit("lab-tick.service")
machine.succeed("journalctl -u lab-tick.service -n 20 | grep -q tick")

Lab 6 — Flake check wiring audit

nix flake show
nix build .#checks.$(nix eval --raw --impure --expr 'builtins.currentSystem') -L 2>/dev/null || \
  nix flake check -L

List every check attr and its purpose in one table.

Lab 7 — Failure screenshot discipline

When a test fails, save:

  1. Full command
  2. Last 50 log lines
  3. Whether rerun without change flakes

Flaky → fix waits; deterministic → fix system under test.

Common gotchas

Symptom What to do
error: unable to start guest KVM permissions, nested virt off
Hangs at wait_for_unit Unit failed; print systemctl status / journal
Works interactively, fails CI Timeouts; resource; sandbox/kvm
TLS/ACME tests fail offline Don’t use real ACME in VM; use self-signed or http-only mode
Eval of test is huge Share modules carefully; don’t import desktop profiles
Flake check runs all checks Tag slow tests; nix build specific attr in iteration

Checkpoint

  • Suite inventory in SUITE.md
  • No critical path depends on blind sleep alone
  • Test run twice green
  • At least one regression-oriented assertion
  • Notes on secrets strategy in tests
  • Three personal gotchas

Commit

git add .
git commit -m "day76: harden nixosTest suite for capstone"

Write three personal gotchas before continuing.

Tomorrow

Day 77 — Ecosystem map: try one of devenv, Snowfall, or nix-darwin; compare to your hand-rolled layout—don’t collect them all.