nixosTest mastery

Updated

July 30, 2026

nixosTest mastery

Goal: Harden a real nixosTest (or flake checks) suite: replace brittle sleep waits, use waitForUnit / port / file helpers correctly, handle multi-node networking, and leave CI-ready tests for Capstone.

Why this chapter exists

Ops chapters introduced nixosTest. Capstone A requires at least one test that catches a real regression. Flaky tests train you to ignore CI—worse than no tests. This chapter is production hygiene for the test layer.

Theory 1 — What nixosTest really runs

pkgs.nixosTest / testers.runNixOSTest (names vary by nixpkgs era) builds QEMU VMs from NixOS configs, wires a Python test driver, and runs scripted interactions.

# sketch — adjust to your nixpkgs helpers
pkgs.nixosTest {
  name = "lab-proxy";
  nodes.machine = { config, pkgs, ... }: {
    imports = [ ../modules/proxy.nix ];
    my.proxy.enable = true;
    # …
  };
  testScript = ''
    machine.start()
    machine.wait_for_unit("multi-user.target")
    machine.wait_for_unit("nginx.service")
    machine.succeed("curl -f http://127.0.0.1/")
  '';
}

Wire into flakes:

checks = {
  lab-proxy = pkgs.nixosTest (import ./tests/lab-proxy.nix { inherit pkgs; });
};

nix build .#checks.x86_64-linux.lab-proxy -L
nix flake check

Note

nixpkgs has migrated helpers (pkgs.testers.runNixOSTest, lib.nixos.runTest, classic nixosTest). Use whatever your 26.05 pin documents; the shape (nodes + testScript) remains the skill.

Theory 2 — Flake (flaky) sources

Cause	Symptom	Fix
Fixed `sleep 10`	Passes on fast hosts, fails on CI	Wait on unit/port/log
Race before network	DNS/curl fail intermittently	`wait_for_open_port`, `wait_until_succeeds`
Wall-clock timeouts too tight	CI under load fails	Budget timeouts; retry patterns in driver
Shared mutable state	Order-dependent	Independent nodes; pure configs
Missing `kvm`	Slow TCG or fail	Ensure nested virt / `supportedFeatures`
Non-determinism in app	Intermittent HTTP 500	Fix app or assert eventually

Theory 3 — Driver API patterns (Python DSL)

Exact methods evolve; concepts stable. Prefer:

machine.start()
machine.wait_for_unit("network-online.target")  # if appropriate
machine.wait_for_unit("myapp.service")
machine.wait_for_open_port(8080)
machine.wait_until_succeeds("curl -f http://127.0.0.1:8080/health")
machine.succeed("systemctl is-active myapp.service")
machine.fail("curl -f http://127.0.0.1:8080/admin")  # expect fail
machine.copy_from_vm("/var/log/myapp.log", "out")
print(machine.succeed("journalctl -u myapp -n 50 --no-pager"))

Anti-pattern

machine.succeed("sleep 30")
machine.succeed("curl …")  # still racy if app slower than 30s under load

sleep as a backoff inside wait_until_succeeds is different from a single blind sleep—prefer the wait helpers.

Multi-node

nodes = {
  server = { … };
  client = { … };
};

server.wait_for_unit("myapp.service")
client.wait_until_succeeds("curl -f http://server/health")

Hostname resolution between nodes is provided by the test network—use node names as hostnames per driver docs.

Theory 4 — Systemd-aware testing

Good services for tests:

Type=notify when possible (ready = really ready)
Explicit After=/Wants= for network
Health endpoint or ExecStartPost readiness

If a unit is active but still binding ports, wait on port not only unit.

machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)

Theory 5 — Secrets and tests

Never require production sops keys for unit tests. Patterns:

Pattern	Use
Test-only plaintext in VM	OK for non-sensitive lab secrets
`sops-nix` with age key generated in test	Heavier; sometimes worth it
Mock module `mkIf config.my.testMode`	Injects dummy secrets
Skip TLS in test, test TLS in one dedicated check	Split concerns

Capstone should have at least one test without human interactive decryption.

Theory 6 — CI integration

# sketch GitHub Actions
- uses: cachix/install-nix-action@…
- run: nix build .#checks.x86_64-linux.lab-proxy -L

Needs:

KVM available or accept TCG slowness with longer timeouts
Binary cache for nixpkgs to avoid rebuilding QEMU world every time
Artifacts: fail logs on failure

nix log .#checks.x86_64-linux.lab-proxy

Elitebook / KVM lab acceptance

Check	Command / note
KVM usable	`/dev/kvm` exists; user in `kvm` group if required
Nested virt if host is VM	BIOS + hypervisor CPU flags
Test finishes	`nix build .#checks… -L` exits 0 twice
Disk headroom	Leave space for VM images under store

ls -l /dev/kvm || echo "no kvm"
# groups | tr ' ' '\n' | grep -E 'kvm|libvirt' || true

Theory 7 — Machine objects and multi-node patterns

# conceptual multi-node
start_all()
server.wait_for_unit("nginx.service")
client.wait_until_succeeds("ping -c1 server")
client.succeed("curl -sf http://server | grep -q ok")

Node names become hostnames in the test network. Use them deliberately; avoid hard-coded IPs unless the test network docs say so.

Theory 8 — Snapshotting vs rebuild cost

Long tests that rebuild the world each run train you to skip them. Techniques:

Technique	Effect
Smaller module under test	Faster eval/build
Shared fixture attrs	Less duplication
Separate `checks.slow-*`	CI nightly vs PR

Mastery is green and runnable.

Worked example — systemd journal assertion

machine.wait_for_unit("lab-tick.service")
machine.succeed("journalctl -u lab-tick.service -n 20 | grep -q tick")

Worked example — regression that fails when firewall wrong

machine.wait_for_open_port(443)
# backend should not be world-open if design is proxy-only
machine.fail("curl -f --max-time 2 http://127.0.0.1:8080/")

Adjust ports to your design; the skill is asserting policy, not only liveness.

Lab 0 — Workspace

mkdir -p ~/lab/nixos/internals/tests
cd ~/lab/nixos/internals/tests

Prefer extending your real flake checks; if missing, create a minimal test module now.

Lab 1 — Inventory existing tests

nix flake show
# list checks
rg -n "nixosTest|runNixOSTest|testScript" -g '*.nix' /path/to/flake

In SUITE.md, list each check: purpose, flake attribute, known flakes.

Lab 2 — Eliminate sleep-only waits

Find sleep in test scripts:

rg -n "sleep" -g '*test*' /path/to/flake

Replace with wait_for_unit / wait_for_open_port / wait_until_succeeds. Re-run:

nix build .#checks.<system>.<name> -L

Run twice consecutively—tests that pass once and fail once are not done.

Acceptance: Critical path has no blind sleep as sole wait; two consecutive greens.

Lab 3 — Add one regression that would have bitten you

Pick a real past bug (wrong firewall port, typo in upstream, module mkIf inverted). Write a test that fails if that bug returns.

Lab 4 — Multi-node or client/server (stretch)

If single-node only so far, add a second node that curls the first or document why single-node is enough for Capstone A and add a second assertion class instead (journal + HTTP).

Lab 5 — Failure UX

Break the test on purpose (wrong unit name). Confirm the log shows journal excerpts. Add machine.succeed("journalctl -u …") on failure paths if helpful. Record debugging time in SUITE.md.

Lab 6 — Flake check wiring audit

nix flake show
nix build .#checks.$(nix eval --raw --impure --expr 'builtins.currentSystem') -L 2>/dev/null || \
  nix flake check -L

List every check attr and its purpose in one table.

Lab 7 — Failure screenshot discipline

When a test fails, save:

Full command
Last 50 log lines
Whether rerun without change flakes

Flaky → fix waits; deterministic → fix system under test.

Common gotchas

Symptom	What to do
`error: unable to start guest`	KVM permissions, nested virt off
Hangs at `wait_for_unit`	Unit failed; print `systemctl status` / journal
Works interactively, fails CI	Timeouts; resource; sandbox/kvm
TLS/ACME tests fail offline	Don’t use real ACME in VM; use self-signed or http-only mode
Eval of test is huge	Share modules carefully; don’t import desktop profiles
Flake check runs all checks	Tag slow tests; `nix build` specific attr in iteration
Secrets block CI	Dummy secrets / testMode

Checkpoint

Suite inventory in SUITE.md
No critical path depends on blind sleep alone
Test run twice green
At least one regression-oriented assertion
Notes on secrets strategy in tests
KVM/lab acceptance noted
Three personal gotchas

Commit

git add .
git commit -m "lab(nixos): harden nixosTest suite for capstone"

Write three personal gotchas before continuing.

→ Ecosystem map — try one adjacent tool; keep/drop without Capstone rewrite.

nixosTest mastery

Why this chapter exists

Theory 1 — What nixosTest really runs

Theory 2 — Flake (flaky) sources

Theory 3 — Driver API patterns (Python DSL)

Anti-pattern

Multi-node

Theory 4 — Systemd-aware testing

Theory 5 — Secrets and tests

Theory 6 — CI integration

Elitebook / KVM lab acceptance

Theory 7 — Machine objects and multi-node patterns

Theory 8 — Snapshotting vs rebuild cost

Worked example — systemd journal assertion

Worked example — regression that fails when firewall wrong

Lab 0 — Workspace

Lab 1 — Inventory existing tests

Lab 2 — Eliminate sleep-only waits

Lab 3 — Add one regression that would have bitten you

Lab 4 — Multi-node or client/server (stretch)

Lab 5 — Failure UX

Lab 6 — Flake check wiring audit

Lab 7 — Failure screenshot discipline

Common gotchas

Checkpoint

Commit

Next