Day 76 — nixosTest mastery

Updated

July 30, 2026

Day 76 — nixosTest mastery

Stage VII · ~4h
Goal: Harden a real nixosTest (or flake checks) suite: replace brittle sleep waits, use waitForUnit / port / file helpers correctly, handle multi-node networking, and leave CI-ready tests for Capstone.

Why this day exists

Days 64–65 introduced nixosTest. Capstone A requires at least one test that catches a real regression. Flaky tests train you to ignore CI—worse than no tests. Today is production hygiene for the test layer.

Theory 1 — What nixosTest really runs

pkgs.nixosTest / testers.runNixOSTest (names vary by nixpkgs era) builds QEMU VMs from NixOS configs, wires a Python test driver, and runs scripted interactions.

# sketch — adjust to your nixpkgs helpers
pkgs.nixosTest {
  name = "lab-proxy";
  nodes.machine = { config, pkgs, ... }: {
    imports = [ ../modules/proxy.nix ];
    my.proxy.enable = true;
    # …
  };
  testScript = ''
    machine.start()
    machine.wait_for_unit("multi-user.target")
    machine.wait_for_unit("nginx.service")
    machine.succeed("curl -f http://127.0.0.1/")
  '';
}

Wire into flakes:

checks.x86_64-linux.lab-proxy = self.nixosConfigurations… # or pkgs.nixosTest { … }
# common pattern:
checks = {
  lab-proxy = pkgs.nixosTest (import ./tests/lab-proxy.nix { inherit pkgs; });
};
nix build .#checks.x86_64-linux.lab-proxy -L
nix flake check

Theory 2 — Flake (flaky) sources

Cause Symptom Fix
Fixed sleep 10 Passes on fast hosts, fails on CI Wait on unit/port/log
Race before network DNS/curl fail intermittently wait_for_open_port, wait_until_succeeds
Wall-clock timeouts too tight CI under load fails Budget timeouts; retry patterns in driver
Shared mutable state Order-dependent Independent nodes; pure configs
Missing kvm Slow TCG or fail Ensure nested virt / supportedFeatures
Non-determinism in app Intermittent HTTP 500 Fix app or assert eventually

Theory 3 — Driver API patterns (Python DSL)

Exact methods evolve; concepts stable. Prefer:

machine.start()
machine.wait_for_unit("network-online.target")  # if appropriate
machine.wait_for_unit("myapp.service")
machine.wait_for_open_port(8080)
machine.wait_until_succeeds("curl -f http://127.0.0.1:8080/health")
machine.succeed("systemctl is-active myapp.service")
machine.fail("curl -f http://127.0.0.1:8080/admin")  # expect fail
machine.copy_from_vm("/var/log/myapp.log", "out")
print(machine.succeed("journalctl -u myapp -n 50 --no-pager"))

Anti-pattern

machine.succeed("sleep 30")
machine.succeed("curl …")  # still racy if app slower than 30s under load

sleep as a backoff inside wait_until_succeeds is different from a single blind sleep—prefer the wait helpers.

Multi-node

nodes = {
  server = { … };
  client = { … };
};
server.wait_for_unit("myapp.service")
client.wait_until_succeeds("curl -f http://server/health")

Hostname resolution between nodes is provided by the test network—use node names as hostnames per driver docs.

Theory 4 — Systemd-aware testing

Good services for tests:

  • Type=notify when possible (ready = really ready)
  • Explicit After=/Wants= for network
  • Health endpoint or ExecStartPost readiness

If a unit is active but still binding ports, wait on port not only unit.

machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)

Theory 5 — Secrets and tests

Never require production sops keys for unit tests. Patterns:

Pattern Use
Test-only plaintext in VM OK for non-sensitive lab secrets
sops-nix with age key generated in test Heavier; sometimes worth it
Mock module mkIf config.my.testMode Injects dummy secrets
Skip TLS in test, test TLS in one dedicated check Split concerns

Capstone should have at least one test without human interactive decryption.

Theory 6 — CI integration

# sketch GitHub Actions
- uses: cachix/install-nix-action@…
- run: nix build .#checks.x86_64-linux.lab-proxy -L

Needs:

  • KVM available or accept TCG slowness with longer timeouts
  • Binary cache for nixpkgs to avoid rebuilding QEMU world every time
  • Artifacts: fail logs on failure
nix log .#checks.x86_64-linux.lab-proxy

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day76/tests
cd ~/lab/90daysofx/02-nixos/day76

Prefer extending your real flake checks; if missing, create a minimal test module today.

Lab 1 — Inventory existing tests

nix flake show
# list checks
rg -n "nixosTest|runNixOSTest|testScript" -g '*.nix' /path/to/flake

In SUITE.md, list each check: purpose, flake attribute, known flakes.

Lab 2 — Eliminate sleep-only waits

Find sleep in test scripts:

rg -n "sleep" -g '*test*' /path/to/flake

Replace with wait_for_unit / wait_for_open_port / wait_until_succeeds. Re-run:

nix build .#checks.<system>.<name> -L

Run twice consecutively—flakes that pass once and fail once are not done.

Lab 3 — Add one regression that would have bitten you

Pick a real past bug (wrong firewall port, typo in upstream, module mkIf inverted). Write a test that fails if that bug returns.

Example idea:

# ensure proxy port open, backend not world-open if that's your design
machine.succeed("curl -f http://127.0.0.1/")
machine.fail("curl -f http://127.0.0.1:8080/")  # if backend bound localhost only—adjust to design

Lab 4 — Multi-node or client/server (stretch if time)

If single-node only so far, add a second node that curls the first or document why single-node is enough for Capstone A and add a second assertion class instead (journal + HTTP).

Lab 5 — Failure UX

Break the test on purpose (wrong unit name). Confirm the log shows journal excerpts. Add machine.succeed("journalctl -u …") on failure paths if helpful. Record debugging time in SUITE.md.

Theory 7 — Machine objects and multi-node patterns

# conceptual multi-node
start_all()
server.wait_for_unit("nginx.service")
client.wait_until_succeeds("ping -c1 server")
client.succeed("curl -sf http://server | grep -q ok")

Node names become hostnames in the test network. Use them deliberately; avoid hard-coded IPs unless the test network docs say so.

Theory 8 — Snapshotting vs rebuild cost

Long tests that rebuild the world each run train you to skip them. Techniques:

Technique Effect
Smaller module under test Faster eval/build
Shared fixture attrs Less duplication
Separate checks.slow-* CI nightly vs PR

Mastery is green and runnable.

Worked example — systemd journal assertion

machine.wait_for_unit("lab-tick.service")
machine.succeed("journalctl -u lab-tick.service -n 20 | grep -q tick")

Lab 6 — Flake check wiring audit

nix flake show
nix build .#checks.$(nix eval --raw --impure --expr 'builtins.currentSystem') -L 2>/dev/null || \
  nix flake check -L

List every check attr and its purpose in one table.

Lab 7 — Failure screenshot discipline

When a test fails, save:

  1. Full command
  2. Last 50 log lines
  3. Whether rerun without change flakes

Flaky → fix waits; deterministic → fix system under test.

Common gotchas

Symptom What to do
error: unable to start guest KVM permissions, nested virt off
Hangs at wait_for_unit Unit failed; print systemctl status / journal
Works interactively, fails CI Timeouts; resource; sandbox/kvm
TLS/ACME tests fail offline Don’t use real ACME in VM; use self-signed or http-only mode
Eval of test is huge Share modules carefully; don’t import desktop profiles
Flake check runs all checks Tag slow tests; nix build specific attr in iteration

Checkpoint

  • Suite inventory in SUITE.md
  • No critical path depends on blind sleep alone
  • Test run twice green
  • At least one regression-oriented assertion
  • Notes on secrets strategy in tests
  • Three personal gotchas

Commit

git add .
git commit -m "day76: harden nixosTest suite for capstone"

Write three personal gotchas before continuing.

Tomorrow

Day 77 — Ecosystem map: try one of devenv, Snowfall, or nix-darwin; compare to your hand-rolled layout—don’t collect them all.