More testing patterns

More testing patterns

Goal: Grow a suite of at least three meaningful flake checks (including nixosTest and lighter eval/build checks) that mirror CI shape without becoming a flaky tar pit.

Note

Builds on the nixosTest chapter (nixosTest). Baseline: flakes, 26.05 pins, modules you already own. Diversify kinds of checks; do not only clone one VM test three times.

Why this day exists

One demo test rots. A small suite with clear ownership becomes the merge gate for roles and packages. Today you diversify test layers, kill flakes early, and time the suite so the CI pipelines chapter CI has a budget.

---

Theory 1 — Test pyramid for NixOS fleets

Layer	Examples	Speed	Catches
Eval	nix flake show, option asserts, nix eval	Fast	Typos, missing options
Build	nix build .#pkg, host toplevel	Medium	Compile/package breaks
VM (nixosTest)	Service behavior	Slow	Integration
Deploy dry	deploy-rs checks, colmena eval	Medium	Hive wiring
Live canary	Real host probes	Manual/slow	Reality

Aim: many fast, few slow. Capstone and CI die when every PR boots three full VMs for a typo in a comment.

        /live canary\
       /  nixosTest   \
      / build / toplevel \
     / eval / lint / fmt  \

---

Theory 2 — Three checks minimum (template)

# flake outputs fragment — illustrative
checks = forAllSystems (system:
  let
    pkgs = import nixpkgs {
      inherit system;
      overlays = [ self.overlays.default ];
    };
  in {
    # 1) package as check (build is the test)
    cooltool-build = self.packages.${system}.cooltool;

    # 2) pure command check
    cooltool-runs = pkgs.runCommand "cooltool-runs" {
      nativeBuildInputs = [ self.packages.${system}.cooltool ];
    } ''
      cooltool --help >/dev/null
      touch $out
    '';
  } // nixpkgs.lib.optionalAttrs (system == "x86_64-linux") {
    # 3+) VM tests — Linux only
    ssh-hardening = import ./tests/ssh-hardening.nix { inherit pkgs self; };
    web-role = import ./tests/web-role.nix { inherit pkgs self; };
  }
);

nix build .#checks.x86_64-linux.cooltool-runs -L
nix build .#checks.x86_64-linux.ssh-hardening -L
nix flake check -L

---

Theory 3 — Lightweight non-VM checks

Format / lint

# Prefer your Stage III treefmt/nixfmt wiring if present
fmt = pkgs.runCommand "fmt-check" {
  nativeBuildInputs = [ pkgs.nixfmt-rfc-style ];
} ''
  # example only — real checks usually run treefmt --ci on src
  touch $out
'';

Eval / build host configuration

edge-toplevel =
  self.nixosConfigurations.edge.config.system.build.toplevel;

nix build .#nixosConfigurations.edge.config.system.build.toplevel -L

Strong, but heavy—cache it; don’t duplicate five slight variants.

Structured eval assertions

nix eval .#nixosConfigurations.edge.config.networking.hostName
nix eval --json .#nixosConfigurations.edge.config.services.openssh.enable

Wrap in a runCommand with jq if you want it as a formal check:

host-name-ok = pkgs.runCommand "host-name-ok" {
  nativeBuildInputs = [ pkgs.jq ];
} ''
  echo ${pkgs.lib.escapeShellArg (
    builtins.toJSON self.nixosConfigurations.edge.config.networking.hostName
  )} | jq -e '. == "edge"'
  touch $out
'';

(Prefer cleaner patterns as your flake matures; the point is fast feedback.)

---

Theory 4 — Anti-flake tactics

Problem	Fix
Sleep-based waits	wait_for_unit / wait_for_open_port
Order-dependent multi-test	Isolated nodes per test file
Network to internet in tests	Forbid; vendor fixtures
Shared mutable state	Each test pure
Over-broad flake check on Darwin	optionalAttrs for Linux VM tests
30-minute suite	Split “CI fast” vs “nightly full”
Secret decryption in CI	Dummy modules / test keys only

# nixosTest script — good waits
machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)
machine.succeed("curl -f http://127.0.0.1/")

# Run one check without the world
nix build .#checks.x86_64-linux.ssh-hardening -L

---

Theory 5 — Mapping tests to modules

Module / concern	Check ideas
profiles.hardened	sshd -T probes, package absences
roles.web	curl client→server via proxy
Stage V package	help/version smoke
Disko	eval fileSystems keys (not full wipe)
sops-nix	assert secret options configured; decrypt with test key only
firewall	multi-node: client cannot hit closed port
deploy hive	eval nodes exist / --dry-activate if available

Every check should name an owner module in a README table so rotting tests get a human.

---

Theory 6 — Suite layout and CI shape

tests/
  ssh-hardening.nix
  web-role.nix
  cooltool.nix
  fixtures/
    test-secrets.nix
  README.md          # table: check → layer → minutes → owner

# flake fragment
checks.x86_64-linux = {
  inherit (self.packages.x86_64-linux) cooltool;
  cooltool-runs = /* … */;
  ssh-hardening = /* … */;
  web-role = /* … */;
  edge-toplevel =
    self.nixosConfigurations.edge.config.system.build.toplevel;
};

the CI pipelines chapter will run these on GitHub Actions. Design attrs you can select:

nix build -L \
  .#checks.x86_64-linux.cooltool-runs \
  .#checks.x86_64-linux.ssh-hardening

---

Worked example — fail-inject once

# 1) break intentionally
# e.g. change expected hostname assertion or stop nginx in test config
nix build .#checks.x86_64-linux.web-role -L   # expect fail
# 2) restore
# 3) confirm green
# 4) confirm other checks still independent
nix build .#checks.x86_64-linux.cooltool-runs -L

If breaking test A always breaks unrelated B, your suite shares too much impure state.

---

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day65

Step 1 — Inventory today’s checks

mkdir -p ~/lab/90daysofx/02-nixos/day65
cd /path/to/your/flake
nix flake show 2>&1 | tee ~/lab/90daysofx/02-nixos/day65/flake-show.txt

List current checks and gaps.

Step 2 — Add check #1 (package)

Build + run your Stage V package (runCommand or package attr as check).

Step 3 — Add check #2 (nixosTest)

Keep/improve the nixosTest chapter test; ensure waits are unit/port based.

Step 4 — Add check #3 (config)

Either second nixosTest or toplevel build for one host or eval assertion.

Step 5 — Fail inject matrix

Break each check once; confirm isolation; restore green.

Step 6 — Timing budget

/usr/bin/time -p nix flake check -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day65/timing.txt

If over your CI budget, document split: checks.ci vs full suite.

Step 7 — README table

Check	Layer	Minutes	Owner module

Step 8 — Stability

Rerun thrice (or twice if time-boxed); note flakiness tickets.

Step 9 — Secrets policy for tests

Write one paragraph: how CI avoids prod sops keys (fixtures, dummy tokens).

---

Common gotchas

Symptom / mistake	What to do
flake check builds everything forever	Select attrs; fix IFD explosions
Darwin CI without KVM	Gate VM tests on Linux
Secret decryption in checks	Dummy modules / mock
Duplicate heavy toplevels	Share less; accept one + cache
Tests not in merge gate	the CI pipelines chapter wires CI
Sleep-only “green”	Replace with readiness waits
Three identical VM tests	Diversify layers

---

Checkpoint

• ≥3 meaningful checks green
  
• At least one VM test and one non-VM test
  
• Timing measured
  
• README/table of suite with owners
  
• Fail-inject done once per check
  
• Flakiness notes empty or ticketed
  
• Three personal gotchas

Commit

git add .
git commit -m "day65: expanded flake check suite"

Write three personal gotchas before continuing.