Day 65 — More tests

Day 65 — More tests

Stage VI · ~4h
Goal: Grow a suite of at least three meaningful flake checks (including nixosTest and lighter eval/build checks) that mirror CI shape without becoming a flaky tar pit.

Note

Builds on Day 64 (nixosTest). Baseline: flakes, 26.05 pins, modules you already own. Diversify kinds of checks; do not only clone one VM test three times.

Why this day exists

One demo test rots. A small suite with clear ownership becomes the merge gate for roles and packages. Today you diversify test layers, kill flakes early, and time the suite so Day 66 CI has a budget.

---

Theory 1 — Test pyramid for NixOS fleets

Layer	Examples	Speed	Catches
Eval	nix flake show, option asserts, nix eval	Fast	Typos, missing options
Build	nix build .#pkg, host toplevel	Medium	Compile/package breaks
VM (nixosTest)	Service behavior	Slow	Integration
Deploy dry	deploy-rs checks, colmena eval	Medium	Hive wiring
Live canary	Real host probes	Manual/slow	Reality

Aim: many fast, few slow. Capstone and CI die when every PR boots three full VMs for a typo in a comment.

        /live canary\
       /  nixosTest   \
      / build / toplevel \
     / eval / lint / fmt  \

---

Theory 2 — Three checks minimum (template)

# flake outputs fragment — illustrative
checks = forAllSystems (system:
  let
    pkgs = import nixpkgs {
      inherit system;
      overlays = [ self.overlays.default ];
    };
  in {
    # 1) package as check (build is the test)
    cooltool-build = self.packages.${system}.cooltool;

    # 2) pure command check
    cooltool-runs = pkgs.runCommand "cooltool-runs" {
      nativeBuildInputs = [ self.packages.${system}.cooltool ];
    } ''
      cooltool --help >/dev/null
      touch $out
    '';
  } // nixpkgs.lib.optionalAttrs (system == "x86_64-linux") {
    # 3+) VM tests — Linux only
    ssh-hardening = import ./tests/ssh-hardening.nix { inherit pkgs self; };
    web-role = import ./tests/web-role.nix { inherit pkgs self; };
  }
);

nix build .#checks.x86_64-linux.cooltool-runs -L
nix build .#checks.x86_64-linux.ssh-hardening -L
nix flake check -L

---

Theory 3 — Lightweight non-VM checks

Format / lint

# Prefer your Stage III treefmt/nixfmt wiring if present
fmt = pkgs.runCommand "fmt-check" {
  nativeBuildInputs = [ pkgs.nixfmt-rfc-style ];
} ''
  # example only — real checks usually run treefmt --ci on src
  touch $out
'';

Eval / build host configuration

edge-toplevel =
  self.nixosConfigurations.edge.config.system.build.toplevel;

nix build .#nixosConfigurations.edge.config.system.build.toplevel -L

Strong, but heavy—cache it; don’t duplicate five slight variants.

Structured eval assertions

nix eval .#nixosConfigurations.edge.config.networking.hostName
nix eval --json .#nixosConfigurations.edge.config.services.openssh.enable

Wrap in a runCommand with jq if you want it as a formal check:

host-name-ok = pkgs.runCommand "host-name-ok" {
  nativeBuildInputs = [ pkgs.jq ];
} ''
  echo ${pkgs.lib.escapeShellArg (
    builtins.toJSON self.nixosConfigurations.edge.config.networking.hostName
  )} | jq -e '. == "edge"'
  touch $out
'';

(Prefer cleaner patterns as your flake matures; the point is fast feedback.)

---

Theory 4 — Anti-flake tactics

Problem	Fix
Sleep-based waits	wait_for_unit / wait_for_open_port
Order-dependent multi-test	Isolated nodes per test file
Network to internet in tests	Forbid; vendor fixtures
Shared mutable state	Each test pure
Over-broad flake check on Darwin	optionalAttrs for Linux VM tests
30-minute suite	Split “CI fast” vs “nightly full”
Secret decryption in CI	Dummy modules / test keys only

# nixosTest script — good waits
machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)
machine.succeed("curl -f http://127.0.0.1/")

# Run one check without the world
nix build .#checks.x86_64-linux.ssh-hardening -L

---

Theory 5 — Mapping tests to modules

Module / concern	Check ideas
profiles.hardened	sshd -T probes, package absences
roles.web	curl client→server via proxy
Stage V package	help/version smoke
Disko	eval fileSystems keys (not full wipe)
sops-nix	assert secret options configured; decrypt with test key only
firewall	multi-node: client cannot hit closed port
deploy hive	eval nodes exist / --dry-activate if available

Every check should name an owner module in a README table so rotting tests get a human.

---

Theory 6 — Suite layout and CI shape

tests/
  ssh-hardening.nix
  web-role.nix
  cooltool.nix
  fixtures/
    test-secrets.nix
  README.md          # table: check → layer → minutes → owner

# flake fragment
checks.x86_64-linux = {
  inherit (self.packages.x86_64-linux) cooltool;
  cooltool-runs = /* … */;
  ssh-hardening = /* … */;
  web-role = /* … */;
  edge-toplevel =
    self.nixosConfigurations.edge.config.system.build.toplevel;
};

Day 66 will run these on GitHub Actions. Design attrs you can select:

nix build -L \
  .#checks.x86_64-linux.cooltool-runs \
  .#checks.x86_64-linux.ssh-hardening

---

Worked example — fail-inject once

# 1) break intentionally
# e.g. change expected hostname assertion or stop nginx in test config
nix build .#checks.x86_64-linux.web-role -L   # expect fail
# 2) restore
# 3) confirm green
# 4) confirm other checks still independent
nix build .#checks.x86_64-linux.cooltool-runs -L

If breaking test A always breaks unrelated B, your suite shares too much impure state.

---

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day65

Step 1 — Inventory today’s checks

mkdir -p ~/lab/90daysofx/02-nixos/day65
cd /path/to/your/flake
nix flake show 2>&1 | tee ~/lab/90daysofx/02-nixos/day65/flake-show.txt

List current checks and gaps.

Step 2 — Add check #1 (package)

Build + run your Stage V package (runCommand or package attr as check).

Step 3 — Add check #2 (nixosTest)

Keep/improve Day 64 test; ensure waits are unit/port based.

Step 4 — Add check #3 (config)

Either second nixosTest or toplevel build for one host or eval assertion.

Step 5 — Fail inject matrix

Break each check once; confirm isolation; restore green.

Step 6 — Timing budget

/usr/bin/time -p nix flake check -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day65/timing.txt

If over your CI budget, document split: checks.ci vs full suite.

Step 7 — README table

Check	Layer	Minutes	Owner module

Step 8 — Stability

Rerun thrice (or twice if time-boxed); note flakiness tickets.

Step 9 — Secrets policy for tests

Write one paragraph: how CI avoids prod sops keys (fixtures, dummy tokens).

---

Common gotchas

Symptom / mistake	What to do
flake check builds everything forever	Select attrs; fix IFD explosions
Darwin CI without KVM	Gate VM tests on Linux
Secret decryption in checks	Dummy modules / mock
Duplicate heavy toplevels	Share less; accept one + cache
Tests not in merge gate	Day 66 wires CI
Sleep-only “green”	Replace with readiness waits
Three identical VM tests	Diversify layers

---

Checkpoint

• ≥3 meaningful checks green
  
• At least one VM test and one non-VM test
  
• Timing measured
  
• README/table of suite with owners
  
• Fail-inject done once per check
  
• Flakiness notes empty or ticketed
  
• Three personal gotchas

Commit

git add .
git commit -m "day65: expanded flake check suite"

Write three personal gotchas before continuing.

Tomorrow

Day 66 — CI pipeline: GitHub Actions + Nix + cache write.