Troubleshooting drills

Updated

July 30, 2026

Troubleshooting drills

Goal: Inject 2–3 realistic failures on a disposable lab VM, recover them, and write durable playbook entries covering boot, build/eval, service, and network classes.

Warning

Do not run injection drills on a daily driver or the only copy of important data. Snapshot the VM first.

Why this chapter exists

Theory without broken systems is cosplay. Capstone DR and rollback assume you already have a personal playbook. This chapter builds it under time pressure.

Theory 1 — Failure class matrix

Class Symptoms First tools
Eval nixos-rebuild dies before build; option errors; infinite recursion nix eval, bisect modules, evaluator
Build Compile/hash/sandbox failures nix log, store and daemon
Boot Unbootable, emergency shell, wrong generation Console, bootloader generations, stage-1 logs
Activation Switch fails mid-script nixos-rebuild output, systemd activation logs
Service Unit failed, restart loop systemctl, journalctl
Network No SSH, DNS, firewall drops ip, nft/iptables, ss, console
Secrets Service can’t read age/sops path Permissions, sops keys, activation ordering
Disk/state Full disk, wrong mount, impermanence surprise df, mount, disko layout

For each incident, write:

Symptom → Class → Hypothesis → Command evidence → Fix → Prevention

Theory 2 — Golden command palette

# system health
systemctl --failed
journalctl -p err -b --no-pager | tail -100
journalctl -u <unit> -e --no-pager

# nix
nix log /nix/store/….drv
nix-store --verify --check-contents   # heavy; rare
nixos-rebuild dry-activate --flake .#host

# network
ip a; ip r
ss -lntp
ping -c2 1.1.1.1
resolvectl status || cat /etc/resolv.conf
nft list ruleset | head   # if nft

# boot / generations
ls -la /nix/var/nix/profiles/system*
# previous generation boot from bootloader menu

Remote when SSH dies

  • Hypervisor console
  • deploy-rs / colmena won’t save you if network is wrong—console will

Elitebook lab specifics

Resource Note
Console virt-manager console / virsh console / Incus console
Nested virt Needed for nixosTest, not for basic SSH drills
Free space Disk-full injections need headroom to recover

Theory 3 — Injection catalog (choose carefully)

Pick 2–3 from different classes:

# Injection Recovery skill
1 Break a module option type / typo in flake host Eval literacy
2 Firewall deny SSH (on VM with console!) Network + console
3 Stop/mask a critical unit or bad ExecStart Service debug
4 Wrong proxy upstream Service + HTTP
5 Mis-permission a secret path Secrets activation
6 Fill disk with a large dummy file Disk hygiene
7 Bad fileSystems entry on snapshot VM only Boot recovery

Never inject disk/boot failures without a hypervisor snapshot.

Theory 4 — Time discipline

For each incident, timebox:

Phase Budget
Detect 5 min
Classify 5 min
Fix 20–40 min
Write playbook entry 10 min

If stuck past budget, snapshot rollback and write “what I needed to learn” instead of thrashing.

Theory 5 — Playbook entry format

playbook/PL-XXX.md:

# PL-003 — nginx 502 after deploy

## Symptoms
- curl localhost returns 502
- nginx active, upstream down

## Class
Service / proxy

## Diagnose
systemctl status myapp
journalctl -u myapp -n 100
curl -v upstream

## Fix
…

## Prevention
- nixosTest curl through proxy
- healthcheck in CI

## Related
TLS front door; nixosTest mastery

Theory 6 — Rollback as a diagnostic tool

sudo nixos-rebuild switch --rollback
# or boot previous generation from loader

Rollback answers: “Is this a config regression?” If symptoms vanish, bisect generations / git. If symptoms remain, suspect state/disk/network outside the generation.

Theory 7 — Layered isolation order

When unsure, isolate in this order:

1. Physical/virt console access?
2. Network path (ping/DNS/SSH)?
3. Boot/generation?
4. Eval of flake?
5. Activation/systemd?
6. App/data layer?

Jumping to “rewrite the flake” during a network outage wastes the drill clock.

Worked example — playbook stub

## Symptom
SSH hangs after rebuild

## Class
network / firewall

## Diagnose
ss -tlnp; nft list ruleset; journalctl -u sshd

## Fix
re-add allowedTCPPorts = [ 22 ]; console rebuild

## Prevention
nixos-rebuild test first; console attached

Lab 0 — Snapshot

# hypervisor UI or:
# virsh snapshot-create-as lab troubleshoot-pre --disk-only --atomic
# incus snapshot create nixlab troubleshoot-pre

Confirm console access works before breaking SSH.

mkdir -p ~/lab/nixos/internals/troubleshoot/playbook
cd ~/lab/nixos/internals/troubleshoot

Acceptance gate before inject: snapshot exists; console proven; you know wipe boundaries.

Lab 1 — Baseline health

On healthy lab:

systemctl --failed | tee baseline-failed.txt
curl -fsS http://127.0.0.1/ || true
nixos-rebuild dry-activate --flake .#<host> | tee baseline-dry.txt

Lab 2 — Inject & recover #1 (eval or build)

Example eval injection:

# deliberate typo in host config
networking.hostName = 123; # type error
time nixos-rebuild build --flake .#<host> 2>&1 | tee incident1.log
# fix
# re-build green

Write playbook/PL-001.md with timing notes.

Lab 3 — Inject & recover #2 (service or secrets)

Example:

  • Point systemd ExecStart at wrong binary via your module, or
  • Break a runtime secret path after backup of working config

Recover using journal + unit status. Playbook PL-002.md.

Lab 4 — Inject & recover #3 (network) — console required

Example:

networking.firewall.allowedTCPPorts = [ ]; # removed 22
# or extra iptables drop — prefer declarative so rebuild can fix

From console, restore SSH, verify from client. PL-003.md.

If you cannot safely do network injection, substitute a proxy misconfig incident and document why.

Acceptance: At least two distinct classes recovered; network class only with console.

Lab 5 — Compile master index

playbook/README.md table of all PL entries + class tags. This file is Capstone ops gold.

Lab 6 — Timeboxed chaos #4 (state)

Optional fourth injection: delete a non-critical state file (not /etc/ssh host keys unless snapshot). Recover from backup notes. Emphasize state ≠ generation.

Lab 8 — End-state health

systemctl --failed
curl -fsS http://127.0.0.1/ || true
# leave host better than you found it

Acceptance: Host healthy; snapshot can be deleted only after health confirmed.

Common gotchas

Mistake Fix
Breaking SSH without console Snapshot + console first
Only reading theory Actually inject
Fixing without writing playbook Entry is the deliverable
Multiple simultaneous injections One at a time
Leaving host broken overnight Revert snapshot if needed
Forgetting prevention section Tests/asserts become prevention

Checkpoint

  • VM snapshot taken
  • ≥2 incidents injected and fixed
  • ≥2 playbook entries with diagnose/fix/prevent
  • Master playbook index
  • Host healthy at end of drills
  • Three personal gotchas

Commit

git add .
git commit -m "lab(nixos): troubleshooting drill and ops playbook"

Write three personal gotchas before continuing.

Next

→ Architecture review — freeze map and gaps before Capstone thrash.