Day 81 — Troubleshooting drill

Updated

July 30, 2026

Day 81 — Troubleshooting drill

Stage VII · ~4h
Goal: Inject 2–3 realistic failures on a disposable lab VM, recover them, and write durable playbook entries covering boot, build/eval, service, and network classes.

Warning

Do not run injection drills on a daily driver or the only copy of important data. Snapshot the VM first.

Why this day exists

Theory without broken systems is cosplay. Capstone DR (Day 88) and rollback (Day 89) assume you already have a personal playbook. Today builds it under time pressure.

Theory 1 — Failure class matrix

Class Symptoms First tools
Eval nixos-rebuild dies before build; option errors; infinite recursion nix eval, bisect modules, Day 71–72
Build Compile/hash/sandbox failures nix log, sandbox notes (Day 73)
Boot Unbootable, emergency shell, wrong generation Console, bootloader generations, stage-1 logs
Activation Switch fails mid-script nixos-rebuild output, systemd activation logs
Service Unit failed, restart loop systemctl, journalctl
Network No SSH, DNS, firewall drops ip, nft/iptables, ss, console
Secrets Service can’t read age/sops path Permissions, sops keys, activation ordering
Disk/state Full disk, wrong mount, impermanence surprise df, mount, disko layout

For each incident, write:

Symptom → Class → Hypothesis → Command evidence → Fix → Prevention

Theory 2 — Golden command palette

# system health
systemctl --failed
journalctl -p err -b --no-pager | tail -100
journalctl -u <unit> -e --no-pager

# nix
nix log /nix/store/….drv
nix-store --verify --check-contents   # heavy; rare
nixos-rebuild dry-activate --flake .#host

# network
ip a; ip r
ss -lntp
ping -c2 1.1.1.1
resolvectl status || cat /etc/resolv.conf
nft list ruleset | head   # if nft

# boot / generations
ls -la /nix/var/nix/profiles/system*
# previous generation boot from bootloader menu

Remote when SSH dies

  • Hypervisor console
  • deploy-rs / colmena won’t save you if network is wrong—console will

Theory 3 — Injection catalog (choose carefully)

Pick 2–3 from different classes:

# Injection Recovery skill
1 Break a module option type / typo in flake host Eval literacy
2 Firewall deny SSH (on VM with console!) Network + console
3 Stop/mask a critical unit or bad ExecStart Service debug
4 Wrong proxy upstream Service + HTTP
5 Mis-permission a secret path Secrets activation
6 Fill disk with a large dummy file Disk hygiene
7 Bad fileSystems entry on snapshot VM only Boot recovery

Never inject disk/boot failures without a hypervisor snapshot.

Theory 4 — Time discipline

For each incident, timebox:

Phase Budget
Detect 5 min
Classify 5 min
Fix 20–40 min
Write playbook entry 10 min

If stuck past budget, snapshot rollback and write “what I needed to learn” instead of thrashing.

Theory 5 — Playbook entry format

playbook/PL-XXX.md:

# PL-003 — nginx 502 after deploy

## Symptoms
- curl localhost returns 502
- nginx active, upstream down

## Class
Service / proxy

## Diagnose
systemctl status myapp
journalctl -u myapp -n 100
curl -v upstream

## Fix
…

## Prevention
- nixosTest curl through proxy
- healthcheck in CI

## Related
Day 38 proxy, Day 76 tests

Lab 0 — Snapshot

# hypervisor UI or:
# virsh snapshot-create-as lab day81-pre --disk-only --atomic

Confirm console access works before breaking SSH.

mkdir -p ~/lab/90daysofx/02-nixos/day81/playbook
cd ~/lab/90daysofx/02-nixos/day81

Lab 1 — Baseline health

On healthy lab:

systemctl --failed | tee baseline-failed.txt
curl -fsS http://127.0.0.1/ || true
nixos-rebuild dry-activate --flake .#<host> | tee baseline-dry.txt

Lab 2 — Inject & recover #1 (eval or build)

Example eval injection:

# deliberate typo in host config
networking.hostName = 123; # type error
time nixos-rebuild build --flake .#<host> 2>&1 | tee incident1.log
# fix
# re-build green

Write playbook/PL-001.md with timing notes.

Lab 3 — Inject & recover #2 (service or secrets)

Example:

  • Point systemd ExecStart at wrong binary via your module, or
  • Break a runtime secret path after backup of working config

Recover using journal + unit status. Playbook PL-002.md.

Lab 4 — Inject & recover #3 (network) — console required

Example:

networking.firewall.allowedTCPPorts = [ ]; # removed 22
# or extra iptables drop — prefer declarative so rebuild can fix

From console, restore SSH, verify from client. PL-003.md.

If you cannot safely do network injection, substitute a proxy misconfig incident and document why.

Lab 5 — Compile master index

playbook/README.md table of all PL entries + class tags. This file is Capstone ops gold.

Theory 6 — Rollback as a diagnostic tool

sudo nixos-rebuild switch --rollback
# or boot previous generation from loader

Rollback answers: “Is this a config regression?” If symptoms vanish, bisect generations / git. If symptoms remain, suspect state/disk/network outside the generation.

Theory 7 — Layered isolation order

When unsure, isolate in this order:

1. Physical/virt console access?
2. Network path (ping/DNS/SSH)?
3. Boot/generation?
4. Eval of flake?
5. Activation/systemd?
6. App/data layer?

Jumping to “rewrite the flake” during a network outage wastes the drill clock.

Worked example — playbook stub

## Symptom
SSH hangs after rebuild

## Class
network / firewall

## Diagnose
ss -tlnp; nft list ruleset; journalctl -u sshd

## Fix
re-add allowedTCPPorts = [ 22 ]; console rebuild

## Prevention
nixos-rebuild test first; console attached

Lab 6 — Timeboxed chaos #4 (state)

Optional fourth injection: delete a non-critical state file (not /etc/ssh host keys unless snapshot). Recover from backup notes. Emphasize state ≠ generation.

Common gotchas

Mistake Fix
Breaking SSH without console Snapshot + console first
Only reading theory Actually inject
Fixing without writing playbook Entry is the deliverable
Multiple simultaneous injections One at a time
Leaving host broken overnight Revert snapshot if needed

Checkpoint

  • VM snapshot taken
  • ≥2 incidents injected and fixed
  • ≥2 playbook entries with diagnose/fix/prevent
  • Master playbook index
  • Host healthy at end of day
  • Three personal gotchas

Commit

git add .
git commit -m "day81: troubleshooting drill and ops playbook"

Write three personal gotchas before continuing.

Tomorrow

Day 82 — Architecture review: map Stages I–VI onto your repo; prioritized gap list before Capstone implementation days.