Day 27 — Troubleshooting & Architecture Review

Updated

July 30, 2025

Day 27 — Troubleshooting & Architecture Review

Day 81 — Troubleshooting drill

Stage VII · ~4h
Goal: Inject 2–3 realistic failures on a disposable lab VM, recover them, and write durable playbook entries covering boot, build/eval, service, and network classes.

Warning

Do not run injection drills on a daily driver or the only copy of important data. Snapshot the VM first.

Why this day exists

Theory without broken systems is cosplay. Capstone DR (Day 88) and rollback (Day 89) assume you already have a personal playbook. Today builds it under time pressure.

Theory 1 — Failure class matrix

Class	Symptoms	First tools
Eval	`nixos-rebuild` dies before build; option errors; infinite recursion	`nix eval`, bisect modules, Day 71–72
Build	Compile/hash/sandbox failures	`nix log`, sandbox notes (Day 73)
Boot	Unbootable, emergency shell, wrong generation	Console, bootloader generations, `stage-1` logs
Activation	Switch fails mid-script	`nixos-rebuild` output, systemd activation logs
Service	Unit failed, restart loop	`systemctl`, `journalctl`
Network	No SSH, DNS, firewall drops	`ip`, `nft`/`iptables`, `ss`, console
Secrets	Service can’t read age/sops path	Permissions, sops keys, activation ordering
Disk/state	Full disk, wrong mount, impermanence surprise	`df`, `mount`, disko layout

For each incident, write:

Symptom → Class → Hypothesis → Command evidence → Fix → Prevention

Theory 2 — Golden command palette

# system health
systemctl --failed
journalctl -p err -b --no-pager | tail -100
journalctl -u <unit> -e --no-pager

# nix
nix log /nix/store/….drv
nix-store --verify --check-contents   # heavy; rare
nixos-rebuild dry-activate --flake .#host

# network
ip a; ip r
ss -lntp
ping -c2 1.1.1.1
resolvectl status || cat /etc/resolv.conf
nft list ruleset | head   # if nft

# boot / generations
ls -la /nix/var/nix/profiles/system*
# previous generation boot from bootloader menu

Remote when SSH dies

Hypervisor console
deploy-rs / colmena won’t save you if network is wrong—console will

Theory 3 — Injection catalog (choose carefully)

Pick 2–3 from different classes:

#	Injection	Recovery skill
1	Break a module option type / typo in flake host	Eval literacy
2	Firewall deny SSH (on VM with console!)	Network + console
3	Stop/mask a critical unit or bad `ExecStart`	Service debug
4	Wrong proxy upstream	Service + HTTP
5	Mis-permission a secret path	Secrets activation
6	Fill disk with a large dummy file	Disk hygiene
7	Bad `fileSystems` entry on snapshot VM only	Boot recovery

Never inject disk/boot failures without a hypervisor snapshot.

Theory 4 — Time discipline

For each incident, timebox:

Phase	Budget
Detect	5 min
Classify	5 min
Fix	20–40 min
Write playbook entry	10 min

If stuck past budget, snapshot rollback and write “what I needed to learn” instead of thrashing.

Theory 5 — Playbook entry format

playbook/PL-XXX.md:

# PL-003 — nginx 502 after deploy

## Symptoms
- curl localhost returns 502
- nginx active, upstream down

## Class
Service / proxy

## Diagnose
systemctl status myapp
journalctl -u myapp -n 100
curl -v upstream

## Fix
…

## Prevention
- nixosTest curl through proxy
- healthcheck in CI

## Related
Day 38 proxy, Day 76 tests

Lab 0 — Snapshot

# hypervisor UI or:
# virsh snapshot-create-as lab day81-pre --disk-only --atomic

Confirm console access works before breaking SSH.

mkdir -p ~/lab/90daysofx/02-nixos/day81/playbook
cd ~/lab/90daysofx/02-nixos/day81

Lab 1 — Baseline health

On healthy lab:

systemctl --failed | tee baseline-failed.txt
curl -fsS http://127.0.0.1/ || true
nixos-rebuild dry-activate --flake .#<host> | tee baseline-dry.txt

Lab 2 — Inject & recover #1 (eval or build)

Example eval injection:

# deliberate typo in host config
networking.hostName = 123; # type error

time nixos-rebuild build --flake .#<host> 2>&1 | tee incident1.log
# fix
# re-build green

Write playbook/PL-001.md with timing notes.

Lab 3 — Inject & recover #2 (service or secrets)

Example:

Point systemd ExecStart at wrong binary via your module, or
Break a runtime secret path after backup of working config

Recover using journal + unit status. Playbook PL-002.md.

Lab 4 — Inject & recover #3 (network) — console required

Example:

networking.firewall.allowedTCPPorts = [ ]; # removed 22
# or extra iptables drop — prefer declarative so rebuild can fix

From console, restore SSH, verify from client. PL-003.md.

If you cannot safely do network injection, substitute a proxy misconfig incident and document why.

Lab 5 — Compile master index

playbook/README.md table of all PL entries + class tags. This file is Capstone ops gold.

Theory 6 — Rollback as a diagnostic tool

sudo nixos-rebuild switch --rollback
# or boot previous generation from loader

Rollback answers: “Is this a config regression?” If symptoms vanish, bisect generations / git. If symptoms remain, suspect state/disk/network outside the generation.

Theory 7 — Layered isolation order

When unsure, isolate in this order:

1. Physical/virt console access?
2. Network path (ping/DNS/SSH)?
3. Boot/generation?
4. Eval of flake?
5. Activation/systemd?
6. App/data layer?

Jumping to “rewrite the flake” during a network outage wastes the drill clock.

Worked example — playbook stub

## Symptom
SSH hangs after rebuild

## Class
network / firewall

## Diagnose
ss -tlnp; nft list ruleset; journalctl -u sshd

## Fix
re-add allowedTCPPorts = [ 22 ]; console rebuild

## Prevention
nixos-rebuild test first; console attached

Lab 6 — Timeboxed chaos #4 (state)

Optional fourth injection: delete a non-critical state file (not /etc/ssh host keys unless snapshot). Recover from backup notes. Emphasize state ≠ generation.

Lab 7 — Master index cross-links

Ensure docs/playbooks/index.md links to Day 13 rollback, Day 37 firewall, Day 34 sops, Day 51 build debug. Troubleshooting is a graph, not a single page.

Common gotchas

Mistake	Fix
Breaking SSH without console	Snapshot + console first
Only reading theory	Actually inject
Fixing without writing playbook	Entry is the deliverable
Multiple simultaneous injections	One at a time
Leaving host broken overnight	Revert snapshot if needed

Checkpoint

Commit

git add .
git commit -m "day81: troubleshooting drill and ops playbook"

Write three personal gotchas before continuing.

Tomorrow

Day 82 — Architecture review: map Stages I–VI onto your repo; prioritized gap list before Capstone implementation days.

Day 82 — Architecture review

Stage VII · ~4h
Goal: Map Stages I–VI (and early VII) onto your actual repository, produce ARCHITECTURE.md, and a prioritized gap list that drives Capstone Days 83–87—before writing more features ad hoc. Assume Capstone hosts pin NixOS 26.05.

Tip

Stage VIII is implementation and proof, not discovery. Teams that skip architecture review thrash during capstone: secrets half-done, tests theater, deploy only from a laptop. Today freezes an honest map of what exists vs Capstone A requirements.

Why this day exists

You have ~80 days of skills. Capstone fails when those skills never land in one coherent flake. Architecture review is the integration gate before code thrash: map, gaps, sequence—then implement.

Theory 1 — Capstone A bar (recall)

Required:

≥1 NixOS host role understood end-to-end
Home Manager or minimal service-user story
Secrets: sops-nix or agenix
Reverse-proxy / edge pattern
Deploy path: deploy-rs and/or Colmena
≥1 nixosTest or flake check catching real regressions
CI builds and preferably pushes a cache

Days 88–89 add wipe/rebuild + rollback proof. Day 82 gaps should make 83–87 trivial to sequence.

Item	Your status (fill today)
Host role	solid / partial / missing
HM or service user
Secrets
Edge/proxy
Deploy tool
Tests
CI + cache

Theory 2 — Stage coverage matrix

Stage	Theme	Evidence in repo (examples)
I	Foundations, flake shell	`devShells`, language notes
II	First host, users, net, systemd	`nixosConfigurations.*`, host modules
III	Layout, HM, checks	`home-manager`, flake structure
IV	Secrets, harden, proxy, data	sops, firewall, nginx/caddy
V	Packaging, overlays, caches	custom packages, cachix config
VI	Disko, deploy, images, tests, CI	deploy configs, workflows, tests
VII	Internals, upgrades	day71–81 notes; not all need code

Mark each: solid / partial / missing / N/A. Partial without a demo command is missing in disguise.

Theory 3 — Architecture views (write these)

A. Context

Who uses the system? Laptop only? Lab server? Family services? Threat model one paragraph.

B. Containers of code

flake.nix
hosts/
modules/
homes/
secrets/
tests/
deploy/
.github/ or .forgejo/
docs/

C. Runtime view

[client] → [proxy:443] → [app:8080]
                ↓
             [sops secrets]
                ↓
             [deploy-rs/colmena from CI or admin]

D. Trust view

Who can decrypt secrets?
Who is trusted-users?
Which caches are trusted?
Where do age master keys live offline?

E. Failure view

Boot rollback
Deploy failure
Secret loss (Day 88 needs offline keys)
Cache outage (can you still build?)

F. Data view

What is stateful?
Where are backups (Day 69)?
Impermanence list if any (Day 58)?

Theory 4 — Prioritization

Score gaps:

Priority	Definition
P0	Capstone A impossible without it
P1	Required for Day 88–89 proof quality
P2	Hardening / UX polish
P3	Nice / Capstone B

Sequence Days 83–87 as:

Day	Suggested focus
83	P0 host + module skeleton gaps
84	Secrets + proxy E2E
85	Deploy + CI + cache
86	Tests + runbooks + diagrams
87	Freeze, bugfix, checklist

Adjust if your matrix differs—write the plan. A plan you do not write will be renegotiated under panic every day.

Theory 5 — Non-goals for Capstone A

Explicitly list what you will not do (protects freeze day):

Multi-region mesh
Full k3s
Every self-hosted app
Capstone B fleet (unless ahead)
Experimental CA store (Day 75)
Perfect HA for lab toys

Non-goals are features, not apologies. Scope control is expertise.

Theory 6 — Evidence standards

Claim	Acceptable evidence
“We have sops”	Decrypt path works on host; secrets.nix in git
“We have CI”	Green URL + workflow file
“We have tests”	Check that fails when you break a module
“We deploy”	Log of remote activation this month
“Architecture exists”	`ARCHITECTURE.md` with diagrams

Screenshots of dashboards without repo paths do not count for Capstone A.

Lab 0 — Workspace

mkdir -p ~/lab/90daysofx/02-nixos/day82
cd ~/lab/90daysofx/02-nixos/day82

Work against the real Capstone flake path.

Lab 1 — Automated inventory

cd /path/to/flake
find . -name '*.nix' | head -200 | tee ~/lab/90daysofx/02-nixos/day82/nix-files.txt
nix flake show 2>&1 | tee ~/lab/90daysofx/02-nixos/day82/flake-show.txt
rg -n "sops|agenix|deploy-rs|colmena|home-manager|nixosTest|cachix" -g '*.nix' \
  | tee ~/lab/90daysofx/02-nixos/day82/feature-hits.txt
ls -la .github/workflows 2>/dev/null || ls -la .forgejo/workflows 2>/dev/null || echo "no CI yet"

Skim flake-show for dead attrs and surprise outputs.

Lab 2 — Stage matrix

Fill COVERAGE.md:

Stage	Status	Paths	Demo command	Notes
I
…
Capstone A items 1–7	each row

Be brutal. “I watched a video about sops” ≠ solid.

Lab 3 — `ARCHITECTURE.md` (required)

Minimum sections:

Purpose & scope
Host roles
Module map
Secrets model
Edge/proxy
Deploy & CI
Test strategy
Trust & keys
Backup/DR pointers
Gap list summary (link)
Non-goals
Pin / release (26.05) policy

Diagrams: ASCII is enough.

Lab 4 — Prioritized backlog

GAPS.md:

ID	Gap	Pri	Capstone day	Acceptance test
G01	No CI cache push	P0	85	workflow uploads
…

Every P0/P1 needs an acceptance test you could demo in under five minutes.

Lab 5 — Checklist freeze seed

Create CAPSTONE-A.md checklist (boxes empty/partial/done) that Days 83–89 will tick. Include wipe/rebuild and rollback rows even though execution is later.

# Capstone A checklist

## Core
- [ ] Host role …
## Days
- [ ] 83 …
- [ ] 88 wipe/rebuild
- [ ] 89 rollback + full tests

Lab 6 — Risk register (short)

Three risks that could sink Capstone A:

Risk	Likelihood	Impact	Mitigation
Only age key on lab disk			Offline copy now
No console on remote			Enable before firewall games
Tests don’t fail on breaks			Day 86 priority

Common gotchas

Mistake	Fix
Optimistic “partial = done”	Require demo command proof
Skipping docs because “I know it”	Capstone is proof for future you
40 P0s	Reclassify; cut scope
Planning Capstone B now	Park in non-goals
No acceptance tests on gaps	Unverifiable work
Architecture file never opened again	Link it from CAPSTONE-A.md

Checkpoint

Inventory artifacts (flake-show, feature hits)
COVERAGE.md stage matrix
ARCHITECTURE.md complete
GAPS.md prioritized with day mapping
CAPSTONE-A.md checklist seeded
Three personal gotchas

Commit

git add .
git commit -m "day82: architecture review and capstone gap plan"

Write three personal gotchas before continuing.

Tomorrow

Day 83 — Capstone 1/5: implement highest P0 gaps; keep the live checklist honest.

Day 27 — Troubleshooting & Architecture Review

Day 81 — Troubleshooting drill

Why this day exists

Theory 1 — Failure class matrix

Theory 2 — Golden command palette

Remote when SSH dies

Theory 3 — Injection catalog (choose carefully)

Theory 4 — Time discipline

Theory 5 — Playbook entry format

Lab 0 — Snapshot

Lab 1 — Baseline health

Lab 2 — Inject & recover #1 (eval or build)

Lab 3 — Inject & recover #2 (service or secrets)

Lab 4 — Inject & recover #3 (network) — console required

Lab 5 — Compile master index

Theory 6 — Rollback as a diagnostic tool

Theory 7 — Layered isolation order

Worked example — playbook stub

Lab 6 — Timeboxed chaos #4 (state)

Lab 7 — Master index cross-links

Common gotchas

Checkpoint

Commit

Tomorrow

Day 82 — Architecture review

Why this day exists

Theory 1 — Capstone A bar (recall)

Theory 2 — Stage coverage matrix

Theory 3 — Architecture views (write these)

A. Context

B. Containers of code

C. Runtime view

D. Trust view

E. Failure view

F. Data view

Theory 4 — Prioritization

Theory 5 — Non-goals for Capstone A

Theory 6 — Evidence standards

Lab 0 — Workspace

Lab 1 — Automated inventory

Lab 2 — Stage matrix

Lab 3 — ARCHITECTURE.md (required)

Lab 4 — Prioritized backlog

Lab 5 — Checklist freeze seed

Lab 6 — Risk register (short)

Common gotchas

Checkpoint

Commit

Tomorrow

Lab 3 — `ARCHITECTURE.md` (required)