Day 89 — Rollback & full tests

Updated

July 30, 2026

Day 89 — Rollback & full tests

Stage VIII · ~4h
Goal: Prove generation rollback works after a bad change, run the full test/CI suite green, and finalize the troubleshooting checklist that ties Day 81 playbooks to Capstone reality (post-DR host on NixOS 26.05).

Note

Day 88 proved rebuild from zero. Day 89 proves you need not always go to zero: generations + tests + a crisp checklist cover the common middle ground. Do the real break—theater fails the day.

Why this day exists

Most production incidents are not “disk gone.” They are “bad deploy an hour ago.” Without rollback muscle and a short checklist, you re-image for a typo. Capstone A is incomplete until both DR and rollback are proven with logs.

Theory 1 — Rollback layers

Layer When to use How
Bootloader generation Unbootable / broken after reboot Select previous generation at boot
Runtime rollback System boots but config bad nixos-rebuild switch --rollback or activate previous profile
Git revert Want source truth back git revert / deploy freeze tag
Deploy tool Remote Redeploy known-good attr/tag
Full DR Disk/keys/corruption Day 88 path

Capstone must document which you try first for remote lab (console vs SSH).

Commands (lab)

# list generations
sudo nix-env -p /nix/var/nix/profiles/system --list-generations
ls -la /nix/var/nix/profiles/system*

# rollback switch
sudo nixos-rebuild switch --rollback
# or
sudo /nix/var/nix/profiles/system-GEN-link/bin/switch-to-configuration switch

With flakes, rebuild a known commit:

sudo nixos-rebuild switch --flake github:you/capstone?rev=<good>#lab
# or local checkout of freeze tag
sudo nixos-rebuild switch --flake .#lab

Theory 2 — Safe bad-change protocol

  1. Note current generation number.
  2. Introduce deliberate recoverable fault (typo unit, bad proxy upstream).
  3. switch and confirm break.
  4. Rollback.
  5. Confirm health.
  6. Re-deploy good config from git so source matches running system.

Never leave “rolled back runtime + dirty git” undocumented.

Good faults for drill Bad faults for drill
Wrong upstream port on proxy rm -rf /nix
Broken oneshot that fails activation Disk wipe (already Day 88)
Deliberately failing health unit Firewall lockout without console

Theory 3 — Full suite definition

“Full” means everything Capstone claims:

nix flake check -L
# explicit if check is subset:
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L
nix build .#checks.<system>.<each> -L
# CI: confirm latest main/freeze pipeline green

Post-DR host should also pass runtime probes from RUNBOOK.

Suite piece Pass criteria
nix flake check Exit 0; log saved
System toplevel build Exit 0
CI pipeline Green on freeze or main+fixes
Runtime probes Same as Day 88 healthy bar

Theory 4 — Final troubleshooting checklist

Produce docs/TROUBLESHOOTING.md (one pager + links):

# Troubleshooting checklist (Capstone)

## 1. Is it eval/build?
- [ ] `nix build .#nixosConfigurations.lab.config.system.build.toplevel`
- [ ] Read error; module bisect

## 2. Is it activation/service?
- [ ] `systemctl --failed`
- [ ] `journalctl -u … -b`

## 3. Is it network?
- [ ] console access?
- [ ] `ss -lntp`; firewall; proxy

## 4. Is it secrets?
- [ ] secret path exists; mode; sops key

## 5. Rollback?
- [ ] previous generation
- [ ] redeploy freeze tag

## 6. DR?
- [ ] docs/DR-PLAN.md

See playbook/PL-*.md for worked incidents.

Tie to Day 81

If you wrote playbooks on Day 81, link each PL-* from this checklist. Capstone should not have two competing “how to debug” docs.

Theory 5 — Source of truth after rollback

profile rollback  ≠  git fixed
        │
        ▼
you must redeploy good flake so next boot/rebuild matches intent
State Risk
Good profile, dirty bad git Next deploy re-breaks
Good git, bad profile Fixed by switch/deploy
Both good, untested Still run probes

Document the post-rollback “realign” step in RUNBOOK § Rollback.

Theory 6 — Remote rollback realities

Access Prefer
SSH still works switch --rollback or redeploy
SSH dead, console lives Bootloader generation
No console, SSH dead Avoidable with magic rollback (deploy-rs) / out-of-band
Disk corrupted Day 88

Practice bootloader selection once on the VM before Capstone freeze if you never have.

Lab 0 — Baseline after Day 88

mkdir -p ~/lab/90daysofx/02-nixos/day89
cd /path/to/capstone-flake
git checkout capstone-a-freeze  # or main if fast-forwarded with fixes
sudo nix-env -p /nix/var/nix/profiles/system --list-generations \
  | tee ~/lab/90daysofx/02-nixos/day89/gens-before.txt

Health probe baseline from RUNBOOK → baseline-health.txt.

Lab 1 — Inject recoverable failure

Example:

# temporarily wrong proxy upstream port in your proxy module
# or a deliberately failing systemd oneshot for the drill
sudo nixos-rebuild switch --flake .#lab 2>&1 | tee bad-switch.log
curl -vf … || true
systemctl --failed | tee failed-bad.txt

Record generation number of bad state in notes.

Lab 2 — Rollback drill

sudo nixos-rebuild switch --rollback 2>&1 | tee rollback.log
# health
curl -f …
systemctl --failed | tee failed-after-rollback.txt

If using deploy-rs/colmena, also practice redeploying freeze tag from admin machine.

Document exact commands in docs/RUNBOOK.md § Rollback (verify accuracy against what you just typed).

Lab 3 — Realign source

git status
# ensure working tree is good config (revert drill commit or reset file)
sudo nixos-rebuild switch --flake .#lab
# or deploy .#lab / colmena apply --on lab

Running system and git should agree. Record nixos-version + generation after realign.

Lab 4 — Full tests

nix flake check -L 2>&1 | tee ~/lab/90daysofx/02-nixos/day89/full-check.log

Fix any flakes introduced by Day 88 doc-only commits or drift. CI must be green.

Lab 5 — Finalize artifacts

  1. docs/TROUBLESHOOTING.md
  2. CAPSTONE-A Day 89 rows
  3. Short docs/PROOF.md:
Proof Evidence path
Deploy day85 log / command
Secrets+proxy day84 notes
CI/cache CI URL
Wipe/rebuild DR-REPORT RTO
Rollback day89 rollback.log
Tests full-check.log

Lab 6 — Teach-back (5 minutes)

Without notes, say out loud:

  1. First three steps when SSH works but service is wrong
  2. When you choose bootloader vs --rollback
  3. When you escalate to Day 88

If you stumble, fix TROUBLESHOOTING.md until you do not.

Common gotchas

Symptom What to do
Rollback works but flake still bad Redeploy good git; don’t stop at profile rollback
No previous generation You only have one—make two before drill
Boot menu rollback needed Practice on VM with console
Tests need old host keys Keep test dummy secrets
Skipping real break Drill fails the day
CI green, runtime red Runtime probes are part of “full”
Checklist too long One page + links; details in playbooks

Checkpoint

  • Bad generation induced and observed
  • Rollback restores health
  • Git and running system realigned
  • Full nix flake check (or documented full suite) green
  • TROUBLESHOOTING.md + PROOF.md
  • CAPSTONE-A nearly complete

Commit

git add .
git commit -m "day89: rollback drill full tests troubleshooting checklist"

Write three personal gotchas before continuing.

Tomorrow

Day 90 — Retrospective & roadmap: teach-back plan, personal roadmap (26.11, PRs, Capstone B), volume close.