Day 22 — Stage II gate

Updated

July 30, 2026

Day 22 — Stage II gate

Stage II · ~4h (review + proof)
Goal: Prove a minimal production-shaped flake-managed NixOS lab: users, SSH keys, firewall, one service, one custom module—cold boot and rollback work; the flake is the only source of truth.

Why this day exists

Stage III multiplies structure (layouts, Home Manager). Entering it with a snowflake host guarantees pain. Today is an honesty gate: the machine is rebuildable, operable, and documented.

Theory 1 — Syllabus exit criteria

You may leave Stage II when:

  1. From cold boot you can roll back a bad generation
  2. The flake is the only source of truth for the lab host
  3. Host includes: declarative users, SSH key access, firewall policy, at least one service/timer, one custom module
  4. Notes cover boot path + network + recovery

Theory 2 — Minimal production-shaped host (definition for this gate)

Capability Evidence
Flake host nixosConfigurations.<name> rebuilds
Pin flake.lock committed; nixpkgs 26.05
User Non-root admin in wheel
Remote access SSH key login; password SSH off (or justified)
Firewall Enabled; SSH allowed; extras documented
Packages Daily tools declarative
Service Custom or high-level service active
Module Local options+config module imported
Docs rebuild, rollback, boot, network
Recovery Rollback card tested

Not required yet: secrets machinery, Home Manager, Disko redesign, multi-host, reverse proxy zoo.

Theory 3 — Cold boot proof

running system (gen N)
  → reboot
  → firmware → loader → kernel → multi-user
  → SSH/console login works
  → systemctl is-system-running (or degraded explained)

Cold boot catches “worked in this session only” activation myths.

Theory 4 — Source of truth test

Signal flake is SoT Smell of drift
Rebuild from git repo only Hand edits in /etc that vanish/surprise
README rebuild command works “Also run these shell fixes”
hardware-config in git Hardware only on machine, never backed up

Theory 5 — Gate knowledge verbalization

Prepare short answers:

  1. switch vs test vs boot
  2. What rollback does not restore
  3. Why SSH public keys in git are OK but passwords are not
  4. Store vs state paths
  5. Why custom modules use mkIf cfg.enable
  6. How you open the boot generation menu

Worked examples bank

Example A — Gate rebuild

cd ~/nixos-config
sudo nixos-rebuild switch --flake .#lab

Example B — Eval sanity

nix flake show
nix eval .#nixosConfigurations.lab.config.networking.hostName
nix eval .#nixosConfigurations.lab.config.networking.firewall.allowedTCPPorts

Example C — Service proof

systemctl is-enabled sshd || systemctl status sshd
systemctl list-timers --all | head

Lab 1 — Gate repository hygiene

cd ~/nixos-config
git status
git log --oneline | head
test -f flake.lock && echo lock-ok

Ensure structure roughly:

flake.nix
flake.lock
hosts/lab/configuration.nix
hosts/lab/hardware-configuration.nix
modules/*.nix
docs/{boot,network,rollback,filesystems,modules-study}.md
README.md

Missing docs → write them now (brief is fine).

Lab 2 — Checklist (tick only if true)

Stage II gate checklist

  • NixOS 26.05 lab boots (VM preferred)
  • nix --version / nixos-version recorded
  • Flake nixosConfigurations rebuild switch works
  • flake.lock committed
  • Hostname + flake attr documented
  • Admin user declarative + wheel
  • SSH key login works
  • PasswordAuthentication disabled or exception documented
  • Firewall enabled; SSH port allowed
  • Port audit exists
  • environment.systemPackages / programs.* used intentionally
  • ≥1 custom systemd service or timer or meaningful services.*
  • ≥1 custom module with enable option
  • Assertion or deliberate option validation somewhere
  • Generations listed; rollback performed at least once in Stage II
  • Boot loader + generation menu documented
  • Mount inventory + persist list written
  • README rebuild instructions accurate
  • No private keys or live passwords in git
  • Cold boot → login succeeds

Any empty box is a stop for Stage III.

Lab 3 — Cold boot ceremony

  1. VM snapshot labeled day22-pre-coldboot (optional safety)
  2. Reboot lab
  3. Login console and/or SSH
  4. systemctl is-system-running || true
  5. hostname; curl localhost service if any
  6. Record timestamp + generation

Lab 4 — Rollback proof (again)

  1. Make a tiny visible change (motd package, environment.etc."day22".text, …)
  2. switch
  3. Confirm change
  4. sudo nixos-rebuild switch --rollback
  5. Confirm reversion
  6. Re-apply good config if needed

Lab 5 — Architecture one-pager

docs/host-diagram.md:

# lab host
- flake: ~/nixos-config #lab
- users: …
- ssh: keys only …
- firewall: …
- services/timers: …
- modules: …
- boot: …
- disks: …
- what is state: …

Lab 6 — Teach-back (10–15 min)

Explain aloud or written:

“Here is how I rebuild this host from git; here is how I recover from a bad generation; here is what I must back up because Nix will not.”

If rough, revisit Days 13–15 and 21.

Theory 6 — Generation evidence commands

sudo nix-env -p /nix/var/nix/profiles/system --list-generations | tail -5
readlink -f /run/current-system
nixos-version

Gate notes should include current generation number and flake shortrev if you embed git info (optional).

Theory 7 — What “flake is SoT” excludes

Still allowed outside the flake (state):

  • Home directory contents (until HM owns them)
  • SSH host keys
  • Service databases (when you add them)
  • Hypervisor snapshots

Not allowed as required ops procedure:

  • “Also apt install…”
  • “Then manually systemctl edit…”
  • “Copy this binary into /usr/local”

Worked example D — Eval probes for gate

cd ~/nixos-config
nix eval .#nixosConfigurations.lab.config.users.users.alice.isNormalUser
nix eval .#nixosConfigurations.lab.config.services.openssh.enable
nix eval .#nixosConfigurations.lab.config.networking.firewall.enable

Boolean/string outputs are quick CI-less checks that config intent is present.

Lab 7 — Secret smell re-scan

cd ~/nixos-config
rg -n "initialPassword|password\s*=\s*\"" || true
rg -n "PRIVATE KEY|AKIA" || true

Stage IV will add secret machinery; Stage II must not already leak.

Lab 8 — Offline notes pack

Bundle in docs/ or a gate/ folder:

  1. Rebuild one-liner
  2. Rollback one-liner
  3. Boot menu steps
  4. Console lockout recovery
  5. Network identity (IP/interface)

Print or keep on the host that is not the VM (so a broken VM does not eat the runbook).

Common gotchas

Symptom Theory
Gate “passes” but docs missing Future you fails
Secrets in history Stage IV will not save past leaks
Service only started manually Not declarative
Hardware file not in git Disk loss → pain
Still using channels for system Diverges from flake story

Checkpoint

  • All Stage II checklist boxes ticked
  • Cold boot verified
  • Rollback re-verified
  • Host diagram written
  • Teach-back completed
  • Ready for Stage III layout + Home Manager

Commit

cd ~/nixos-config
git add .
git commit -m "day22: stage II gate production-shaped host"

Tag optional:

git tag stage-ii-gate

Tomorrow

Day 23 — Flake layout. Split hosts/ / modules/ / homes/ cleanly; stop growing a single tangled config as Stage III begins (Home Manager arrives soon after).