Upgrade planning

Updated

July 30, 2026

Upgrade planning

Goal: Produce a complete dry-run upgrade runbook from 26.05 → 26.11 (next cadence release): input bumps, test gates, staged rollout, rollback, and risk register—without requiring a production upgrade today.

Note

NixOS releases target roughly May/November cadence (.05 / .11). 26.11 may be upcoming, in beta, or stable depending on when you read this. The deliverable is the plan; execute only on lab when ready.

Why this chapter exists

Release-note literacy without a runbook still produces Friday-night lock bumps. Upgrades are projects: branch, gates, lab first, rollback written cold.

Theory 1 — What actually changes in a flake upgrade

flake.nix inputs (nixpkgs url/ref)
    ↓
flake.lock (revs, narHashes)
    ↓
eval modules against new nixpkgs
    ↓
realize new toplevel
    ↓
switch / reboot if kernel
    ↓
stateful service migrations (stateVersion-aware)
Step Failure mode
Lock bump Unrelated input moves (HM, sops, deploy-rs) break together
Eval Option removed/renamed; type errors
Build Package broken; FODs; your overlay
Switch Service fails activation
Runtime Data migration, socket changes, API versions
Reboot Bootloader/kernel/initrd issues

Theory 2 — Staging strategy

Environments

Stage Purpose
Dev laptop / lab VM First eval/build/test
CI on branch Checks + cache populate
Non-prod server Real networking/secrets
Prod Last; maybe canary host

Git strategy

main (26.05 pin)
  └─ branch upgrade/26.11
       ├─ bump nixpkgs only
       ├─ fix eval
       ├─ bump companion inputs (HM, etc.)
       └─ PR with checklist

Avoid one commit that bumps six inputs and refactors modules.

Companion inputs

Home Manager, sops-nix, deploy-rs, disko, agenix—each has branches/tags tracking nixpkgs. Read their upgrade notes. Mismatched HM vs nixpkgs is a classic footgun.

Theory 3 — Tests-first gate

Minimum gates before switch on anything important:

nix flake check
nix build .#nixosConfigurations.<host>.config.system.build.toplevel
nix build .#checks.<system>.<integration>
# deploy to lab only:
# deploy .#lab -- or colmena apply --on lab

Optional: nixos-rebuild build-vm / smoke VM.

Policy: red checks ⇒ no promote.

Theory 4 — Rollback is part of the plan

Layer Mechanism
Boot Previous generation in bootloader
Runtime nixos-rebuild switch --rollback / rollback generation
Git Revert lock on branch
Secrets Keys unchanged; ciphertext may need re-encrypt only if tooling changes
Data DB dumps / zfs snapshots before upgrade

Document RTO goals for lab vs prod. Capstone disaster recovery practices wipe/rebuild and rollback drills.

Concrete rollback commands (fill for your path)

# local rebuild host
sudo nixos-rebuild switch --rollback
# list generations
sudo nix-env -p /nix/var/nix/profiles/system --list-generations
# deploy-rs / colmena: redeploy previous known-good commit
git revert <lock-bump-commit>
# then redeploy

Theory 5 — stateVersion and stateful services

Upgrading nixpkgs ≠ automatically bumping stateVersion.

Plan section required:

  • Current stateVersion
  • Whether next release notes require migration
  • Snapshot/backup steps for PostgreSQL, Vaultwarden, etc. you actually run
  • Who verifies app-level health post-switch

Theory 6 — Communication & windows

Even for homelab:

  • Maintenance window
  • What breaks for family/users
  • Where logs live
  • Abort criteria (e.g. “if proxy down >15m, rollback”)

Abort criteria template

Signal Action
Eval/build red on CI Do not deploy
systemctl --failed non-empty after switch Investigate ≤15m then rollback
Proxy healthcheck fail Rollback
No console / no SSH Prefer prevention; fix access before upgrade

Theory 7 — Elitebook / small-lab constraints

Constraint Plan adjustment
16 GB RAM Lower max-jobs during upgrade builds
Limited disk GC old generations after successful upgrade, not mid-build; ensure free space for two closures
Single physical host Prefer guest upgrade first; snapshot VM
Nested virt Confirm KVM still works after kernel bump

Runbook template (you will fill this)

Create UPGRADE-26.11.md with these sections:

0. Metadata

  • Author, date, target hosts
  • Current pins (flake.lock nixpkgs rev)
  • Target ref (github:NixOS/nixpkgs/nixos-26.11 or unstable interim)

1. Preconditions

  • Backups verified restorable
  • CI green on main
  • Disk space for two closures
  • Access to console/IPMI if remote
  • sops/age keys available offline

2. Release notes

  • Links to 26.11 notes (when published) + diff against your 26.05 IMPACT.md method
  • ACTION items list

3. Technical steps (lab first)

git checkout -b upgrade/26.11
# edit flake.nix input for nixpkgs
nix flake lock --update-input nixpkgs
# then carefully update HM / sops / etc.
nix flake check
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L
# deploy lab
# health checks

4. Health verification

  • HTTP endpoints
  • systemctl --failed
  • Specific app probes
  • Logs

5. Rollback

Exact commands for your deploy path (deploy-rs/colmena/rebuild).

6. Risks

Risk Likelihood Impact Mitigation
…

7. Decision

Go / no-go criteria for prod.

Lab 0 — Workspace

mkdir -p ~/lab/nixos/internals/upgrade
cd ~/lab/nixos/internals/upgrade

Lab 1 — Inventory pins

cd /path/to/flake
nix flake metadata --json > ~/lab/nixos/internals/upgrade/metadata.json
cp flake.lock ~/lab/nixos/internals/upgrade/flake.lock.baseline

List every input and its tracking branch/tag in INPUTS.md.

Lab 2 — Write the full runbook

Complete UPGRADE-26.11.md from the template. Mark unknown 26.11 specifics as TBD when notes ship but keep the structure solid. Reuse the release notes method for notes intake.

Acceptance: All seven template sections present; rollback commands are copy-pasteable for your path.

Lab 3 — Dry-run mechanics on lab (optional execution)

If 26.11 exists and you have time:

  1. Branch
  2. Update nixpkgs only
  3. Eval and fix until toplevel builds
  4. Do not require prod switch

If 26.11 is not out: dry-run against nixos-unstable on a disposable branch/VM only, or stop at lock-file rehearsal:

# rehearsal of commands without pushing
nix flake lock --update-input nixpkgs --output-lock-file /tmp/test.lock

Document what you did / skipped.

Lab 4 — Test plan matrix

TEST-PLAN.md:

Check Command Blocks promote?
flake check nix flake check Yes
host toplevel nix build … Yes
nixosTest X … Yes
manual proxy curl … Yes for edge host
backup restore drill … Prod only

Lab 5 — Abort & rollback tabletop

Without breaking prod, write the exact rollback commands for:

  1. Failed switch, system still booting old generation
  2. Booted new generation, services bad
  3. Git revert of lock after partial deploy

Acceptance: Three scenarios with commands; no “I’ll figure it out live.”

Lab 6 — Disk space math

df -h /nix
nix path-info -S /run/current-system 2>/dev/null || true

Record free space and estimated second closure headroom in the runbook preconditions.

Common gotchas

Mistake Fix
Big-bang all inputs Stage nixpkgs first
No backup verification Test restore once
Forgetting companion inputs HM/sops/disko matrix
Upgrading prod first Lab → CI → prod
No abort criteria Write them cold
Bumping stateVersion with release Separate decision
GC mid-upgrade Free space first; GC after success

Checkpoint

  • INPUTS.md + lock baseline
  • Complete UPGRADE-26.11.md runbook
  • TEST-PLAN.md with blockers
  • Rollback tabletop written
  • Explicit statement: prod upgraded or not (expected: not required)
  • Three personal gotchas

Commit

git add .
git commit -m "lab(nixos): 26.05 to 26.11 upgrade runbook dry-run"

Write three personal gotchas before continuing.

Next

→ Troubleshooting drills — inject, recover, playbook.