Automated Fleet Rollout and Disaster Recovery

Updated

September 12, 2026

Automated Fleet Rollout and Disaster Recovery

A production architecture is only as dependable as its playbook. The boring default is: Colmena canary (--on app-01), health probe, then tagged apply; rebuild a dead box with nixos-anywhere from the same 26.05 flake — not a restore CD.

Mental model

The flake is the cluster. Colmena pushes closures. Canary is one host, then a tag. Disaster recovery is Disko + nixos-anywhere + restic of /persist (not of /nix/store).

git  →  colmena apply --on app-01  →  curl /healthz
     →  colmena apply --on @app
     →  serial db-01 after restic snapshot

Public IPs in the hive are the SSH targets. East-west traffic stays on wg0 (10.100.0.0/24). Rollback is the previous generation, then a git revert so the hive matches metal.

Worked examples

Case 1: Hive for the five-host fleet

Save as flake.nix:

# flake.nix
{
  description = "Desk production fleet";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

  outputs = { self, nixpkgs }: {
    colmena = {
      meta.nixpkgs = import nixpkgs { system = "x86_64-linux"; };

      defaults = { ... }: {
        system.stateVersion = "26.05";
        deployment.targetUser = "root";
      };

      gw-01 = {
        deployment.targetHost = "203.0.113.10";
        deployment.tags = [ "edge" ];
        imports = [ ./hosts/gw-01.nix ];
      };
      app-01 = {
        deployment.targetHost = "203.0.113.11";
        deployment.tags = [ "app" ];
        imports = [ ./hosts/app-01.nix ];
      };
      app-02 = {
        deployment.targetHost = "203.0.113.12";
        deployment.tags = [ "app" ];
        imports = [ ./hosts/app-02.nix ];
      };
      db-01 = {
        deployment.targetHost = "203.0.113.21";
        deployment.tags = [ "db" ];
        imports = [ ./hosts/db-01.nix ];
      };
      ops-01 = {
        deployment.targetHost = "203.0.113.30";
        deployment.tags = [ "ops" ];
        imports = [ ./hosts/ops-01.nix ];
      };
    };
  };
}
nix shell nixpkgs#colmena --command colmena eval

Five nodes. One 26.05 lock. follows on disko/sops if those inputs exist. If colmena eval says the flake attr moved, the Colmena pin wants colmenaHive + colmena.lib.makeHive — same hosts, different wrapper (see the Colmena chapter). deployment.allowLocalDeployment = true on ops-01 if that box applies itself.

Case 2: Canary, probe, then the tag

colmena apply --dry-run --on app-01
colmena apply --on app-01
curl -sf https://api.desk.internal/healthz
colmena apply --on @app
[app-01] Built /nix/store/7q1a…-nixos-system-app-01-26.05
[app-01] Activation successful
HTTP/2 200

Non-200: do not apply @app. Roll back app-01 (nixos-rebuild switch --rollback on the host, or Colmena’s rollback flag), revert git.

Never --on @all for a kernel bump. db-01 is serial, after restic snapshots shows a fresh copy.

Case 3: Zero-touch replace of app-02

Boot replacement metal into rescue (or a NixOS installer ISO). Disko’s passwordFile is a path on the installer after kexec. Copy the LUKS key there, generate hardware config into git, and do not reboot stateful boxes until /persist is restored.

From ops-01:

# LUKS: remote dest (Disko passwordFile) then local source
nix run github:nix-community/nixos-anywhere -- \
  --flake .#app-02 \
  --disk-encryption-keys /tmp/luks.key /run/secrets/luks/app-02 \
  --generate-hardware-config nixos-generate-config ./hosts/app-02/hardware-configuration.nix \
  --extra-files /tmp/app-02-seed \
  --no-reboot \
  root@203.0.113.99
[nixos-anywhere] Connected to rescue over SSH.
[nixos-anywhere] kexec NixOS installer...
[nixos-anywhere] Copied disk encryption key → /tmp/luks.key
[nixos-anywhere] Formatting disks via Disko...
[nixos-anywhere] Copying closure …-nixos-system-app-02-26.05
[nixos-anywhere] Copied extra files onto the new root
[nixos-anywhere] Installation complete. (--no-reboot)

--disk-encryption-keys <remote> <local> runs after kexec, before Disko. Repeat the flag per disk. The key is not in the flake.

--generate-hardware-config nixos-generate-config <path> writes initrd modules and NIC names back to the laptop so the next commit has a real hardware file. Backend can be nixos-facter instead; pick one.

--extra-files <dir> is a tree rooted at / of the new install (etc/ssh/ssh_host_ed25519_key, persist/…). Prepare it with cp --parents. This is how sops still decrypts on first boot (same host key). --copy-host-keys only copies keys from the machine you are wiping, which a blank replacement does not have.

--no-reboot leaves disks mounted so Case 4 can restore. Default phases are kexec,disko,install,reboot. Already on a NixOS installer ISO: --phases disko,install (skip kexec). --vm-test exercises Disko without touching metal.

Then reboot (or omit --no-reboot on stateless app-*). The node joins WireGuard (10.100.0.12). Ephemeral workers have nothing to restic. Stateful db-01 needs a snapshot before you wipe, and a restore before you reboot into postgres.

Case 4: Restore /persist on db-01 (not the store)

# still in the installer (or chroot) after --no-reboot, before first Stage 2 postgres:
sudo restic -r s3:s3.amazonaws.com/desk-restic snapshots
sudo restic -r s3:s3.amazonaws.com/desk-restic restore latest --target /mnt/persist
# then reboot into the new generation
sudo umount -R /mnt
sudo reboot

After boot:

sudo systemctl start postgresql
sudo -u postgres pg_isready

--target /persist vs /mnt/persist depends on whether you have already switched root. Wrong prefix dumps the tree next to the subvolume and postgres starts empty. The restic password is sops on the box (or RESTIC_PASSWORD_FILE), not argv. /nix/store comes from the flake + cache. Restoring a NAR of the old store is not DR.

If you omitted --no-reboot, postgres may already have initdb’d an empty cluster on the new persist. Restore onto that is a fight. Stop postgres, empty the data dir, restore, start. Better: never let it boot empty.

Case 5: Quarterly drill is the playbook

Save as docs/dr-drill.md:

# docs/dr-drill.md
1. Snapshot restic on db-01.
2. Wipe staging-app in the lab (not prod).
3. nixos-anywhere --flake .#staging-app root@…
4. curl /healthz from gw.
5. Time it. File the minutes in this repo.

If the drill needs a human in a web console past step 3, the ISO / Disko / sops recipients are incomplete. Fix the flake, not the wiki.

The trap

The trap is not testing disaster recovery until a production incident. The other trap is applying db-01 in the same parallel wave as @app while a migration runs. Serial the stateful node. Canary the stateless ones.

Rollback without a git revert means the next Colmena apply re-breaks the box.

A third trap is rebooting nixos-anywhere before /persist is restored on a stateful host — or omitting --disk-encryption-keys so Disko waits on a passphrase nobody will type in rescue. A fourth is generating hardware config and never committing it: the next install misses a kernel module.

The boring rule

  • One hive, one 26.05 lock. SSH from ops-01.
  • --dry-run, --on app-01, healthz, then --on @app. Serial db-01.
  • Dead metal: nixos-anywhere + Disko. --disk-encryption-keys, --generate-hardware-config, --extra-files for host keys. --no-reboot until persist is restored on stateful nodes.
  • Persist: restic. Store: substituters.
  • Rollback is a generation, then git revert.
  • DR drill in git. Time it.

Try this

  1. colmena eval and list tags.
  2. Dry-run --on app-01; read which units would restart.
  3. Write the restic repository URL (not the password) next to the drill doc.
  4. Wipe a lab VM and nixos-anywhere it with --no-reboot; restore a dummy file into /mnt/persist; reboot; confirm it survived. wg show has peers.