Operations and fleet

Updated

July 30, 2026

Operations and fleet

This part of the standalone NixOS book is organized by topic (normal book chapters), not by calendar day.

Read chapters in order within the part when learning; skip or skim freely when using the book as a reference.

Baseline: Nix 2.34.x, flakes-first, NixOS 26.05. Deploy tool CLIs, Disko modes, and generator format names evolve—prefer current upstream READMEs when a flag renames.

What this part is for

Packaging craft produces store paths. Operations and fleet ships them safely:

  1. Disks as code (Disko) so reinstalls are boring
  2. Impermanence patterns so roots stop accumulating snowflakes
  3. Deploy tools (deploy-rs, Colmena) for remote activation
  4. Shared modules so ten hosts do not fork ten firewalls
  5. Images (nixos-generators) for cloud and edge
  6. Terraform boundary so cloud APIs and NixOS stay friends
  7. Tests and CI so regressions die in PRs
  8. GC hygiene, Secure Boot/TPM awareness, backups so the fleet survives Monday

Without this part, NixOS is a clever single host. With it, NixOS is an operations platform.

Prerequisites

Skill Source
Flake host, rebuild, generations NixOS host part
Modules as consumer and author NixOS host part
Secrets (sops-nix or equivalent) Services and security
Binary caches / builders vocabulary Packaging craft
Optional: packaging your own modules’ packages Packaging craft

You need at least one disposable machine or VM. Disko and impermanence are unsafe on irreplaceable disks.

Chapter map

Chapter File Focus
Disko 01-disko Declarative partitioning / filesystems
Impermanence 02-impermanence Ephemeral root + persist lists
deploy-rs 03-deploy-rs Profile deploy over SSH
Colmena 04-colmena Hive, tags, multi-node
Shared modules 05-shared-modules Roles, profiles, host thinness
Images / generators 06-images-generators ISO, qcow, cloud images
Terraform boundary 07-terraform-boundary TF provisions; Nix configures
nixosTest 08-nixos-tests VM integration tests
More testing 09-more-tests Check suite / pyramid
CI pipelines 10-ci-pipeline Forge CI + flakes + cache
GC hygiene 11-gc-hygiene Roots, generations, optimise
Secure Boot / TPM 12-secure-boot-tpm Lanzaboote awareness, LUKS+TPM
Backups 13-backups Restic/Borg/ZFS + restore drill

Suggested path: Disko → Impermanence → deploy-rs → Colmena → shared modules, then images ↔︎ terraform, then tests → CI, finish with GC / SB-TPM / backups.

Mental model — fleet pipeline

modules + host data
        │
        ├─► nixosConfigurations / hive
        │         │
        │         ├─► deploy-rs / Colmena / nixos-rebuild
        │         ├─► nixos-generators images
        │         └─► nixosTest VMs
        │
        ├─► Disko layouts + (optional) impermanence
        ├─► secrets activation (sops-nix / agenix)
        └─► binary cache + remote builders
Stage Failure mode First fix locus
Eval Module merge errors Shared modules / host
Build Compile / missing cache Packaging / CI cache
Copy SSH / disk Deploy tool / network
Activate Switch scripts, boot Host modules
Prove Tests / canaries nixosTest / live probes
Survive Disk full / no backup GC / backups

Disk and identity

Disko describes blocks
Impermanence describes what must survive reboot
Backups describe how state returns after disaster
Secure Boot/TPM describes boot trust and unlock

These four chapters answer different questions. Confusing “declarative disk” with “backed up database” is a classic Capstone failure mode.

Deploy tools at a glance

Tool Strength Typical use
nixos-rebuild --target-host Built-in One-off / small
deploy-rs Profiles, magic rollback patterns Small–medium fleets
Colmena Hive, tags, parallel apply Multi-role fleets

You do not need both forever. Leave this part with a deliberate choice and one working path.

Lab topology suggestions

[admin laptop]
     │ SSH
     ├─► lab-vm-1  (Disko reinstall target)
     ├─► lab-vm-2  (second Colmena node or remote builder)
     └─► optional: cache VM (Harmonia/Attic)

Cloud stretch:
[OpenTofu] → VM + disk + SG → [image from nixos-generators] → deploy-rs

Suggested workspace root:

~/lab/nixos/ops/
  disko/
  impermanence/
  deploy/
  colmena/
  modules/          # may be your main flake
  images/
  terraform/
  tests/
  ci/
  gc/
  secureboot/
  backups/

Many labs live inside one monorepo flake with nixosConfigurations, deploy, colmena, and checks—prefer that once you have more than two hosts.

How chapters are structured

  1. Goal — done criteria
  2. Why it matters — ops motivation
  3. Theory — models + complete Nix fragments
  4. Worked examples — 26.05-shaped
  5. Lab — multi-step, disposable infrastructure
  6. Pitfalls — symptom → fix
  7. Checkpoint — self-review
  8. Commit — optional discipline

Reading paths

Path A — “Homelab single server → clean reinstall”

  1. Disko
  2. Impermanence (or conscious reject)
  3. deploy-rs
  4. Backups
  5. GC hygiene

Path B — “Three nodes with roles”

  1. Shared modules
  2. Colmena (or deploy-rs multi-node)
  3. nixosTest + more tests
  4. CI pipeline
  5. Binary cache (packaging part)

Path C — “Cloud image factory”

  1. Shared modules
  2. Images / generators
  3. Terraform boundary
  4. CI pipeline
  5. Secure Boot awareness (if metal/edge)

Glossary (part-local)

Term Meaning
Disko Declarative disk partitioning/formatting for NixOS
Impermanence Ephemeral root; explicit persist paths
Profile Nix profile activation target (system generation)
Hive Colmena multi-node definition
Tag Colmena selection label (@web)
nixosTest QEMU VM integration test harness
nixos-generators Build disk/ISO/cloud images from modules
Boundary Clear split TF vs Nix responsibilities
GC root Reference that keeps store paths alive
Lanzaboote Secure Boot integration for NixOS generations
Restore drill Prove backups by restoring, not only by job logs

Safety rules (non-negotiable)

Rule Why
Disposable disks for Disko experiments Wipes are real
Recovery USB before Secure Boot enrollment Bricks happen
Never paste production secrets into examples Supply chain + git history
PR forks often must not get push tokens Cache/signing TCB
GC without understanding roots Can remove needed paths; rarely deletes running system if roots correct—but can remove rollback options

Quality bar for “fleet ready”

Criterion Evidence
Storage is code Disko config in git; reinstall notes
Hosts are thin Roles/profiles, not copy-paste
Remote activate works deploy-rs or Colmena success log
Regressions caught nix flake check / CI green
State is honest Backup + restore drill document
Disk won’t fill silently GC policy + monitoring note
Boot trust known SB/TPM dossier or enablement notes

Relationship to other parts

Part Connection
NixOS host Single-host skills scale to fleets here
Services / security Secrets, firewall, TLS feed deploy targets
Packaging Caches/builders feed CI and deploy speed
Internals Deeper module/eval performance later
Capstone Combines ops chapters into a rebuildable system
Projects Optional drills; this part is the ops spine

No day-calendar language; no parallel series cross-links. This is the standalone library’s operations volume.

Anti-patterns

Anti-pattern Prefer
Hand-edited /etc on prod nodes Module + deploy
Unique firewall per host by copy-paste Shared profile
Image modules forked from live hosts One module graph → image + deploy
TF user_data installing Docker via curl Golden NixOS image + Nix modules
Tests only on laptop CI nix flake check
Backup job green without restore Restore drill
“We’ll enable Secure Boot later” with no notes Written path + recovery plan

Part checkpoint (before you leave ops)

  • Disko layout for at least one disposable host; reinstall or format proof
  • Impermanence implemented or written reject with inventory of what must persist
  • One successful remote deploy (deploy-rs and/or Colmena)
  • Roles/profiles refactor started (hosts thinner than before)
  • One image artifact built from shared modules or documented stretch plan
  • Terraform/Nix boundary sketch written
  • At least one nixosTest (or documented KVM blocker)
  • Three-check suite shape in flake
  • CI workflow runs checks on the forge you use
  • GC policy + roots understanding demonstrated
  • Secure Boot/TPM awareness dossier
  • Backup tool chosen + restore drill documented

Suggested lab commit series

git commit -m "ops: disko layout and reinstall notes"
git commit -m "ops: impermanence or reject rationale"
git commit -m "ops: deploy-rs end-to-end"
git commit -m "ops: colmena hive and tags"
git commit -m "ops: shared roles and profiles"
git commit -m "ops: nixos-generators image"
git commit -m "ops: terraform boundary sketch"
git commit -m "ops: nixosTest for role module"
git commit -m "ops: flake check suite"
git commit -m "ops: CI workflow"
git commit -m "ops: GC policy"
git commit -m "ops: secure boot / TPM notes"
git commit -m "ops: backup + restore drill"

How to use this part as reference

Symptom Jump to
Reinstall tribal knowledge Disko
Root full of mystery state Impermanence
Need SSH activate deploy-rs
Need tags / many hosts Colmena
Copy-paste hosts Shared modules
Cloud boot drift Images; Terraform boundary
“Works on my VM” nixosTest; more tests; CI
Disk full mid-deploy GC hygiene
Evil maid / datacenter unlock Secure Boot / TPM
DB gone after rebuild Backups

Next

Start with Disko on a disposable VM: declare ESP + root, format, install from a flake, and prove a second reinstall from the same code. Everything else in this part assumes storage is no longer folklore.