Day 21 — Colmena, Shared Modules & nixos-generators

Updated

July 30, 2025

Day 21 — Colmena, Shared Modules & nixos-generators

Day 60 — Colmena

Stage VI · ~4h
Goal: Express a small hive with Colmena, deploy at least two nodes (one may be local), use tags, and know when Colmena wins over deploy-rs for fleet shape. Hosts pin NixOS 26.05.

Note

deploy-rs is excellent node/profile oriented glue. Colmena shines when you think in hives: many hosts, tags (@web, @db), parallel apply, and a flake hive attribute. Real orgs often evaluate both; leave Day 70 with a deliberate choice, not two half-wired tools forever.

Why this day exists

One remote deploy proves the pipe. Fleets need selection, defaults, and blast-radius control. Colmena’s mental model maps cleanly onto roles you will build on Day 61.

Theory 1 — Hive mental model

meta (nixpkgs, specialArgs)
  nodes
    host-a → NixOS module stack
    host-b → …
  defaults → shared modules
Concept Role
meta.nixpkgs Package set for evaluation
defaults Applied to every node
nodes.<name> Per-host config + deployment keys
tags Selection subsets for apply
specialArgs Inject libs/secrets helpers carefully

Theory 2 — Flake-based Colmena (modern)

{
  description = "Day 60 colmena hive";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

  outputs = { self, nixpkgs, ... }: {
    colmena = {
      meta = {
        nixpkgs = import nixpkgs { system = "x86_64-linux"; };
        # specialArgs = { inherit something; };
      };

      defaults = { pkgs, ... }: {
        time.timeZone = "UTC";
        services.openssh.enable = true;
        environment.systemPackages = [ pkgs.vim pkgs.curl ];
        system.stateVersion = "26.05";
      };

      edge = { name, nodes, pkgs, ... }: {
        deployment = {
          targetHost = "edge.lab.internal";
          targetUser = "root";
          tags = [ "edge" "web" ];
        };
        networking.hostName = "edge";
        imports = [ ./hosts/edge ];
      };

      core = {
        deployment = {
          targetHost = "core.lab.internal";
          tags = [ "core" ];
          # allowLocalDeployment = true; # when node is the machine itself
        };
        networking.hostName = "core";
        imports = [ ./hosts/core ];
      };
    };
  };
}
nix shell nixpkgs#colmena -c colmena apply
nix shell nixpkgs#colmena -c colmena apply --on @web
nix shell nixpkgs#colmena -c colmena apply-local --sudo   # local node patterns

CLI details (colmena apply --on host)—run colmena --help for your version. Some pins expose colmenaHive—follow current Colmena flake docs if colmena attr alone is insufficient.

Theory 3 — Tags and selection

colmena apply --on edge
colmena apply --on @web
colmena apply --on @web,@db
# exclusions / filters: check help for your version
Habit Why
Tag by role not only name Scale
Tag by risk (canary) Staged rollouts
Never tag prod+lab together casually Blast radius
Tag by arch if mixed Avoid wrong builds

Canary pattern

1. apply --on @canary
2. health probes
3. apply --on @web   # rest of role

Theory 4 — deploy-rs vs Colmena (honest)

Dimension deploy-rs Colmena
Flake-native profiles Strong Strong hive attr
Multi-host UX Nodes map Hive + tags first-class
Magic rollback Notable feature Different tooling
Learning curve Lower for 1 node Pays off at N>2
Ecosystem Serokell zhaofengli / community

House rule suggestion: deploy-rs for 1–2, Colmena when tags appear—or standardize on one to reduce cognitive load. Pick deliberately by Day 70 and document in FLEET-TOOLING.md.

Theory 5 — Build & push options

Colmena can build locally or on targets depending on deployment.buildOnTarget and related options:

deployment = {
  targetHost = "edge.lab.internal";
  buildOnTarget = false;
  # tags = [ "edge" ];
};
Mode When
Build local, push Fast admin machine; matching arch
Build on target Weak laptop; strong nodes
Remote builder + cache Day 54 + 53 machinery

Combine with Day 53 caches so N nodes do not compile N times.

Theory 6 — Evaluation cost and hive growth

Large hives evaluate all nodes for some operations. Habits:

Habit Why
Thin defaults Shared eval cost is real
Avoid importing desktop on servers Closure + eval bloat
colmena eval / dry-run Catch errors before blast
Split lab vs prod flakes if needed Cognitive firewall

Worked example — two-node visible drift control

# hosts/edge/default.nix
{ pkgs, ... }:
{
  environment.etc."role".text = "edge\n";
  networking.firewall.allowedTCPPorts = [ 80 ];
  # services.caddy… optional
}
# hosts/core/default.nix
{
  environment.etc."role".text = "core\n";
}
colmena apply --on edge
ssh edge cat /etc/role

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day60

Step 1 — Second node

Provision second VM or treat laptop as allowLocalDeployment node + one remote.

Step 2 — Express hive

Move shared SSH/firewall defaults into defaults; keep host-only bits in nodes. Commit lockfile.

Step 3 — First full apply

colmena apply

Capture output; note parallel vs serial behavior.

Step 4 — Tag apply

Tag one host canary; apply only canary; then the rest.

Step 5 — Parallel observation

Apply both; note wall clock vs serial deploy-rs mental model.

Step 6 — Intentional drift fix

SSH to a node; make a manual /etc change on a managed file; re-apply Colmena; confirm declarative truth wins (for managed paths).

Step 7 — Tool choice note

FLEET-TOOLING.md: when to use Colmena vs deploy-rs in your monorepo; one default for Capstone A.

Step 8 — Optional eval-only

colmena eval -E '{ nodes, ... }: builtins.attrNames nodes'

(Adjust to supported eval interface in your Colmena version.)

Step 9 — Failure notes

Document one failed apply (intentional or accidental) and recovery. Link Day 89 rollback thinking.

Step 10 — Inventory of tags

# after hive is stable, write TAGS.md
Tag Hosts Meaning Who may apply
web edge public edge role on-call
core core internal on-call
canary edge first wave you
lab * non-prod anyone on team

Step 11 — Shared defaults audit

List everything in defaults. For each item, mark must-share vs accidentally global (e.g. a desktop package that bloated servers). Move accidents to role modules (Day 61).

Parallelism and blast radius

colmena apply --on @web
        │
        ├─ edge-1  ─┐
        ├─ edge-2  ─┼─ simultaneous pressure on cache/builder
        └─ edge-3  ─┘
Control Use
Tags / canary Limit who changes first
Cache warm Avoid N compile storms
Console access Survive firewall mistakes
One tool default deploy-rs or Colmena per host class

Never demo “apply all prod” on day one of Colmena.

Common gotchas

Symptom / mistake What to do
Hive not discovered Flake output name colmena / colmenaHive per version docs
Wrong nixpkgs system meta.nixpkgs system must match builds
Tags typo List tags; dry-run selection
Local node confusion allowLocalDeployment / apply-local
Double firewall lockout Console access
Gigantic simultaneous compiles Cache + buildOnTarget policy
stateVersion missing on defaults Set "26.05" deliberately
Mixing deploy-rs + Colmena on same host ad hoc One tool per host class

Checkpoint

  • Hive evaluates
  • Two nodes deployed (or local+remote)
  • Tag-filtered apply works
  • Drift overwritten by apply
  • Tooling choice written

Commit

git add .
git commit -m "day60: colmena hive and tagged apply"

Write three personal gotchas before continuing.

Tomorrow

Day 61 — Shared modules / roles: stop copy-pasting host configs.

Day 61 — Shared modules & roles

Stage VI · ~4h
Goal: Refactor fleet config into roles and profiles so hosts become thin compositions—no copy-paste SSH/firewall/observability blocks.

Why this day exists

Two Colmena nodes already tempt duplication. Ten nodes without roles guarantee drift: one host missing a hardening line is an incident. Modules are the compression algorithm of NixOS fleets.

Theory 1 — Layering

nixpkgs modules
  └─ your library modules (roles/, profiles/, services/)
        └─ host modules (hosts/edge)
              └─ secrets / hardware / disko (host-specific)
Layer Contains Avoid
Role web, db, observer — purpose Hardware paths
Profile baseline, hardened, workstation One-off hacks
Host hostname, addresses, disk, keys Recreating baseline
Service module Single app options Global side effects silent

Theory 2 — Role module pattern

# modules/roles/web.nix
{ config, lib, pkgs, ... }:
let
  cfg = config.roles.web;
in {
  options.roles.web = {
    enable = lib.mkEnableOption "web edge role";
    domain = lib.mkOption {
      type = lib.types.str;
      example = "service.example.com";
    };
  };

  config = lib.mkIf cfg.enable {
    networking.firewall.allowedTCPPorts = [ 80 443 ];
    services.caddy.enable = true;
    services.caddy.virtualHosts.${cfg.domain}.extraConfig = ''
      respond "ok"
    '';
    # or nginx—match Stage IV choice
  };
}
# hosts/edge/default.nix
{
  imports = [
    ../../modules/profiles/baseline.nix
    ../../modules/roles/web.nix
  ];
  roles.web = {
    enable = true;
    domain = "edge.lab.internal";
  };
  networking.hostName = "edge";
}

Theory 3 — Baseline profile

# modules/profiles/baseline.nix
{ lib, pkgs, ... }:
{
  time.timeZone = "UTC";
  i18n.defaultLocale = "en_US.UTF-8";
  services.openssh = {
    enable = true;
    settings = {
      PasswordAuthentication = false;
      PermitRootLogin = "prohibit-password";
    };
  };
  environment.systemPackages = with pkgs; [ vim curl jq htop ];
  nix.settings = {
    experimental-features = [ "nix-command" "flakes" ];
    auto-optimise-store = true;
  };
  networking.firewall.enable = true;
}
# modules/profiles/hardened.nix
{
  imports = [ ./baseline.nix ];
  # Day 36 patterns: sysctl, auditd optional, disable unused
  # Keep reversible for lab
}

Theory 4 — specialArgs and inventory

# flake / colmena meta
specialArgs = {
  inventory = {
    domain = "lab.internal";
    adminKeys = [ "ssh-ed25519 AAAA…" ];
  };
};
# modules/profiles/baseline.nix
{ inventory, pkgs, ... }:
{
  users.users.root.openssh.authorizedKeys.keys = inventory.adminKeys;
}

Hosts remain data; modules remain behavior.

Theory 5 — Assertions and defaults

{ config, lib, ... }:
{
  config = lib.mkIf config.roles.web.enable {
    assertions = [
      {
        assertion = config.roles.web.domain != "";
        message = "roles.web.domain must be set";
      }
    ];
  };
}
services.openssh.settings.PasswordAuthentication = lib.mkDefault false;
# host may mkForce true only with comment of shame

Worked example — directory layout

flake.nix
modules/
  profiles/baseline.nix
  profiles/hardened.nix
  roles/web.nix
  roles/devbox.nix
  services/myapp.nix
hosts/
  edge/default.nix
  core/default.nix
  edge/disko.nix
  core/disko.nix

Colmena defaults = { imports = [ ./modules/profiles/baseline.nix ]; … } or import baseline per host—pick one to avoid double-import.

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day61

Step 1 — Inventory duplication

rg -n "openssh|PasswordAuthentication|timeZone" hosts modules and list triplicates.

Step 2 — Extract profiles/baseline.nix

Remove duplicated chunks from hosts; rebuild/deploy still works.

Step 3 — Create one role

roles.web or roles.devbox with mkEnableOption. Enable on one host only.

Step 4 — Second host consumes baseline only

Prove role is optional.

Step 5 — Assertion

Add assertion; break it on purpose; fix.

Step 6 — mkDefault vs host override

Show host override of a baseline package list or setting with comment.

Step 7 — Document module contract

Each role file starts with a 5-line comment: purpose, ports, state dirs, secrets needed, who may enable.

Step 8 — Deploy

Colmena/deploy-rs apply; no behavior regression.

Theory 6 — Host vs role vs profile (decision tree)

Is it true for every machine in the fleet?
  yes → modules/baseline or common/*
Is it true for a class (web, db, laptop)?
  yes → modules/roles/<role>.nix
Is it true for one hostname only?
  yes → hosts/<name>/*.nix

Mis-layering causes “why did the DB host get the laptop touchpad module?”

Theory 7 — Import graphs and cycles

# bad: a imports b imports a

Prefer a DAG: flake → host → roles → baseline. Share via imports = [ … ], not mutual imports. If you need shared options, put option declarations in a thin options.nix imported by both.

Worked example — role enable pattern

# modules/roles/web.nix
{ config, lib, ... }:
let cfg = config.myOrg.roles.web; in {
  options.myOrg.roles.web.enable = lib.mkEnableOption "web edge role";
  config = lib.mkIf cfg.enable {
    imports = [ ../services/proxy.nix ];
    # myOrg.roles.web specific asserts
  };
}

Hosts only flip myOrg.roles.web.enable = true.

Lab — draw the import DAG

ASCII in docs/modules-dag.md for your real flake. If you cannot draw it, simplify.

Lab — extract one host-only blob

Find 15+ lines only relevant to one host still sitting in common modules. Move to hosts/<name>/. Rebuild both mental models: common shrinks, host clarifies.

Theory 8 — Naming conventions for shared options

Pattern Example
Org prefix myOrg.roles.web.enable
Feature modules myOrg.services.proxy.enable
Avoid bare enableWeb at top level

Consistent prefixes make nixos-option and search usable across hosts.

Common gotchas

Symptom / mistake What to do
Double imports Import baseline once
Role enables heavy services globally Gate with mkIf cfg.enable
Host still 400 lines Extract again; hardware alone may stay long
specialArgs missing Thread through nixosSystem / colmena meta
Priority wars Learn mkForce / mkDefault; document
Circular imports Unidirectional: hosts → roles → not back

Checkpoint

  • Baseline profile shared
  • At least one role module with options
  • Hosts are thin compositions
  • Assertion demonstrated
  • Module contracts commented

Commit

git add .
git commit -m "day61: roles and shared baseline modules"

Write three personal gotchas before continuing.

Tomorrow

Day 62 — Images & nixos-generators: cloud images and custom ISOs from the same modules.

Day 62 — Images & nixos-generators

Stage VI · ~4h
Goal: Build a bootable artifact (cloud image, ISO, or VM format) from your flake modules using nixos-generators (and related patterns), then boot it once and prove it is the same module graph as a live host—not a snowflake fork.

Note

Images are first-class deploy targets in Stage VI, not a side quest. The win is one module tree → install path and golden image path. NixOS 26.05 pin stays consistent with the rest of this volume.

Why this day exists

Not every node is nixos-install by hand. Cloud and edge want images: qcow2, AMIs (via pipelines), raw disks, installer ISOs. Generating images from the same modules as deploy targets closes a huge consistency gap—and catches “works on my VM but not in the AMI” earlier.

Without an image story you either:

  • Hand-maintain cloud user-data that reimplements half of NixOS, or
  • Drift image hosts away from Colmena/deploy-rs nodes until nobody trusts either.

Theory 1 — Image = NixOS config + format

same modules as hosts/edge
        │
        ▼
nixos-generators / make-disk-image / sd-image
        │
        ▼
artifact (iso, qcow2, raw, gce, azure, …)
Format (examples) Use
iso Installer / live recovery
qcow / qcow2 / vmware / virtualbox Hypervisors
amazon / gce / azure Cloud (tooling evolves)
sd-card / raw SBC / dd

Check the current nixos-generators format list for your pin—names and defaults change. Prefer the project README over stale blog posts.

Mental model

An image build is still a NixOS evaluation: modules merge, activation scripts exist, store paths land on a disk layout the format owns. You are not “exporting files”; you are realizing a system derivation into a disk picture.

Theory 2 — flake + nixos-generators

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
  inputs.nixos-generators.url = "github:nix-community/nixos-generators";
  inputs.nixos-generators.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { self, nixpkgs, nixos-generators, ... }: {
    packages.x86_64-linux.edge-qcow = nixos-generators.nixosGenerate {
      system = "x86_64-linux";
      format = "qcow";
      modules = [
        ./modules/profiles/baseline.nix
        ./hosts/edge/image.nix
      ];
    };

    packages.x86_64-linux.lab-iso = nixos-generators.nixosGenerate {
      system = "x86_64-linux";
      format = "iso";
      modules = [ ./hosts/lab-iso.nix ];
    };
  };
}
nix build .#edge-qcow -L
ls -lh result*
# often result is a symlink to the image file or a directory containing it

follows discipline

Always pin generators’ nixpkgs to your flake’s nixpkgs. Mixed pins produce “image builds but host rebuilds something else” headaches that look like ghosts.

Theory 3 — Image-specific config

Images need first-boot realities: growpart, cloud-init or pure Nix alternatives, SSH keys baked or injected, serial console, predictable hostnames.

# hosts/edge/image.nix
{ modulesPath, pkgs, ... }:
{
  imports = [
    # sometimes:
    # (modulesPath + "/profiles/qemu-guest.nix")
  ];

  services.openssh.enable = true;
  users.users.root.openssh.authorizedKeys.keys = [
    "ssh-ed25519 AAAA… image-lab"
  ];

  # Disallow password auth even on images
  services.openssh.settings.PasswordAuthentication = false;

  networking.hostName = "edge-image";
  system.stateVersion = "26.05";

  # Smaller images: careful with documentation packages
  documentation.nixos.enable = false;
  documentation.man.enable = false;
}
Concern Image habit
SSH access Baked lab key or cloud metadata; never password-only
Disk grow Guest agents / cloud-init / nixos grow modules
Console Serial-friendly; qemu-guest profiles when KVM
Hostname Static lab name vs cloud-init rewrite—pick deliberately
Docs packages Off for edge images; on for installer ISOs if you want man pages

Cloud-init vs pure NixOS images is a design choice; lab can use baked keys and skip cloud-init entirely.

Theory 4 — Consistency with deploy hosts

Risk Mitigation
Image modules diverge from live host Share roles/* and profiles/*
Disko vs image partitioning Images often use generator partitioning; Disko for bare metal
Secrets in images Never bake production private keys into public artifacts
Huge closure Slim profiles for images
Different stateVersion One policy; document exceptions

Shared vs image-only modules

modules/profiles/baseline.nix   ← both image + live
modules/roles/edge.nix          ← both
hosts/edge/hardware.nix         ← live only (Disko, UUID)
hosts/edge/image.nix            ← image only (guest, keys, slim)

If baseline grows a desktop stack “because laptop,” your AMI explodes. Keep baseline honest.

Theory 5 — Alternatives awareness

Tool Notes
nixos-generators Friendly multi-format entry point
make-disk-image internals Under many formats; lower-level
nixos-rebuild build-vm Quick smoke VM, not always production image
Packer + Nix Cross-cloud pipelines
Discovered AMI publish Separate CI concern (Day 66+)
nixos-generators + upload scripts Your glue, not the generator’s job

Theory 6 — Size, caches, and CI

Image builds are full system closures. First build is slow; second should hit substitutes.

Lever Effect
Binary cache with your system paths Huge wall-clock win
documentation.*.enable = false Noticeable MB
Drop unused locales / firmware Context-dependent
Separate “lab fat” vs “edge slim” profiles Avoid one profile for all
CI artifact store Keep last N images; GC ruthlessly

Record image size in notes every rebuild during the lab—train the instinct that modules have mass.

Theory 7 — Security of publishable artifacts

Anything you upload to object storage is world-readable unless you lock it down.

# before publish: secret hygiene on image module path
rg -n "PRIVATE|AGE-SECRET|BEGIN OPENSSH PRIVATE|sk-" hosts modules || true
OK to bake Never bake
Lab public SSH keys Age/sops private keys
Hardened sshd defaults Cloud API tokens
Public CA certs Production host private keys
Non-secret config networking.wireless PSKs

Production: inject secrets at first deploy (Day 34–35 patterns) or via cloud metadata your threat model accepts.

Worked example — boot qcow with QEMU

nix build .#edge-qcow -L
IMG=$(readlink -f result)
# if result is a dir, find the qcow2:
find result -name '*.qcow2' -o -name '*.img'

qemu-system-x86_64 \
  -enable-kvm \
  -m 2048 \
  -drive file="$IMG",if=virtio \
  -nic user,hostfwd=tcp::2222-:22 \
  -nographic
# ssh -p 2222 root@127.0.0.1

Adjust for copy-on-write overlay so you do not dirty the store image:

qemu-img create -f qcow2 -b "$IMG" -F qcow2 /tmp/edge-overlay.qcow2
qemu-system-x86_64 \
  -enable-kvm -m 2048 \
  -drive file=/tmp/edge-overlay.qcow2,if=virtio \
  -nic user,hostfwd=tcp::2222-:22 \
  -nographic

Proof of shared modules

# after SSH in
cat /etc/os-release
nixos-version
# if you bake a marker:
# cat /etc/day62-marker

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day62

Step 1 — Add generator input

Pin; follows nixpkgs; commit flake.lock.

Step 2 — Define slim image module

Reuse baseline; bake lab SSH key only. Set system.stateVersion = "26.05".

Step 3 — Build one format

Prefer qcow/iso based on what you can boot easily.

nix build .#edge-qcow -L
du -h $(readlink -f result)

Step 4 — Boot once

QEMU, VirtualBox, or cloud upload. Capture login proof (SSH or console) in notes.

Step 5 — Compare modules

Table: image module vs live hosts/edge—shared vs divergent. Fix any accidental fork.

Step 6 — Size pass

Disable docs; remove heavy packages; rebuild; note delta in MB and wall time.

Step 7 — Secret hygiene check

rg on image path—must be clean for publishable images. Record command + empty result.

Step 8 — Optional second format

ISO installer with same baseline for recovery story (feeds Day 88 thinking).

Step 9 — Write IMAGE.md

Field Value
Format
Flake attr
Modules shared
Image-only modules
Size
Boot proof path to log
Publish? yes/no + why

Common gotchas

Symptom / mistake What to do
Wrong format name Check generator docs for pin
Cannot SSH keys, firewall, grow services, serial console
Image boots black screen serial console -nographic; qemu-guest modules
Mutating store image Use overlay qcow
20GB “minimal” Slim profile; no chrome; no full texlive
Secrets baked Rebuild without; use cloud metadata / sops on first deploy
Generators nixpkgs drift inputs.nixos-generators.inputs.nixpkgs.follows
Arch mismatch (build aarch on x86) Cross or native builder (Day 54)
ISO vs installed system confusion ISO boots live env; install path still needs nixos-install story

Checkpoint

  • Generator produces an artifact
  • Artifact boots
  • SSH or console access works
  • Shared modules reused (table in IMAGE.md)
  • Secret hygiene checked
  • Size recorded before/after slim pass

Commit

git add .
git commit -m "day62: nixos-generators image boot proof"

Write three personal gotchas before continuing.

Tomorrow

Day 63 — Terraform boundary: TF provisions, Nix configures (sketch is enough).