Day 60 — Colmena

Updated

July 30, 2026

Day 60 — Colmena

Stage VI · ~4h
Goal: Express a small hive with Colmena, deploy at least two nodes (one may be local), use tags, and know when Colmena wins over deploy-rs for fleet shape. Hosts pin NixOS 26.05.

Note

deploy-rs is excellent node/profile oriented glue. Colmena shines when you think in hives: many hosts, tags (@web, @db), parallel apply, and a flake hive attribute. Real orgs often evaluate both; leave Day 70 with a deliberate choice, not two half-wired tools forever.

Why this day exists

One remote deploy proves the pipe. Fleets need selection, defaults, and blast-radius control. Colmena’s mental model maps cleanly onto roles you will build on Day 61.

Theory 1 — Hive mental model

meta (nixpkgs, specialArgs)
  nodes
    host-a → NixOS module stack
    host-b → …
  defaults → shared modules
Concept Role
meta.nixpkgs Package set for evaluation
defaults Applied to every node
nodes.<name> Per-host config + deployment keys
tags Selection subsets for apply
specialArgs Inject libs/secrets helpers carefully

Theory 2 — Flake-based Colmena (modern)

{
  description = "Day 60 colmena hive";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

  outputs = { self, nixpkgs, ... }: {
    colmena = {
      meta = {
        nixpkgs = import nixpkgs { system = "x86_64-linux"; };
        # specialArgs = { inherit something; };
      };

      defaults = { pkgs, ... }: {
        time.timeZone = "UTC";
        services.openssh.enable = true;
        environment.systemPackages = [ pkgs.vim pkgs.curl ];
        system.stateVersion = "26.05";
      };

      edge = { name, nodes, pkgs, ... }: {
        deployment = {
          targetHost = "edge.lab.internal";
          targetUser = "root";
          tags = [ "edge" "web" ];
        };
        networking.hostName = "edge";
        imports = [ ./hosts/edge ];
      };

      core = {
        deployment = {
          targetHost = "core.lab.internal";
          tags = [ "core" ];
          # allowLocalDeployment = true; # when node is the machine itself
        };
        networking.hostName = "core";
        imports = [ ./hosts/core ];
      };
    };
  };
}
nix shell nixpkgs#colmena -c colmena apply
nix shell nixpkgs#colmena -c colmena apply --on @web
nix shell nixpkgs#colmena -c colmena apply-local --sudo   # local node patterns

CLI details (colmena apply --on host)—run colmena --help for your version. Some pins expose colmenaHive—follow current Colmena flake docs if colmena attr alone is insufficient.

Theory 3 — Tags and selection

colmena apply --on edge
colmena apply --on @web
colmena apply --on @web,@db
# exclusions / filters: check help for your version
Habit Why
Tag by role not only name Scale
Tag by risk (canary) Staged rollouts
Never tag prod+lab together casually Blast radius
Tag by arch if mixed Avoid wrong builds

Canary pattern

1. apply --on @canary
2. health probes
3. apply --on @web   # rest of role

Theory 4 — deploy-rs vs Colmena (honest)

Dimension deploy-rs Colmena
Flake-native profiles Strong Strong hive attr
Multi-host UX Nodes map Hive + tags first-class
Magic rollback Notable feature Different tooling
Learning curve Lower for 1 node Pays off at N>2
Ecosystem Serokell zhaofengli / community

House rule suggestion: deploy-rs for 1–2, Colmena when tags appear—or standardize on one to reduce cognitive load. Pick deliberately by Day 70 and document in FLEET-TOOLING.md.

Theory 5 — Build & push options

Colmena can build locally or on targets depending on deployment.buildOnTarget and related options:

deployment = {
  targetHost = "edge.lab.internal";
  buildOnTarget = false;
  # tags = [ "edge" ];
};
Mode When
Build local, push Fast admin machine; matching arch
Build on target Weak laptop; strong nodes
Remote builder + cache Day 54 + 53 machinery

Combine with Day 53 caches so N nodes do not compile N times.

Theory 6 — Evaluation cost and hive growth

Large hives evaluate all nodes for some operations. Habits:

Habit Why
Thin defaults Shared eval cost is real
Avoid importing desktop on servers Closure + eval bloat
colmena eval / dry-run Catch errors before blast
Split lab vs prod flakes if needed Cognitive firewall

Worked example — two-node visible drift control

# hosts/edge/default.nix
{ pkgs, ... }:
{
  environment.etc."role".text = "edge\n";
  networking.firewall.allowedTCPPorts = [ 80 ];
  # services.caddy… optional
}
# hosts/core/default.nix
{
  environment.etc."role".text = "core\n";
}
colmena apply --on edge
ssh edge cat /etc/role

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day60

Step 1 — Second node

Provision second VM or treat laptop as allowLocalDeployment node + one remote.

Step 2 — Express hive

Move shared SSH/firewall defaults into defaults; keep host-only bits in nodes. Commit lockfile.

Step 3 — First full apply

colmena apply

Capture output; note parallel vs serial behavior.

Step 4 — Tag apply

Tag one host canary; apply only canary; then the rest.

Step 5 — Parallel observation

Apply both; note wall clock vs serial deploy-rs mental model.

Step 6 — Intentional drift fix

SSH to a node; make a manual /etc change on a managed file; re-apply Colmena; confirm declarative truth wins (for managed paths).

Step 7 — Tool choice note

FLEET-TOOLING.md: when to use Colmena vs deploy-rs in your monorepo; one default for Capstone A.

Step 8 — Optional eval-only

colmena eval -E '{ nodes, ... }: builtins.attrNames nodes'

(Adjust to supported eval interface in your Colmena version.)

Step 9 — Failure notes

Document one failed apply (intentional or accidental) and recovery. Link Day 89 rollback thinking.

Step 10 — Inventory of tags

# after hive is stable, write TAGS.md
Tag Hosts Meaning Who may apply
web edge public edge role on-call
core core internal on-call
canary edge first wave you
lab * non-prod anyone on team

Step 11 — Shared defaults audit

List everything in defaults. For each item, mark must-share vs accidentally global (e.g. a desktop package that bloated servers). Move accidents to role modules (Day 61).

Parallelism and blast radius

colmena apply --on @web
        │
        ├─ edge-1  ─┐
        ├─ edge-2  ─┼─ simultaneous pressure on cache/builder
        └─ edge-3  ─┘
Control Use
Tags / canary Limit who changes first
Cache warm Avoid N compile storms
Console access Survive firewall mistakes
One tool default deploy-rs or Colmena per host class

Never demo “apply all prod” on day one of Colmena.

Common gotchas

Symptom / mistake What to do
Hive not discovered Flake output name colmena / colmenaHive per version docs
Wrong nixpkgs system meta.nixpkgs system must match builds
Tags typo List tags; dry-run selection
Local node confusion allowLocalDeployment / apply-local
Double firewall lockout Console access
Gigantic simultaneous compiles Cache + buildOnTarget policy
stateVersion missing on defaults Set "26.05" deliberately
Mixing deploy-rs + Colmena on same host ad hoc One tool per host class

Checkpoint

  • Hive evaluates
  • Two nodes deployed (or local+remote)
  • Tag-filtered apply works
  • Drift overwritten by apply
  • Tooling choice written

Commit

git add .
git commit -m "day60: colmena hive and tagged apply"

Write three personal gotchas before continuing.

Tomorrow

Day 61 — Shared modules / roles: stop copy-pasting host configs.