Critical vs Non-Critical Hives

Updated

September 12, 2026

Critical vs Non-Critical Hives

NixOS/infra splits builders / hydra / ofborg from non-critical-infra/ (community hosts, staging hydra, a second Colmena hive, its own .sops.yaml and devShells.non-critical-infra). The boring default is: two Colmena tags (or two flakes) so a botched app host cannot share sops keys or apply waves with gw-01.

The capstone hive is one file. This lab splits it the way a real org does once “the website” and “the CA / builders” must not share a blast radius.

Mental model

flake.nix
  colmena.meta
  defaults            →  ssh keys, firewall, stateVersion
  @core  gw-01 db-01 ops-01     secrets/core/
  @apps  app-01 app-02          secrets/apps/

NixOS/infra goes further: a directory with its own sops and colmena.sh. Desk: one flake, two tags, two secret trees. Two flakes when the team that owns apps must not eval the core modules.

Their non-critical README: secrets are need-to-have; add your key on a PR; someone who already has access runs updatekeys. That is the onboarding ritual. Not a Slack DM of key.txt.

Worked examples

Case 1: Tags on the existing hive

Save as flake.nix:

# flake.nix
{
  description = "Desk hive with core vs apps";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

  outputs = { self, nixpkgs }: {
    colmena = {
      meta.nixpkgs = import nixpkgs { system = "x86_64-linux"; };

      defaults = { ... }: {
        system.stateVersion = "26.05";
        deployment.targetUser = "root";
        imports = [ ./modules/users-from-keys.nix ];
      };

      gw-01 = {
        deployment.targetHost = "203.0.113.10";
        deployment.tags = [ "core" "edge" ];
        imports = [ ./hosts/gw-01.nix ];
      };
      db-01 = {
        deployment.targetHost = "203.0.113.21";
        deployment.tags = [ "core" "db" ];
        imports = [ ./hosts/db-01.nix ];
      };
      ops-01 = {
        deployment.targetHost = "203.0.113.30";
        deployment.tags = [ "core" "ops" ];
        imports = [ ./hosts/ops-01.nix ];
      };
      app-01 = {
        deployment.targetHost = "203.0.113.11";
        deployment.tags = [ "apps" ];
        imports = [ ./hosts/app-01.nix ];
      };
      app-02 = {
        deployment.targetHost = "203.0.113.12";
        deployment.tags = [ "apps" ];
        imports = [ ./hosts/app-02.nix ];
      };
    };
  };
}
nix shell nixpkgs#colmena --command colmena eval -E '{ nodes }: builtins.attrNames nodes'

Five names. Apply waves:

colmena apply --on @core --dry-run
colmena apply --on @apps --dry-run

Never --on @all for a kernel bump on @core. Apps can fan out. db-01 stays serial inside @core (capstone DR).

Case 2: Two sops trees

secrets/
  core/
    luks.yaml          # recipients: desk-core only
    restic.yaml
  apps/
    db-password.yaml   # recipients: desk-core + app runtime? no — hosts decrypt

App hosts decrypt db-password with the host age key, not alice’s laptop. Alice’s key is for editing the YAML. That is the same split as NixOS/infra: humans in .sops.yaml, machines as extra recipients.

Save as modules/sops-core.nix:

# modules/sops-core.nix
{ config, ... }:
{
  sops.defaultSopsFile = ../secrets/core/luks.yaml;
  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
}

Save as modules/sops-apps.nix:

# modules/sops-apps.nix
{
  sops.defaultSopsFile = ../secrets/apps/db-password.yaml;
  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
}

Import sops-core.nix only on @core. Import sops-apps.nix on @apps (and db-01 if the DB password lives there). An app box that can decrypt LUKS material is the wrong blast radius.

Case 3: Named devShells, not one kitchen sink

NixOS/infra exposes devShells.builders (agenix) and devShells.non-critical-infra (colmena, sops, ssh-to-age) and devShells.terraform (OpenTofu withPlugins). Desk:

# flake.nix fragment
{
  devShells.x86_64-linux.core = pkgs.mkShell {
    packages = [ pkgs.colmena pkgs.sops pkgs.ssh-to-age ];
  };
  devShells.x86_64-linux.apps = pkgs.mkShell {
    packages = [ pkgs.colmena pkgs.sops ];
  };
  devShells.x86_64-linux.tf = pkgs.mkShell {
    packages = [
      (pkgs.opentofu.withPlugins (p: [ p.aws p.local ]))
    ];
  };
}

The app team’s direnv is use flake .#apps. They do not get the OpenTofu wrapper or core sops files in $PWD if those live in another worktree. Need-to-have.

Case 4: Staging is a host, not a branch of secrets

Their Colmena list includes staging-hydra.targetHost = "staging-hydra.nixos.org" in the non-critical hive. A hydra change is exercised there before the production hydra input moves.

Desk equivalent:

# hosts/staging-app.nix
{
  imports = [ ./app-01.nix ];
  networking.hostName = "staging-app";
  # acmeCA staging; smaller RAM; same modules
}
colmena apply --on staging-app
curl -sf https://staging.api.desk.internal/healthz
colmena apply --on @apps

Do not “staging” by decrypting prod sops on a laptop and hoping. Same modules, different host, different ACME, different tag.

Case 5: colmena.sh is a one-liner, not a platform

NixOS/infra’s non-critical-infra/colmena.sh is a thin wrapper so humans do not forget --flake. Desk:

# colmena-apps.sh
#!/usr/bin/env bash
set -euo pipefail
exec nix shell nixpkgs#colmena --command colmena "$@" --on @apps
chmod +x colmena-apps.sh
./colmena-apps.sh apply --dry-run

No extra DSL. The wrapper only restricts the tag. A wrapper that defaults to @all is worse than none.

The trap

The trap is one sops file for LUKS, restic, and the app DB password. A contractor who needs desk-api then decrypts disk keys. Split trees. Split tags.

The other trap is applying @core and @apps in one command because “it’s faster.” Core is serial and canary. Apps are tagged.

The boring rule

  • @core vs @apps (or two flakes). No @all for kernels.
  • secrets/core/ vs secrets/apps/. Host age keys decrypt; humans rekey.
  • Staging is a host with the same modules.
  • Named devShells per team. Need-to-have sops.
  • A 10-line colmena-*.sh that pins the tag is enough.

Try this

  1. Add tags to the capstone hive; colmena eval and list @core vs @apps.
  2. Move a dummy secret into secrets/apps/ and confirm a --dry-run on @core does not need it.
  3. Write colmena-apps.sh; run --dry-run; confirm it refuses to take --on gw-01 or wrap so that --on is ignored.
  4. Add staging-app as a sixth host; apply it before @apps.