Remote builders

Updated

July 30, 2026

Remote builders

Goal: Offload realization to a remote Nix machine (Linux builder VM is ideal), understand builders / build machine config, and know when remote build beats “just use a cache.” Align lab with nixos-26.05 clients and builders when possible.

Note

Remote builders move work; binary caches move results. Confusing them wastes weeks. You usually want both: build once on a beefy box, push to Cachix/Attic, everyone substitutes.

Why it matters

Laptops should not compile Chromium. CI runners should not all rebuild the same uncached leaf. Remote builders are how Darwin talks to Linux, how small VMs ship large closures, and how fleets keep eval local while realization lives elsewhere.

Theory 1 — Cache vs remote builder

	Binary cache	Remote builder
Needs existing path?	Yes	No
Uses your flake inputs	Downloads NARs	Evaluates plan, builds drv remotely
Good for	Repeat installs	First build, exotic systems
Trust	Signing keys	SSH + Nix auth + who runs builder
Network pattern	Pull results	Upload inputs + download outputs

[dev] --build--> [builder] --push--> [cache] --substitute--> [fleet]

Theory 2 — Configuration surfaces (2026)

NixOS (`nix.buildMachines`)

{
  nix.distributedBuilds = true;
  nix.buildMachines = [
    {
      hostName = "builder.lab.internal";
      system = "x86_64-linux";
      protocol = "ssh-ng"; # preferred modern protocol when available
      sshUser = "nixbuilder";
      sshKey = "/root/.ssh/id_build"; # or agent patterns
      maxJobs = 8;
      speedFactor = 4;
      supportedFeatures = [ "nixos-test" "big-parallel" "kvm" ];
      mandatoryFeatures = [ ];
    }
  ];
  nix.settings = {
    builders-use-substitutes = true; # remote may fetch deps itself
  };
}

Ad-hoc one-shot

nix build .#pkg --builders 'ssh-ng://nixbuilder@builder.lab.internal x86_64-linux' --max-jobs 0
# classic:
# --option builders 'ssh://…'

Features

Feature	Typical meaning
`big-parallel`	Allow heavy parallel builds
`kvm`	Nested VM tests (`nixosTest`)
`nixos-test`	OS tests scheduling
`benchmark`	Specialization (rare)

If a drv requires kvm and the builder lacks it, scheduling skips that machine—often looking like “mysterious local build” or hard failure depending on flags.

Theory 3 — SSH and store talk

Remote build roughly:

Local eval (usually) produces derivations
Upload input paths the remote lacks
Remote builds
Download result paths back (unless you only needed remote side effects)

# Test SSH as the Nix daemon user (often root on NixOS)
sudo ssh -i /root/.ssh/id_build nixbuilder@builder.lab.internal nix-store --version

Daemon context: On multi-user Nix, root (nix-daemon) must reach the builder—not only your user ssh config—unless using remote build modes that run as you (know your setup).

Check	Why
Non-interactive SSH	No password prompts in daemon
`known_hosts` stable	Automation vs MITM prompts
Dedicated build key	Least privilege vs your personal key
Builder disk free	Full disk mid-build is common
Builder Nix version band	Avoid obscure protocol skew

Theory 4 — Cross and multi-system

Goal	Approach
Build `aarch64-linux` from `x86_64` laptop	aarch64 builder or qemu-binfmt (slower)
Build Linux from Darwin	Linux remote builder almost always
Same-arch offload only	Simplest lab
`nixosTest` needing kvm	Builder with kvm feature + hardware

# Prefer real hardware/VM matching system
system = "aarch64-linux";

Honest lab goal: one successful remote same-arch build. Cross is stretch, not gate.

Theory 5 — Security notes

Risk	Mitigation
Builder can see sources	Treat builder as trusted environment
SSH key sprawl	Dedicated build key; restrict `authorized_keys`
Untrusted builder returns malicious paths	Only use builders you control; CI policies
World-open builder	VPN/Tailscale; no public SSH
Builder as root playground	Separate user; trusted-users carefully

Remote builders are not a sandbox that makes untrusted code safe. A compromised builder is a supply-chain incident.

Decision card — do I need a builder?

Situation	Prefer
Package already on cache	Substituter only
First build of large leaf on weak laptop	Remote builder
Darwin needing Linux closure	Linux builder
CI rebuilds identical drv every job	Push cache after one builder run
Untrusted contributor PR	Do not give them your builder

Theory 6 — Scheduling knobs that matter

Knob	Effect
`max-jobs 0` (client)	Force offload when supported
`speedFactor`	Prefer faster builders
`builders-use-substitutes`	Remote fetches deps itself
`supportedFeatures`	Eligibility filter
Local `max-jobs` > 0	Local may steal work

nix build .#cooltool -L \
  --builders "ssh-ng://nixbuilder@builder x86_64-linux - 8 1 big-parallel" \
  --max-jobs 0

Upload storms

If every client uploads the world to the builder, enable builders-use-substitutes and share the same binary cache with the builder so it pulls deps itself.

Theory 7 — Builder host minimal config

# on builder
users.users.nixbuilder = {
  isNormalUser = true;
  # group membership per current wiki; keep tight
};
nix.settings.trusted-users = [ "root" "nixbuilder" ];
# SSH key auth only; open firewall only to admin net
# Ensure large /nix disk; enable flakes

Exact store write permissions and nix.sshServe patterns vary by version—verify against 26.05 docs when wiring production.

Topology diagram (keep in notes)

[dev laptop / host A] --ssh-ng--> [builder VM B: NixOS 26.05, multi-user Nix]
        |                                |
        +--------> Cachix/Attic <--------+  (optional push from B or A)

Worked example — two-VM lab checklist

1. Inventory local capacity (max-jobs, disk)
2. Provision builder VM with free /nix space
3. Create dedicated SSH key; install pubkey for nixbuilder
4. Prove non-interactive SSH as root (daemon) and as user
5. One-shot remote build with --max-jobs 0
6. Capture log evidence of remote host
7. Persist nix.buildMachines
8. Optional: push results to cache; rebuild substitute-only
9. Failure injection: stop sshd; observe client error; restore

Prove which machine built

nix path-info -Sh .#cooltool
# keep a snippet of the build log that mentions the remote host

If you cannot show evidence of remote work, you may have silently built local—re-check --max-jobs 0 and builder SSH as daemon user.

Lab — multi-step

Suggested workspace: ~/lab/nixos/packaging/builders

Step 1 — Inventory local capacity

nix config show 2>/dev/null | rg 'max-jobs|cores|builders' \
  || nix show-config | rg 'max-jobs|cores|builders'
nproc
df -h /nix

Step 2 — Provision builder

Options:

Second NixOS VM on LAN/VPN
Same machine via ssh://localhost (limited learning value but tests plumbing)
Cloud NixOS box

Install Nix multi-user; enable flakes; ensure free disk (tens of GB).

Step 3 — SSH from daemon-relevant user

ssh -i ~/.ssh/id_build nixbuilder@builder 'nix-store --version'
sudo ssh -i /root/.ssh/id_build nixbuilder@builder 'nix-store --version'

Document which user the daemon uses on your OS.

Step 4 — One-shot remote build

nix build .#cooltool -L \
  --builders "ssh-ng://nixbuilder@builder x86_64-linux - 8 1 big-parallel" \
  --max-jobs 0

Capture log proving remote participation.

Step 5 — Persist config

Add nix.buildMachines on the client host flake or user remote builders file. Rebuild; run without CLI flags.

Step 6 — Feature flag experiment

Request a bogus mandatory feature; observe scheduling failure; remove it. Write one sentence on what you saw.

Step 7 — Combine with cache (optional)

On success, push the result (binary caches chapter). Then disable builder and rebuild from substitute-only.

Step 8 — Write topology diagram

ASCII in NOTES.md: who evals, who builds, who signs, who deploys, trust boundaries.

Step 9 — Failure injection

Stop sshd on builder (or block with firewall).
Retry build; capture the error class (SSH vs scheduling).
Restore builder; confirm recovery.

Document: “client symptoms when builder is down.”

Step 10 — Disk pressure note

Fill or simulate low disk on builder (carefully); document monitoring need (df -h /nix in runbooks).

Exercises

One paragraph: when cache alone is enough vs when you need a builder.
Design features list for a “CI builder” vs a “laptop offload” builder.
Explain why user SSH working is insufficient on multi-user Nix.
Estimate upload size for a clean builder without shared cache (qualitative).
Security review: what can a malicious builder do to your laptop store?

Common gotchas

Symptom / mistake	What to do
`cannot build on …`	Features/system mismatch; check `supportedFeatures`
SSH works as user, not as root	Configure root’s key or daemon SSH
Huge upload times	Enable `builders-use-substitutes`; share cache with builder
Darwin → Linux confusion	Need Linux builder; not just `--system`
Stale known_hosts / MITM prompts	Non-interactive SSH hardening for automation
Builder disk full	GC on builder; monitor
Nested flake eval differences	Same Nix version band; same inputs
Assuming cache = builder	Re-read Theory 1
Silent local build	`--max-jobs 0`; verify logs

Checkpoint

Distinguished cache vs remote builder
Completed at least one remote (or localhost SSH) build
Knew which SSH identity the builder used
Documented persistent config approach
Topology notes include trust
Failure injection documented
Features experiment observed

Commit

git add .
git commit -m "packaging: remote builders lab and topology"

Write three personal gotchas before reproducibility.

Remote builders

Why it matters

Theory 1 — Cache vs remote builder

Theory 2 — Configuration surfaces (2026)

NixOS (nix.buildMachines)

Ad-hoc one-shot

Features

Theory 3 — SSH and store talk

Theory 4 — Cross and multi-system

Theory 5 — Security notes

Decision card — do I need a builder?

Theory 6 — Scheduling knobs that matter

Upload storms

Theory 7 — Builder host minimal config

Topology diagram (keep in notes)

Worked example — two-VM lab checklist

Prove which machine built

Lab — multi-step

Step 1 — Inventory local capacity

Step 2 — Provision builder

Step 3 — SSH from daemon-relevant user

Step 4 — One-shot remote build

Step 5 — Persist config

Step 6 — Feature flag experiment

Step 7 — Combine with cache (optional)

Step 8 — Write topology diagram

Step 9 — Failure injection

Step 10 — Disk pressure note

Exercises

Common gotchas

Checkpoint

Commit

NixOS (`nix.buildMachines`)