Day 54 — Remote builders

Updated

July 30, 2026

Day 54 — Remote builders

Stage V · ~4h
Goal: Offload realization to a remote Nix machine (Linux builder VM is ideal), understand builders / remote-systems.conf style config, and know when remote build beats “just use a cache.” Align lab with nixos-26.05 clients and builders when possible.

Note

Remote builders move work; binary caches move results. Confusing them wastes weeks. You usually want both: build once on a beefy box, push to Cachix/Attic, everyone substitutes.

Why this day exists

Laptops should not compile Chromium. CI runners should not all rebuild the same uncached leaf. Remote builders are how Darwin talks to Linux, how small VMs ship large closures, and how fleets keep eval local while realization lives elsewhere. Stage VI deploys assume this vocabulary.

Theory 1 — Cache vs remote builder

	Binary cache	Remote builder
Needs existing path?	Yes	No
Uses your flake inputs	Downloads NARs	Evaluates plan, builds drv remotely
Good for	Repeat installs	First build, exotic systems
Trust	Signing keys	SSH + Nix auth + who runs builder
Network pattern	Pull results	Upload inputs + download outputs

Often: build once on a beefy builder → push to Cachix/Attic → everyone’s substituters hit.

[dev] --build--> [builder] --push--> [cache] --substitute--> [fleet]

Theory 2 — Configuration surfaces (2026)

NixOS (`nix.buildMachines`)

{
  nix.distributedBuilds = true;
  nix.buildMachines = [
    {
      hostName = "builder.lab.internal";
      system = "x86_64-linux";
      protocol = "ssh-ng"; # preferred modern protocol when available
      sshUser = "nixbuilder";
      sshKey = "/root/.ssh/id_build"; # or agent patterns
      maxJobs = 8;
      speedFactor = 4;
      supportedFeatures = [ "nixos-test" "big-parallel" "kvm" ];
      mandatoryFeatures = [ ];
    }
  ];
  nix.settings = {
    builders-use-substitutes = true; # remote may fetch deps itself
  };
}

Ad-hoc one-shot

nix build .#pkg --builders 'ssh://nixbuilder@builder.lab.internal x86_64-linux'
# classic:
# --option builders 'ssh://…'

Features

Feature	Typical meaning
`big-parallel`	Allow heavy parallel builds
`kvm`	Nested VM tests (`nixosTest`)
`nixos-test`	OS tests scheduling
`benchmark`	Specialization (rare)

If a drv requires kvm and the builder lacks it, scheduling skips that machine—often looking like “mysterious local build” or hard failure depending on flags.

Theory 3 — SSH and store talk

Remote build roughly:

Local eval (usually) produces derivations
Upload input paths the remote lacks
Remote builds
Download result paths back (unless you only needed remote side effects)

# Test SSH as the Nix daemon user (often root on NixOS)
sudo ssh -i /root/.ssh/id_build nixbuilder@builder.lab.internal nix-store --version

Daemon context: On multi-user Nix, root (nix-daemon) must reach the builder—not only your user ssh config—unless using remote build modes that run as you (know your setup).

Check	Why
Non-interactive SSH	No password prompts in daemon
`known_hosts` stable	Automation vs MITM prompts
Dedicated build key	Least privilege vs your personal key
Builder disk free	Full disk mid-build is common

Theory 4 — Cross and multi-system

Goal	Approach
Build `aarch64-linux` from `x86_64` laptop	aarch64 builder or qemu-binfmt (slower)
Build Linux from Darwin	Linux remote builder almost always
Same-arch offload only	Simplest lab
`nixosTest` needing kvm	Builder with kvm feature + hardware

# Prefer real hardware/VM matching system
system = "aarch64-linux";

Honest lab goal: one successful remote same-arch build. Cross is stretch, not gate.

Theory 5 — Security notes

Risk	Mitigation
Builder can see sources	Treat builder as trusted environment
SSH key sprawl	Dedicated build key; restrict `authorized_keys` if hardened
Untrusted builder returns malicious paths	Only use builders you control; CI policies
World-open builder	VPN/Tailscale; no public SSH
Builder as root playground	Separate user; trusted-users carefully

Remote builders are not a sandbox that makes untrusted code safe. A compromised builder is a supply-chain incident.

Theory 6 — Scheduling knobs that matter

Knob	Effect
`max-jobs 0` (client)	Force offload when supported
`speedFactor`	Prefer faster builders
`builders-use-substitutes`	Remote fetches deps itself
`supportedFeatures`	Eligibility filter
Local `max-jobs` > 0	Local may steal work

# force remote in many setups
nix build .#cooltool -L \
  --builders "ssh-ng://nixbuilder@builder x86_64-linux - 8 1 big-parallel" \
  --max-jobs 0

Worked example — two-VM lab topology

[dev laptop / host A] --ssh-ng--> [builder VM B: NixOS 26.05, multi-user Nix]
        |                                |
        +--------> Cachix/Attic <--------+  (optional push from B or A)

Builder VM minimal extras:

# on builder
users.users.nixbuilder = {
  isNormalUser = true;
  # group membership per current wiki; keep tight
};
nix.settings.trusted-users = [ "root" "nixbuilder" ];
# SSH key auth only; open firewall only to admin net

Exact nix.sshServe / store write permissions vary by version—verify against 26.05 docs when wiring production.

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day54

Step 1 — Inventory local capacity

nix show-config 2>/dev/null | rg 'max-jobs|cores|builders' || nix config show | rg 'max-jobs|cores|builders'
nproc
df -h /nix

Step 2 — Provision builder

Options:

Second NixOS VM on LAN/VPN
Same machine via ssh://localhost (limited learning value but tests plumbing)
Cloud NixOS box

Install Nix multi-user; enable flakes; ensure free disk (tens of GB).

Step 3 — SSH from daemon-relevant user

ssh -i ~/.ssh/id_build nixbuilder@builder 'nix-store --version'
# and as root if daemon is root:
sudo ssh -i /root/.ssh/id_build nixbuilder@builder 'nix-store --version'

Document which user the daemon uses on your OS.

Step 4 — One-shot remote build

nix build .#cooltool -L \
  --builders "ssh-ng://nixbuilder@builder x86_64-linux - 8 1 big-parallel" \
  --max-jobs 0

Capture log proving remote participation (builder hostname in log lines).

Step 5 — Persist config

Add nix.buildMachines on the client host flake or user remote builders file. Rebuild; run without CLI flags.

Step 6 — Feature flag experiment

Request a bogus mandatory feature; observe scheduling failure; remove it. Write one sentence on what you saw.

Step 7 — Combine with cache (optional)

On success, push the result (Day 53). Then disable builder and rebuild from substitute-only.

Step 8 — Write topology diagram

ASCII in NOTES.md: who evals, who builds, who signs, who deploys, trust boundaries.

Step 9 — Prove which machine built

# after remote build, inspect path info / logs for builder host
nix path-info -Sh .#cooltool
# keep a snippet of the build log that mentions the remote host

If you cannot show evidence of remote work, you may have silently built local—re-check --max-jobs 0 and builder SSH as daemon user.

Step 10 — Failure injection

Stop sshd on builder (or block with firewall).
Retry build; capture the error class (SSH vs scheduling).
Restore builder; confirm recovery.

Document: “client symptoms when builder is down.”

Decision card — do I need a builder today?

Situation	Prefer
Package already on cache	Substituter only
First build of large leaf on weak laptop	Remote builder
Darwin needing Linux closure	Linux builder
CI rebuilds identical drv every job	Push cache after one builder run
Untrusted contributor PR	Do not give them your builder

Common gotchas

Symptom / mistake	What to do
`cannot build on …`	Features/system mismatch; check `supportedFeatures`
SSH works as user, not as root	Configure root’s key or daemon SSH
Huge upload times	Enable `builders-use-substitutes`; share cache with builder
Darwin → Linux confusion	Need Linux builder; not just `--system`
Stale known_hosts / MITM prompts	Non-interactive SSH hardening for automation
Builder disk full	GC on builder; monitor
Nested flake eval differences	Same Nix version band; same inputs
Assuming cache = builder	Re-read Theory 1

Checkpoint

Distinguished cache vs remote builder
Completed at least one remote (or localhost SSH) build
Knew which SSH identity the builder used
Documented persistent config approach
Topology notes include trust

Commit

git add .
git commit -m "day54: remote builders lab and topology"

Write three personal gotchas before continuing.

Tomorrow

Day 55 — Reproducibility honesty: what is and is not bit-for-bit.

Day 54 — Remote builders

Why this day exists

Theory 1 — Cache vs remote builder

Theory 2 — Configuration surfaces (2026)

NixOS (nix.buildMachines)

Ad-hoc one-shot

Features

Theory 3 — SSH and store talk

Theory 4 — Cross and multi-system

Theory 5 — Security notes

Theory 6 — Scheduling knobs that matter

Worked example — two-VM lab topology

Lab — multi-step

Step 1 — Inventory local capacity

Step 2 — Provision builder

Step 3 — SSH from daemon-relevant user

Step 4 — One-shot remote build

Step 5 — Persist config

Step 6 — Feature flag experiment

Step 7 — Combine with cache (optional)

Step 8 — Write topology diagram

Step 9 — Prove which machine built

Step 10 — Failure injection

Decision card — do I need a builder today?

Common gotchas

Checkpoint

Commit

Tomorrow

NixOS (`nix.buildMachines`)