Remote Builders

Updated

September 12, 2026

Remote Builders

Binary caches move results. Remote builders move work. The boring default is: evaluate on the laptop, realise on a machine that has CPU, KVM, and the right system, then push the result to the team cache.

A laptop should not compile Chromium. A Darwin laptop cannot natively produce x86_64-linux test VMs. Both problems are remote builders, not “more -j”.

Mental model

Binary cache Remote builder
Needs the path to already exist? Yes No
What moves first NAR (result) .drv + input paths, then NAR back
Trust Signing keys SSH user on the builder
Typical role Everyone pulls One beefy box (or a queue) builds 
laptop  --drv-->  builder  --push-->  cache  --substitute-->  fleet

nix.buildMachines (NixOS) or --builders (CLI) lists machines:

  • hostName, sshUser, sshKey
  • system (x86_64-linux, aarch64-linux, …)
  • protocol (ssh-ng preferred)
  • maxJobs, speedFactor
  • supportedFeatures (kvm, big-parallel, nixos-test)

If a derivation requires kvm and the builder does not advertise it, Nix builds locally (or fails). Mysterious local compiles are often a missing feature flag.

ssh-ng is the Nix 2 protocol (one SSH session, multiplexed). Plain ssh (legacy) copies NARs over a new connection per path — slower, more fragile. Prefer protocol = "ssh-ng". mandatoryFeatures is the strict sibling of supportedFeatures: the derivation must have those features or Nix will not pick the machine. Use it so a big-parallel Chromium job cannot land on a 2-core builder that merely listed kvm.

Worked examples

Case 1: NixOS client configuration

Save as builders.nix:

# builders.nix
{
  nix.distributedBuilds = true;
  nix.settings.builders-use-substitutes = true;

  nix.buildMachines = [
    {
      hostName = "builder.desk.internal";
      system = "x86_64-linux";
      protocol = "ssh-ng";
      sshUser = "nixbuilder";
      sshKey = "/root/.ssh/id_builder";
      maxJobs = 8;
      speedFactor = 4;
      supportedFeatures = [ "nixos-test" "big-parallel" "kvm" ];
    }
  ];
}

builders-use-substitutes = true lets the builder download from the team cache instead of asking the laptop to upload gcc.

The SSH key is a host key for root (the daemon builds as root’s SSH). Protect it with sops. nixbuilder on the far side must be a trusted Nix user.

sudo nixos-rebuild switch

Case 2: One-shot without a module

nix build .#desk-api --max-jobs 0 \
  --builders 'ssh-ng://nixbuilder@builder.desk.internal x86_64-linux - 8 1 kvm,big-parallel,nixos-test'

--max-jobs 0 means “do not build locally.” If the builder is down, the command fails instead of melting the laptop.

Case 3: Builder host — NixOS

Save as builder-host.nix:

# builder-host.nix
{ pkgs, ... }:

{
  users.users.nixbuilder = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDeskLaptopRootBuilderKey desk-laptop"
    ];
  };

  nix.settings.trusted-users = [ "root" "nixbuilder" ];
  nix.settings.max-jobs = 8;

  services.openssh.enable = true;
}

trusted-users is required for nixbuilder to start builds as the daemon. Do not put your whole team in trusted-users on a laptop; do it on the builder.

Test SSH as the key the daemon will use:

sudo ssh -i /root/.ssh/id_builder nixbuilder@builder.desk.internal nix-store --version

If this asks for a password, the daemon will hang.

Case 4: Darwin → Linux

An Apple silicon laptop producing Linux containers:

# darwin-builders.nix
{
  nix.distributedBuilds = true;
  nix.buildMachines = [
    {
      hostName = "linux-builder.desk.internal";
      system = "aarch64-linux";
      protocol = "ssh-ng";
      sshUser = "nixbuilder";
      sshKey = "/var/root/.ssh/id_builder";
      maxJobs = 8;
      supportedFeatures = [ "kvm" "big-parallel" "nixos-test" ];
    }
  ];
}

linux-builder can be a VM on the Mac (nix.linux-builder.enable on nix-darwin) or a real ARM server. Prefer a real server for nixosTest (KVM). The VM is fine for GOARCH=arm64 CGO-off Go; it is a poor Hydra.

Root on the laptop does not read ~deskadmin/.ssh/config. Put Host builder in /root/.ssh/config or use an IP in hostName.

Case 5: Features — why tests ran locally

nix build .#checks.x86_64-linux.desk-vm --max-jobs 0

If the check is a nixosTest, the builder must list kvm and nixos-test. Missing those, Nix tries the local machine; on Darwin that is a slow VM or a failure.

nix show-config | grep builders

Confirm the live config, not just the .nix file you meant to switch.

The trap

The trap is pointing buildMachines at a machine that is also a substituter with a different nixpkgs. You upload inputs, it rebuilds everything because its store is on 23.11. Same flake.lock everywhere; builder substitutes from the team cache (builders-use-substitutes).

The other trap is user SSH config (~/.ssh/config Host aliases) that root’s daemon does not read. Use hostName IPs or put the alias in /root/.ssh/config.

The boring rule

  • Cache for results, builders for first compiles and foreign system.
  • ssh-ng, builders-use-substitutes, --max-jobs 0 when you mean “remote only.”
  • Advertise kvm / nixos-test / big-parallel honestly.
  • Daemon SSH keys, not your interactive ssh-agent, unless you designed for that.
  • Builder nixpkgs + cache = laptop flake lock + team cache.

Try this

  1. sudo ssh from Case 3 until nix-store --version prints without a password.
  2. nix build nixpkgs#hello --max-jobs 0 --builders '…' and watch the builder’s nix-daemon log (journalctl -fu nix-daemon).
  3. Remove kvm from supportedFeatures, run a nixosTest, and note where it builds. Put kvm back.
  4. After a remote build, nix path-info -Shr ./result on the laptop — the NAR should have been copied back. Then cachix push so the next laptop never builds it.