Setting Up Binary Caches

Updated

September 12, 2026

Setting Up Binary Caches

Compiling llvm on every laptop is not a culture. The boring default is: a signed substituter (cache.nixos.org plus one team cache), public keys in nix.conf, and CI that uploads what it built.

Mental model

When Nix needs /nix/store/<hash>-pkg it:

  1. Looks in the local store.
  2. Queries each substituter for a .narinfo signed with a trusted public key.
  3. Downloads the NAR if the signature matches.
  4. Builds from the .drv only if nobody had the bytes.
Piece Role
Substituter URL https://cache.nixos.org, https://desk.cachix.org, http://attic.desk.internal
Trusted public key cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
Signing private key Lives on CI / the cache host. Never in git.

Cachix is the hosted boring choice. Attic or Harmonia is the self-hosted one. The NixOS module is the same shape: extra substituters + extra keys.

laptop  --query-->  desk-cache  --miss-->  cache.nixos.org  --miss-->  build
CI      --build-->  sign  --put-->  desk-cache

A substituter without its public key is silently skipped (or refused). You compile the world and blame the network.

nix.settings.substituters replaces the default list if you assign it — keep https://cache.nixos.org in the list. extra-substituters appends. Same for keys: trusted-public-keys vs extra-trusted-public-keys. require-sigs stays true (the default). nixConfig.extra-substituters inside flake.nix only applies when the user/daemon accept-flake-config (or a trusted user); it is not a substitute for /etc/nix/nix.conf. Never put a signing secret in nixConfig.

Worked examples

Case 1: Consume a team cache on NixOS

Save as cache_config.nix:

# cache_config.nix
{
  nix.settings = {
    substituters = [
      "https://cache.nixos.org"
      "https://desk-cache.cachix.org"
    ];
    trusted-public-keys = [
      "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
      "desk-cache.cachix.org-1:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa="
    ];
    require-sigs = true;
  };
}

Replace the desk-cache key with the real key Cachix showed when you created the cache. A truncated key in this book is not trusted; Nix will ignore that substituter.

sudo nixos-rebuild switch
nix show-config | grep -E 'substituters|trusted-public-keys'

On a multi-user machine the daemon reads /etc/nix/nix.conf. ~/.config/nix/nix.conf is ignored for daemon builds.

Untrusted users cannot add substituters. If a laptop must use the team cache without being root:

# extraTrusted
{
  nix.settings.trusted-users = [ "root" "deskadmin" ];
}

Prefer listing the cache in the system nix.settings (Case 1) so every uid gets it. trusted-users is also who may import unsigned paths — keep it short.

Case 2: Push from a workstation (once)

nix build .#desk-api
cachix watch-exec desk-cache -- nix build .#desk-api

or:

nix build .#desk-api
cachix push desk-cache ./result

CACHIX_AUTH_TOKEN is an environment variable from the Cachix dashboard. It is a secret. It is not a flake input. Laptops pull; CI pushes. Do not require every laptop to upload.

Case 3: Verify a hit versus a miss

nix build .#desk-api --print-build-logs 2>&1 | tee /tmp/desk-build.log
grep -E 'copying path|building' /tmp/desk-build.log | head

copying path '…' from 'https://desk-cache.cachix.org' is a hit. building '…drv' is a miss. A team that still builds llvm locally has a key mismatch or is pushing different hashes (different nixpkgs lock).

nix build nixpkgs#hello --max-jobs 0

--max-jobs 0 forbids a local compile. If hello errors, the substituter list is empty or unsigned.

nix show-config | grep -E 'substituters|trusted-public-keys' 
curl -fsS https://desk-cache.cachix.org/nix-cache-info | head

StoreDir: /nix/store in nix-cache-info is a live cache. HTTP 404 here is why every laptop rebuilds llvm.

Case 4: Serve a tiny cache with nix copy

Without Cachix, an S3 bucket or a directory works:

nix copy --to file://$PWD/desk-nar .#desk-api
nix copy --from file://$PWD/desk-nar --to ssh://deskadmin@desk-vm .#desk-api

file:// is a lab. Production is HTTP + signatures (nix-serve, Harmonia, Attic). Unsigned file:// on a USB stick is fine for air-gap day; it is not a fleet cache.

Case 5: Generate a signing key for a self-hosted cache

nix key generate-secret --key-name desk-cache > /run/secrets/desk-cache.secret
nix key convert-secret-to-public < /run/secrets/desk-cache.secret

Put the public half in trusted-public-keys. Put the secret on the builder only (sops). nix copy --to 's3://desk-cache?secret-key=/run/secrets/desk-cache.secret' signs on upload.

The trap

The trap is adding a substituter without the public key. Nix will skip it. You will compile the world and blame the network.

The other trap is putting the private signing key in flake.nix or in a GitHub Actions log (echo $CACHIX_AUTH_TOKEN). Rotate immediately.

A third trap: substituters in ~/.config/nix/nix.conf on a multi-user machine. The daemon reads /etc/nix/nix.conf.

A fourth: assigning substituters = [ "https://desk-cache.cachix.org" ]; and dropping cache.nixos.org. Every hello is now a local compile unless your team cache mirrors the world.

The boring rule

  • cache.nixos.org plus one team cache. Not five random community caches you do not audit. Keep cache.nixos.org in the list.
  • Public keys next to substituter URLs, in the daemon’s nix.conf. require-sigs = true.
  • CI pushes. Laptops pull. Do not require every laptop to upload.
  • Never commit signing secrets. nixConfig in the flake is not the daemon.
  • Confirm a hit with “copying path from …” before you call the cache done.

Try this

  1. nix show-config | grep -E 'substituters|trusted-public-keys' and identify which keys match which URLs.
  2. nix build nixpkgs#hello --max-jobs 0 (no local build). It should substitute. If it errors, your substituter list is empty.
  3. Create a file:// cache of hello (Case 4) and copy it into a second store (nix copy --from …).
  4. Intentionally typo the public key, rebuild a package you do not have locally, and read the error. Restore the key.