Binary caches

Updated

July 30, 2026

Binary caches

Goal: Pull and push substitutes with cache.nixos.org, Cachix, and/or self-hosted Attic / Harmonia—with a clear model of trust and signatures.

Why it matters

Without substitutes, packaging and fleet deploys become compile farms. With careless substituters, you outsource trust to whoever can serve a path. Caches are a performance and supply-chain topic, not just “make CI fast.”

Theory 1 — Substituters vs builders

eval → wanted store path P
         ├─ trusted substituter has P?  → download (+ verify sigs)
         └─ else                        → build (local or remote builder)

Term	Meaning
Substituter	HTTP(S) NAR store endpoint (`cache.nixos.org`)
Binary cache	Popular name for substituter + signing
Signed NAR	Artifact with signature by a cache key
Trusted key	Public keys your Nix is willing to accept
Priority	Order among substituters

# NixOS example
nix.settings = {
  substituters = [
    "https://cache.nixos.org"
    "https://myorg.cachix.org"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    "myorg.cachix.org-1:BASE64KEY…"
  ];
};

Flake/nix.conf user settings similar under ~/.config/nix/nix.conf or daemon config.

Remote builders (next chapter) move work. Caches move results. You usually want both.

Theory 2 — Trust model (non-negotiable)

Claim	Reality
“HTTPS means safe binaries”	No—HTTPS protects transport, not who built the bits
“I added a cache URL only”	Incomplete without matching public key (or explicit none)
“Private cache = trusted”	Only if builders and keys are controlled
“cache.nixos.org is magic”	Trusted by default for official paths; still a third party

Rule: Every substituter you add is part of your TCB (trusted computing base) for that machine class.

Who can push?  →  Who holds signing keys?  →  Which hosts trust the pubkey?
                      ↓
              What is the revoke plan?

Theory 3 — Cachix (hosted, popular)

Cachix hosts per-project caches with simple push UX.

nix shell nixpkgs#cachix -c cachix --help

# Auth (token from UI; treat as secret)
export CACHIX_AUTH_TOKEN=…   # prefer secret manager / CI secret

cachix use myorg             # writes substituter + pubkey locally (daemon may need trust)
cachix push myorg ./result

CI sketch:

- uses: cachix/install-nix-action@v31
- uses: cachix/cachix-action@v16
  with:
    name: myorg
    authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
- run: nix build .#default -L

Pin action versions to what you verify—follow current Cachix docs for 2026.

Push full runtime closure

nix-store -qR ./result | cachix push myorg
# or modern copy patterns when endpoint supports them

Theory 4 — Self-hosted: Attic & Harmonia

Project	Role (approx.)	When
Attic	Modern multi-tenant binary cache server	Teams wanting self-host push/pull
Harmonia	cache.nixos.org-compatible server in Rust	Serve local store as substituter
nix-serve / nix-serve-ng	Classic signing servers	Legacy blogs

Harmonia sketch (NixOS)

services.harmonia = {
  enable = true;
  # signKeyPath = "/var/lib/secrets/harmonia.secret"; # manage via sops-nix
  # settings…
};
# nginx TLS reverse proxy in front in production

Attic sketch (conceptual)

# server configured with tokens / buckets (see upstream Attic docs)
attic login local http://cache.example.internal:8080 "$TOKEN"
attic push local:lab ./result

Exact CLI flags evolve—read current Attic docs for 26.05-era packaging; labs may use Cachix or local Harmonia/Attic or file-store simulation.

Theory 5 — Push what matters

Push	Skip
Your packages & CI deps	Entire nixpkgs rebuilds by accident
Closure of deploy targets	Personal `/tmp` experiments
Signed artifacts from trusted CI	Unsigned ad-hoc laptop builds for prod (policy choice)

Policy question	Example house rule
Who may push?	CI on `main` only
Who may add a substituter?	Module PR review
Laptop pushes?	Dev cache only, not prod keys

Theory 6 — Substituter order and priority

nix.settings = {
  substituters = [
    "https://cache.nixos.org"
    "https://myorg.cachix.org"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    "myorg.cachix.org-1:…"
  ];
};

Idea	Practice
Official cache first	Common default
Extra caches append	Org/project binaries
Keys must match	Failures if mismatch / unsigned

Never add a substituter without its public key. Trust is cryptographic, not vibes.

Theory 7 — What “cache miss” really means

eval → derive output path hash
  → query substituters for that path
     → hit: download NAR
     → miss: build locally (or remote builder)

A miss is normal for private packages. A miss on hello from cache.nixos.org means network, config, or exotic platform issues.

nix path-info --store https://cache.nixos.org nixpkgs#hello
nix store ping --store https://cache.nixos.org

Force local build (contrast)

nix build .#cooltool --option substitute false -L

Feel the difference; do not leave substitute false in user config.

Theory 8 — Daemon vs user config

On multi-user Nix (NixOS default), daemon settings govern builds:

Location	Affects
`nix.settings` in NixOS	Daemon (system)
`~/.config/nix/nix.conf`	User; may be ignored for daemon builds
`trusted-users`	Who may set extra substituters

Symptom: “I added a cache to my user conf and nothing happened.” Fix: system module or trusted user policy.

Worked examples

NixOS module fragment for team cache

{ lib, ... }:
{
  nix.settings = {
    substituters = lib.mkAfter [
      "https://myorg.cachix.org"
    ];
    trusted-public-keys = lib.mkAfter [
      "myorg.cachix.org-1:REPLACE_ME="
    ];
  };
}

Local file NAR cache (always available lab path)

nix build .#cooltool -o result
nix copy --to "file://$PWD/nar-cache" ./result
# elsewhere or after GC of path (careful):
nix copy --from "file://$PWD/nar-cache" --to "$NIX_STORE" ./result

Query whether a path would substitute

OUT=$(nix build nixpkgs#hello --print-out-paths --no-link)
nix path-info --store https://cache.nixos.org --json "$OUT" | jq .

Lab — multi-step

Suggested workspace: ~/lab/nixos/packaging/cache

Step 1 — Observe default cache

nix build nixpkgs#hello -L
# note download lines mentioning cache.nixos.org
nix config show 2>/dev/null | rg -i 'substituter|trusted-public' \
  || nix show-config | rg -i 'substituter|trusted-public'

Step 2 — Build your package cold-ish

nix build .#cooltool --rebuild -L

Time it roughly.

Step 3 — Choose push target

Path A — Cachix: create free/personal cache if allowed; cachix use; push ./result.
Path B — Self-host lab: enable Harmonia or Attic on a lab VM; configure key; push/copy.
Path C — Document-only: if no account/VM, write full config you would apply and simulate with nix copy --to file:///….

Step 4 — Second machine / second user story

On another user or VM (or after careful path GC):

nix build .#cooltool -L
# should substitute if cache wired and path present

Step 5 — Trust write-up

TRUST.md:

Who can push?
Who holds signing keys?
What hosts trust the pubkey?
What is the revoke plan?

Step 6 — CI note

Add a short paragraph: push from CI only with secrets, not from random laptops, if that is your house rule. Ops CI chapter deepens this.

Step 7 — Negative test

Point at a cache without adding the public key (on a disposable Nix config if you can). Observe failure mode; restore.

Step 8 — Document lab cache policy

In docs/cache.md:

Which substituters are enabled
Which keys
Whether you push CI builds
Who may add a new cache (code review rule)

Exercises

Explain why HTTPS alone is insufficient for binary trust (one paragraph).
Map your org’s (or lab’s) TCB for substituters.
Compare wall time: substitute vs --option substitute false for a medium package.
Draft a revoke checklist for a leaked Cachix token.
Find the public key string for cache.nixos.org on your machine; confirm it matches docs.

Common gotchas

Symptom / mistake	What to do
“don’t know how to trust” / unsigned	Add correct `trusted-public-keys` or sign
User `nix.conf` ignored by daemon	Configure system/daemon NixOS settings
Pushed wrong path	Push derivation outputs you care about; verify with another client
Cache hit but old content	Paths are immutable; rebuild means new hash
Leaked Cachix token	Rotate; treat like deploy key
Priority fights	Order substituters; understand fallback to build
Assumed private URL = auth	Many caches need tokens separately from Nix substituter URL
Pushed entire nixpkgs by accident	Scope push to your closures

Checkpoint

Explain substituter + signature trust
Built with and without substitutes
Pushed or file-store simulated push
Second client pull story demonstrated/documented
TRUST.md written
Daemon vs user config understood
Cache miss vs hit explained

Commit

git add .
git commit -m "packaging: binary caches push/pull and trust notes"

Write three personal gotchas before remote builders.