Day 53 — Binary caches

Updated

July 30, 2026

Day 53 — Binary caches

Stage V · ~4h (lab-heavy)
Goal: Pull and push substitutes with cache.nixos.org, Cachix, and/or self-hosted Attic / Harmonia—with a clear model of trust and signatures.

Why this day exists

Without substitutes, Stage V and fleet deploys become compile farms. With careless substituters, you outsource trust to whoever can serve a path. Caches are a performance and supply-chain topic, not just “make CI fast.”

Theory 1 — Substituters vs builders

eval → wanted store path P
         ├─ trusted substituter has P?  → download (+ verify sigs)
         └─ else                        → build (local or remote builder)

Term	Meaning
Substituter	HTTP(S) NAR store endpoint (`cache.nixos.org`)
Binary cache	Popular name for substituter + signing
Signed NAR	Artifact with signature by a cache key
Trusted key	Public keys your Nix is willing to accept
Priority	Order among substituters

# NixOS example
nix.settings = {
  substituters = [
    "https://cache.nixos.org"
    "https://myorg.cachix.org"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    "myorg.cachix.org-1:BASE64KEY…"
  ];
};

Flake/nix.conf user settings similar under ~/.config/nix/nix.conf or daemon config.

Theory 2 — Trust model (non-negotiable)

Claim	Reality
“HTTPS means safe binaries”	No—HTTPS protects transport, not who built the bits
“I added a cache URL only”	Incomplete without matching public key (or explicit none)
“Private cache = trusted”	Only if builders and keys are controlled
“cache.nixos.org is magic”	Trusted by default for official paths; still a third party

Rule: Every substituter you add is part of your TCB (trusted computing base) for that machine class.

Theory 3 — Cachix (hosted, popular in 2026)

Cachix hosts per-project caches with simple push UX.

# Install client via nix once
nix profile install nixpkgs#cachix

# Auth (token from UI; treat as secret—Day 33–35 patterns)
export CACHIX_AUTH_TOKEN=…   # prefer secret manager / GH Actions secret

cachix use myorg             # writes substituter + pubkey locally (daemon may need trust)
cachix push myorg ./result
# or
nix build .#foo | cachix push myorg

CI sketch:

- uses: cachix/install-nix-action@v31
- uses: cachix/cachix-action@v16
  with:
    name: myorg
    authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
- run: nix build .#default -L

(Pin action versions to what you verify in 2026—don’t invent untrusted SHAs; follow current Cachix docs.)

Theory 4 — Self-hosted: Attic & Harmonia

Project	Role (approx.)	When
Attic	Modern multi-tenant binary cache server	Teams wanting self-host push/pull
Harmonia	cache.nixos.org-compatible server in Rust	Serve local store as substituter
nix-serve / nix-serve-ng	Classic signing servers	Legacy blogs

Harmonia sketch (NixOS)

services.harmonia = {
  enable = true;
  # signKeyPath = "/var/lib/secrets/harmonia.secret"; # manage via sops-nix
  # settings…
};
# nginx TLS reverse proxy in front in production

Attic sketch (conceptual)

# server configured with tokens / buckets (see upstream Attic docs)
attic login local http://cache.example.internal:8080 "$TOKEN"
attic push local:lab ./result

Exact CLI flags evolve—read current Attic docs for 26.05-era packaging; the lab below allows Cachix or a local Harmonia/Attic path.

Theory 5 — Push what matters

Push	Skip
Your packages & CI deps	Entire nixpkgs rebuilds by accident
Closure of deploy targets	Personal `/tmp` experiments
Signed artifacts from trusted CI	Unsigned ad-hoc laptop builds for prod (policy choice)

# Push full runtime closure
nix-store -qR ./result | cachix push myorg
# modern:
nix copy --to 'http://…' ./result   # when endpoint supports it

Worked examples

NixOS module fragment for team cache

{ lib, ... }:
{
  nix.settings = {
    substituters = lib.mkAfter [
      "https://myorg.cachix.org"
    ];
    trusted-public-keys = lib.mkAfter [
      "myorg.cachix.org-1:REPLACE_ME="
    ];
    # optional: allow user to add more without daemon edit fights
    trusted-users = [ "root" "@wheel" ];
  };
}

Query whether a path would substitute

nix path-info --store https://cache.nixos.org --json $(nix build nixpkgs#hello --print-out-paths) | jq .

Build without substitutes (contrast)

nix build .#cooltool --option substitute false -L

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day53

Step 1 — Observe default cache

nix build nixpkgs#hello -L
# note download lines mentioning cache.nixos.org
nix config show | rg -i 'substituter|trusted-public'

Step 2 — Build your Stage V package cold-ish

nix build .#cooltool --rebuild -L

Time it roughly.

Step 3 — Choose push target

Path A — Cachix: create free/personal cache if allowed; cachix use; push ./result.
Path B — Self-host lab: enable Harmonia or Attic on a lab VM; configure key; push/copy.
Path C — Document-only: if no account/VM, write full config you would apply and simulate with nix copy --to file:///… local store dump.

# Path C example: local file nar cache
nix copy --to file://$PWD/nar-cache ./result
nix copy --from file://$PWD/nar-cache --to $NIX_STORE ./result

Step 4 — Second machine / second user story

On another user or VM (or after GC of the path if safe):

nix build .#cooltool -L
# should substitute if cache wired and path present

Step 5 — Trust write-up

TRUST.md:

Who can push?
Who holds signing keys?
What hosts trust the pubkey?
What is the revoke plan?

Step 6 — CI note

Add a short paragraph linking to Day 66: push from GitHub Actions only with secrets, not from random laptops, if that is your house rule.

Step 7 — Negative test

Point at a cache without adding the public key (on a disposable Nix config if you can). Observe failure mode; restore.

Theory 6 — Substituter order and priority

nix.settings = {
  substituters = [
    "https://cache.nixos.org"
    # "https://myorg.cachix.org"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    # "myorg.cachix.org-1:…"
  ];
};

Idea	Practice
Official cache first	Common default
Extra caches append	Org/project binaries
Keys must match	Silent ignore / fail if mismatch depending on config

Never add a substituter without its public key. Trust is cryptographic, not vibes.

Theory 7 — What “cache miss” really means

eval → derive output path hash
  → query substituters for that path
     → hit: download NAR
     → miss: build locally (or remote builder)

A miss is normal for private packages. A miss on hello from cache.nixos.org means network, config, or exotic platform issues.

Worked example — query without installing

nix path-info --store https://cache.nixos.org nixpkgs#hello
nix store ping --store https://cache.nixos.org

Lab — document lab cache policy

In docs/cache.md:

Which substituters are enabled
Which keys
Whether you push CI builds
Who may add a new cache (code review rule)

Lab — force a local build once

nix build nixpkgs#hello --option substitute false -L

Feel the difference; re-enable substitutes after. Do not leave substitute false in user config.

Common gotchas

Symptom / mistake	What to do
“don’t know how to trust” / unsigned	Add correct `trusted-public-keys` or sign
User `nix.conf` ignored by daemon	Configure system/daemon NixOS settings
Pushed wrong path	Push derivation outputs you care about; verify with another client
Cache hit but old content	Paths are immutable; rebuild means new hash
Leaked Cachix token	Rotate; treat like deploy key
Priority fights	Order substituters; understand fallback to build
Assumed private URL = auth	Many caches need tokens separately from Nix substituter URL

Checkpoint

Explain substituter + signature trust
Built with and without substitutes
Pushed or file-store simulated push
Second client pull story demonstrated/documented
TRUST.md written

Commit

git add .
git commit -m "day53: binary caches push/pull and trust notes"

Write three personal gotchas before continuing.

Tomorrow

Day 54 — Remote builders: offload builds to another machine safely.