What evaluation sends to the daemon

Authors

K19G

grok-bot

Updated

October 31, 2026

What evaluation sends to the daemon

Goal: Measure, byte by byte, what a Nix client pushes over the daemon socket while it is only evaluating, account for every byte class (derivations, ${./path} imports, builtins.toFile), and know the three switches that make that traffic vanish when you do not need it.

Version note: every number below was measured with Nix 2.35.3 against nixos-26.05 (github:NixOS/nixpkgs/6fcc1603…, which evaluates hello to hello-2.12.3). The box was an unprivileged x86-64 Linux machine with no system /nix, so the “daemon” is nix daemon --stdio over a chroot store, fronted by a small counting proxy. The protocol on the wire is the same one /nix/var/nix/daemon-socket/socket speaks on a NixOS host; the absolute timings are this box’s, not yours.

Why this chapter exists

The store and daemon chapter established that on a multi-user install the client never writes /nix/store itself. What it leaves implicit is that this is true during evaluation too. “Evaluation” sounds like pure computation inside the client. It is not: every derivation the evaluator instantiates is a .drv file that must become a valid store path, and every ${./some/file} in nixpkgs is a source file that must be copied into the store before its path can be interpolated into a string. On a multi-user host, each of those writes is a request to the daemon.

That is invisible until it isn’t: slow nixos-rebuild evaluations on a loaded builder, CI jobs that only need a drvPath but pay for megabytes of writes, or a remote store (--store ssh-ng://…) where each round trip costs real latency.

Mental model

 nix eval / nix build / nixos-rebuild           nix-daemon (root)
 ┌──────────────────────────────┐   socket    ┌──────────────────────┐
 │ evaluator                    │             │                      │
 │  derivation { … }  ─ write ──┼─ AddToStore ┼─► /nix/store/…​.drv   │
 │  "${./patch.diff}" ─ copy ───┼─ AddToStore ┼─► /nix/store/…-patch  │
 │  builtins.toFile   ─ write ──┼─ AddToStore ┼─► /nix/store/…-name   │
 │  isValidPath? / queryPath ───┼─ queries ───┼─► sqlite db           │
 └──────────────────────────────┘             └──────────────────────┘

Three byte classes go up the socket while evaluating:

Class Source construct Store content address
Derivations every derivation/mkDerivation that gets instantiated text: (a .drv is a text file)
Path imports "${./file}", src = ./dir; coerced to a string fixed:r:sha256 (NAR-hashed)
builtins.toFile small generated scripts and configs text:

The evaluator cannot skip these by default: a derivation’s output path is computed from the store paths of its inputs, and the .drv is written so that a later nix build (possibly by a different process) can realise it.

The lab: a counting proxy

The proxy listens on a Unix socket, starts one nix daemon --stdio per client connection, and counts bytes in each direction. On a real host you would instead forward to /nix/var/nix/daemon-socket/socket; here the daemon serves a throwaway chroot store so every run can start cold.

# countproxy.py: Unix-socket proxy that spawns `nix daemon --stdio` per client
# and counts bytes in each direction.
import os, socket, subprocess, sys, threading, time
SOCK, NIX, STORE = sys.argv[1], sys.argv[2], sys.argv[3]
if os.path.exists(SOCK): os.unlink(SOCK)
srv = socket.socket(socket.AF_UNIX); srv.bind(SOCK); srv.listen(8)
def pump(src_read, dst_write, key, tot):
    while True:
        b = src_read(65536)
        if not b: break
        tot[key] += len(b); dst_write(b)
def handle(c):
    t0 = time.time()
    p = subprocess.Popen([NIX, "daemon", "--stdio", "--force-trusted", "--extra-experimental-features", "daemon-trust-override", "--store", STORE],
                         stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=0)
    tot = {"up": 0, "down": 0}
    def w(b): p.stdin.write(b); p.stdin.flush()
    def up():
        pump(c.recv, w, "up", tot); p.stdin.close()   # client hung up -> daemon exits
    a = threading.Thread(target=up); a.start()
    pump(p.stdout.read, c.sendall, "down", tot); a.join()
    print(f"conn client->daemon={tot['up']} B daemon->client={tot['down']} B "
          f"secs={time.time()-t0:.2f}", flush=True)
    p.wait(); c.close()
while True:
    c, _ = srv.accept(); threading.Thread(target=handle, args=(c,), daemon=True).start()

--force-trusted (behind the daemon-trust-override experimental feature) makes the stdio daemon treat the connection as trusted, which is what root talking to its own daemon gets anyway. Start it against an empty store:

FR=/tmp/nixfresh; rm -rf $FR
python3 countproxy.py /tmp/nixproxy.sock "$(command -v nix)" "local?root=$FR" > proxy.log &
NP=github:NixOS/nixpkgs/6fcc160353f63555b560f25588925133c8f61f2a
nix eval --no-eval-cache --store unix:///tmp/nixproxy.sock --raw $NP#hello.drvPath
/nix/store/v4c5h94di88ccdd9isyzhivdw84k81nh-hello-2.12.3.drv

(On the lab box the client itself ran inside nix shell --store <chroot> so that it could see a /nix/store; on a normal install it already can.) --no-eval-cache matters: with the flake evaluation cache warm, a repeated drvPath lookup never touches the evaluator or the daemon at all.

Case 1: cold store, three runs

Each run recreated the empty daemon store, evaluated once cold, then once more against the now-populated store:

cold 1
wall=1.175s
conn client->daemon=3489309 B daemon->client=936600 B secs=1.25
warm 1
wall=0.475s
conn client->daemon=365513 B daemon->client=177728 B secs=0.45
cold 2
wall=1.159s
conn client->daemon=3489309 B daemon->client=936600 B secs=1.27
warm 2
wall=0.498s
conn client->daemon=365513 B daemon->client=177728 B secs=0.48
cold 3
wall=1.174s
conn client->daemon=3489309 B daemon->client=936600 B secs=1.24
warm 3
wall=0.472s
conn client->daemon=365513 B daemon->client=177728 B secs=0.45

Evaluating one small package from a cold store pushed 3.33 MiB up the socket and received 0.89 MiB of replies. The byte counts are identical across runs: evaluation is deterministic, so so is its traffic. The warm run still sends 357 KiB. Those are validity queries and path registrations the client cannot avoid without already knowing what the daemon has.

Case 2: what the bytes were

Every path the cold run created is still in the throwaway store, so classify them by content address:

nix path-info --store "local?root=$FR" --json --all > fresh-pathinfo.json
python3 - <<'PY'
import json
d = json.load(open('fresh-pathinfo.json'))
items = d.items() if isinstance(d, dict) else [(x['path'], x) for x in d]
cat = {}
for p, i in items:
    ca = i.get('ca') or ''
    if isinstance(ca, dict): ca = ca.get('method', '') + ':'
    k = 'drv' if p.endswith('.drv') else ('toFile (text)' if str(ca).startswith('text') else 'path import (nar)')
    c = cat.setdefault(k, [0, 0]); c[0] += 1; c[1] += i['narSize']
for k, (n, s) in cat.items(): print(f"{k:20s} {n:5d} paths  {s/1024:8.0f} KiB NAR")
PY
path import (nar)      149 paths       831 KiB NAR
drv                    961 paths      1801 KiB NAR
toFile (text)           10 paths         6 KiB NAR

So about 2.6 MiB of payload: 961 derivations (the whole bootstrap chain behind hello, down to hex0-seed), 149 source files copied out of the nixpkgs tree (patches, setup hooks such as make-binary-wrapper.sh), and ten toFile texts. The other ~0.7 MiB on the wire is framing, NAR headers and the queries in between. Notice what is not there: the nixpkgs source tree itself. Nix 2.35 evaluates the flake input from its fetcher cache and only materialises the individual files a derivation actually interpolates.

Case 3: what that costs

The same cold evaluation, with the client opening the chroot store directly instead of going through a daemon:

direct cold wall=0.761s
direct cold wall=0.847s
direct cold wall=0.856s

and with --read-only through the proxy (more below):

read-only
wall=0.322s
conn client->daemon=632 B daemon->client=184 B secs=0.30

On this box, the daemon round trips add roughly 0.3–0.4 s to a ~0.8 s cold evaluation: about 30 % of the wall time is spent shipping and registering store writes rather than evaluating. Skipping instantiation entirely brings it down to 0.32 s and 816 bytes of total traffic (the handshake).

The three switches

Need Switch Effect
Just the drvPath/outPath, or an attribute value nix eval --read-only … (classic: nix-instantiate --readonly-mode) Paths are computed but no .drv or source is written; output path is identical
Repeated identical evaluations of a flake leave the eval cache on (it is, by default) No evaluation, so no traffic
Evaluate against a cheap local store, build elsewhere --eval-store <url> with --store ssh-ng://builder Instantiation writes go to the eval store; only the closure needed to build crosses to the remote

--read-only changes what the evaluator does, not what it returns. Its help text is exact: “Do not instantiate each evaluated derivation.” The drvPath it prints is the same path, but no file exists there afterwards. A later nix build will instantiate it again (and pay the traffic then).

The trap

“Eval is slow, so the daemon must be slow at building.”

On a builder that is busy realising other derivations, the daemon is also the process registering your evaluation’s 961 .drv writes, each a database transaction. Evaluation latency rises with daemon load even when you build nothing. The symptom is nixos-rebuild dry-build or nix flake check that are fast on an idle laptop and slow on the CI builder.

A second, quieter trap: measuring with the eval cache warm. Repeat nix eval .#x.drvPath twice and the second run looks free because it never evaluated. Use --no-eval-cache for any measurement in this chapter.

The boring rule

If you only need a path or a value, evaluate read-only. If you need to build, accept the writes, and keep evaluation on the machine that will build. For CI matrix jobs that compute drvPaths to decide what to build, --read-only is the default you want; the actual build job instantiates once.

Try this

  1. Run the proxy against your own daemon socket and evaluate your NixOS host’s config.system.build.toplevel.drvPath cold. How many derivations, and how many MiB? (Expect far more than hello: a whole system closure.)
  2. Repeat with --read-only and compare the printed path. It should be byte-identical.
  3. Add one "${./big-file}" to a flake output and watch the path-import class grow by that file’s NAR size on the next cold run.
  4. Point --store at ssh-ng:// a builder over a high-latency link and time the cold evaluation with and without --eval-store auto.

Sources

  • Nix 2.35 manual: nix eval (--read-only, --eval-store), common options, store URL types (unix://, local?root=), nix daemon --stdio and --force-trusted, https://nix.dev/manual/nix/latest/
  • Nix manual, “Derivations” and “String interpolation” (path coercion copies to the store), https://nix.dev/manual/nix/latest/language/string-interpolation
  • Nix manual, builtins.toFile, https://nix.dev/manual/nix/latest/language/builtins#builtins-toFile