The secrets problem

Updated

July 30, 2026

The secrets problem

Goal: Internalize why secrets-as-Nix-strings fail on NixOS: the store is a world-readable object database. Find and remove secret anti-patterns from your lab flake before wiring sops-nix.

Important

Never put live passwords, API tokens, or private keys in flake.nix, modules, or HM text files. Examples below use obvious placeholders only.

Why this chapter exists

Real services need credentials. The naive NixOS instinct:

# CATASTROPHIC PATTERN — do not use
services.someApp.password = "hunter2";

or:

# ALSO BAD
environment.variables.API_TOKEN = "sk-live-...";

Those values enter the Nix store, typically world-readable, often copied to binary caches if you ever push paths, and immortalized in old generations until GC.

This chapter is pure threat model + cleanup. Implementation tools: sops-nix (next) and agenix (comparison).

Theory 1 — The store is not a vault

Recall the store and closures chapter: store paths are immutable objects. Permissions are generally:

/nix/store/...  readable by all users on the machine

So:

Belief Reality
“It’s in my private git repo” Still world-readable on every machine that builds/activates
“Only root rebuilds” Any local user can cat the store path
“I’ll delete the file later” Old generations may retain the path
“I’ll push to my cache” You may distribute the secret to the cache’s consumers
“HM mode 600 symlink” Target in store is still world-readable

Evaluation vs runtime

Nix evaluation sees strings. If a module option takes a string password, that string becomes part of a derivation input or config file in the store.

Nix string secret
  → appears in .drv / generated config
  → realized under /nix/store
  → readable by local users (and caches if pushed)

Good secrets appear as paths to runtime files (tmpfs /run/secrets/...) that are not Nix-string content of the secret itself.

Theory 2 — Common anti-patterns catalog

Pattern Why it fails
Password options in nixpkgs modules set to plaintext Store + process listing risk
writeText "secret" "value" Explicit store file with secret
HM home.file with tokens Store-backed home links
Embedding TLS private keys in module text Private key in store
builtins.readFile ./secret.key into config File content becomes string in eval graph
Committing .env with production secrets Git history forever
environment.sessionVariables with tokens Often ends up visible
Embedding secrets in shellHook Project flakes leak too

builtins.readFile trap

# STILL BAD if the result is interpolated into world-readable config
services.foo.token = builtins.readFile ./token.txt;

Reading a non-store private file at eval time can also break purity and CI; when it “works,” it often still copies content into the store via generated configs.

Theory 3 — What “good” looks like (preview)

Approach Idea
sops-nix (primary this book) Encrypted files in git; decrypt on activation to root-owned paths
agenix Age-encrypted secrets; similar activation story
External secret inject Cloud SM, Vault—out of band at runtime
Password hashes only hashedPassword for users—not reversible plaintext
SSH keys Public in git; private only on disk/agent
LoadCredential / EnvironmentFile systemd reads path at start

Good secrets:

  1. Appear in git only encrypted (or not at all)
  2. Land on disk at activation/runtime with tight permissions
  3. Are not Nix string inputs to public derivations
# GOOD shape (conceptual)
# service reads config.sops.secrets."app/db_password".path
# NEVER builtins.readFile that path into another derivation

Theory 4 — Generations and GC

Even after you remove a secret from the flake:

  • Previous system generations may still reference store paths containing it
  • nix-collect-garbage eventually drops unreachable paths—but not if a generation roots them

Incident response sketch:

  1. Rotate the credential (assume compromise)
  2. Remove from config
  3. Delete old generations once new system confirmed
  4. GC
  5. If paths were pushed to a cache: treat cache as compromised for that secret
sudo nix-env --list-generations -p /nix/var/nix/profiles/system
# sudo nix-collect-garbage -d   # only when you accept dropping old gens

Theory 5 — Multi-user threat on a “lab laptop”

If your lab has only one human user, risk feels abstract—until:

  • You enable a network service
  • A second local account exists
  • A container escapes
  • You copy a store path off-machine
  • A backup tarball includes /nix/store

Design as if local multi-user and backup tarballs of /nix/store exist.

Theory 6 — Distinguishing config from secret

Config (OK in git) Secret (not OK plaintext)
Port numbers DB passwords
Public hostnames API tokens
Public SSH keys Private SSH keys
Non-secret feature flags Session signing keys
ACME email (usually) ACME account private material (tool-managed)
Package names Registry pull tokens

Gray areas (document decisions): internal-only hostnames, non-production lab passwords that still teach bad habits—prefer good habits even in lab.

Theory 7 — Process listings and logs

Even outside the store:

Risk Mitigation
ps shows command-line secrets Prefer env files / credentials; never cmd --password x
journald logs echo secrets Never echo "$TOKEN" in units
Core dumps Limit where appropriate; avoid secrets in argv

Theory 8 — Git history is forever (enough)

Force-push rewriting history is incomplete if:

  • Forks and clones exist
  • CI caches retained logs
  • Chat paste included the secret

Rotation is mandatory after exposure. History scrubbing is extra, not a substitute.

Worked example — Safe store readability demo

# DO NOT MERGE — educational package in flake outputs temporarily
# packages.x86_64-linux.fakeSecretDemo =
#   pkgs.writeText "fake-secret" "PLACEHOLDER_ONLY";
# nix build .#fakeSecretDemo
# cat result
# ls -la result

Then delete the demo. Point: content is just a file in the store.

Exercises

Exercise 1 — Threat model paragraph

Write 10–15 lines:

  • Who can read /nix/store on your lab host?
  • Who can read your git remote?
  • What is the worst secret currently at risk (even if lab-only)?

Exercise 2 — Audit the flake for anti-patterns

Run searches from the worked example. Open every hit. Classify:

Hit OK / Bad / Investigate

Fix Bad by removing or replacing with placeholders and a comment // TODO sops.

Exercise 3 — Prove store readability (safe demo)

Build a fake writeText or temporary package; show world-readable content; delete.

Exercise 4 — User password hygiene check

If any user still has plaintext password configuration, migrate to:

  • SSH keys only for remote, and/or
  • hashedPassword generated via mkpasswd
mkpasswd -m yescrypt

Do not commit the interactive password itself.

Exercise 5 — Remove anti-patterns and rebuild

sudo nixos-rebuild switch --flake .#lab

Confirm services you care about still run without embedding secrets yet.

Exercise 6 — Incident checklist card

Write a 5-step card for “secret committed to git”:

  1. Rotate
  2. Rewrite history or accept leak + rotate (policy)
  3. Remove from flake
  4. Drop generations / GC
  5. Audit caches

Exercise 7 — HM file audit

rg -n 'home\.file|xdg\.configFile' homes/ || true

Ensure no token-like .text values.

Exercise 8 — Environment variable audit

rg -n 'environment\.(variables|sessionVariables)|Environment\s*=' .

Exercise 9 — Generation awareness

sudo nix-env --list-generations -p /nix/var/nix/profiles/system | tail

Note how many generations might hold old secrets if any ever existed.

Exercise 10 — Backup story note

If you back up the host, does the backup include store paths? Write one sentence on secret risk in backups.

Exercise 11 — Public vs private key check

Confirm only public SSH keys live in the flake; private keys only on disk/agent.

Exercise 12 — Document TODO list for sops

List every secret the next service chapters will need (DB password, optional Grafana admin, DNS token). No values—names only.

Exercise 13 — Peer review simulation

Read your threat model as if onboarding a coworker. Fix any “lab only, nobody cares” handwaves.

Exercise 14 — CI log risk

If CI evaluates the flake, confirm logs would not print secrets (they should not be in eval). Note any impure readFile risks.

Common gotchas

Symptom / mistake What to do
“Lab only, nobody cares” Habits transfer to prod; practice correctly
Secret in old generation Rotate + GC old gens
HM file “mode 600” but store-backed Still world-readable store object
readFile feels private Content still often ends up in store configs
Binary cache push after leak Rotate; scrub if possible; assume leak
Focusing on encryption theater without rotation Rotation is mandatory after exposure
hashedPassword confused with password Hash only in git
Secrets in commit messages Rotate; amend is not enough if pushed
Demo package left merged Remove educational writeText
Thinking sops alone without path discipline Still don’t readFile into store

Checkpoint

  • Can explain why store ≠ vault in one minute
  • Repo audited with secret heuristics
  • Anti-patterns removed or ticketed for sops-nix
  • Fake store demo understood and deleted
  • User access uses keys/hashes, not plaintext passwords in git
  • Incident rotation checklist written
  • Named list of future secrets (no values)
  • HM and environment audits clean

Journal (optional)

Write three personal gotchas before continuing.