Day 33 — Secret problem

Updated

July 30, 2026

Day 33 — Secret problem

Stage IV · ~4h
Goal: Internalize why secrets-as-Nix-strings fail on NixOS: the store is a world-readable object database. Find and remove secret anti-patterns from your lab flake before wiring sops-nix.

Important

Never put live passwords, API tokens, or private keys in flake.nix, modules, or HM text files. Examples below use obvious placeholders only.

Why this day exists

Stage IV builds real services. Services need credentials. The naive NixOS instinct:

# CATASTROPHIC PATTERN — do not use
services.someApp.password = "hunter2";

or:

# ALSO BAD
environment.variables.API_TOKEN = "sk-live-...";

Those values enter the Nix store, typically world-readable, often copied to binary caches if you ever push paths, and immortalized in old generations until GC.

Today is pure threat model + cleanup. Implementation tools: Days 34–35.

Theory 1 — The store is not a vault

Recall Day 2: store paths are immutable objects. Permissions are generally:

/nix/store/...  readable by all users on the machine

So:

Belief Reality
“It’s in my private git repo” Still world-readable on every machine that builds/activates
“Only root rebuilds” Any local user can cat the store path
“I’ll delete the file later” Old generations may retain the path
“I’ll push to my cache” You may distribute the secret to the cache’s consumers

Evaluation vs runtime

Nix evaluation sees strings. If a module option takes a string password, that string becomes part of a derivation input or config file in the store.

Theory 2 — Common anti-patterns catalog

Pattern Why it fails
Password options in nixpkgs modules set to plaintext Store + process listing risk
writeText "secret" "value" Explicit store file with secret
HM home.file with tokens Store-backed home links
Embedding TLS private keys in module text Private key in store
builtins.readFile ./secret.key into config File content becomes string in eval graph
Committing .env with production secrets Git history forever

builtins.readFile trap

# STILL BAD if the result is interpolated into world-readable config
services.foo.token = builtins.readFile ./token.txt;

Reading a non-store private file at eval time can also break purity and CI; when it “works,” it often still copies content into the store via generated configs.

Theory 3 — What “good” looks like (preview)

Approach Idea
sops-nix (primary this book) Encrypted files in git; decrypt on activation to root-owned paths
agenix Age-encrypted secrets; similar activation story
External secret inject Cloud SM, Vault—out of band at runtime
Password hashes only hashedPassword for users—not reversible plaintext
SSH keys Public in git; private only on disk/agent

Good secrets:

  1. Appear in git only encrypted (or not at all)
  2. Land on disk at activation/runtime with tight permissions
  3. Are not Nix string inputs to public derivations

Theory 4 — Generations and GC

Even after you remove a secret from the flake:

  • Previous system generations may still reference store paths containing it
  • nix-collect-garbage eventually drops unreachable paths—but not if a generation roots them

Incident response sketch:

  1. Rotate the credential (assume compromise)
  2. Remove from config
  3. Delete old generations once new system confirmed
  4. GC
  5. If paths were pushed to a cache: treat cache as compromised for that secret

Theory 5 — Multi-user threat on a “lab laptop”

If your lab has only one human user, risk feels abstract—until:

  • You enable a network service
  • A second local account exists
  • A container escapes
  • You copy a store path off-machine

Design as if local multi-user and backup tarballs of /nix/store exist.

Theory 6 — Distinguishing config from secret

Config (OK in git) Secret (not OK plaintext)
Port numbers DB passwords
Public hostnames API tokens
Public SSH keys Private SSH keys
Non-secret feature flags Session signing keys
ACME email (usually) ACME account private material (tool-managed)

Gray areas (document decisions): internal-only hostnames, non-production lab passwords that still teach bad habits—prefer good habits even in lab.

Lab 1 — Threat model paragraph

Write 10–15 lines:

  • Who can read /nix/store on your lab host?
  • Who can read your git remote?
  • What is the worst secret currently at risk (even if lab-only)?

Lab 2 — Audit the flake for anti-patterns

Run searches from the worked example. Open every hit. Classify:

Hit OK / Bad / Investigate

Fix Bad today by removing or replacing with placeholders and a comment // TODO day34 sops.

Lab 3 — Prove store readability (safe demo)

Create a fake secret pattern on a throwaway branch, rebuild or nix build a writeText, show:

# example educational only
nix build --impure --expr 'with import <nixpkgs> {}; writeText "fake" "not-a-real-secret"' 
# modern equivalent preferred:
nix-build -E 'with import (builtins.getFlake "nixpkgs") {}; writeText "fake" "not-a-real-secret"' 2>/dev/null || true

Simpler pure demo inside your flake temporarily:

# DO NOT MERGE — educational package
packages.x86_64-linux.fakeSecretDemo = pkgs.writeText "fake-secret" "PLACEHOLDER_ONLY";
nix build .#fakeSecretDemo
cat result
ls -la result

Then delete the demo package. Point: content is just a file in the store.

Lab 4 — User password hygiene check

If any user still has plaintext password configuration, migrate to:

  • SSH keys only for remote, and/or
  • hashedPassword generated via mkpasswd
# generate a hash interactively; paste hash only into config
mkpasswd -m yescrypt

Do not commit the interactive password itself.

Lab 5 — Remove anti-patterns and rebuild

sudo nixos-rebuild switch --flake .#lab

Confirm services you care about still run without embedding secrets yet (services that require secrets may wait for Day 34).

Lab 6 — Incident checklist card

Write a 5-step card for “secret committed to git”:

  1. Rotate
  2. Rewrite history or accept leak + rotate (policy)
  3. Remove from flake
  4. Drop generations / GC
  5. Audit caches

Common gotchas

Symptom / mistake What to do
“Lab only, nobody cares” Habits transfer to prod; practice correctly
Secret in old generation Rotate + GC old gens
HM file “mode 600” but store-backed Still world-readable store object
readFile feels private Content still often ends up in store configs
Binary cache push after leak Rotate; scrub if possible; assume leak
Focusing on encryption theater without rotation Rotation is mandatory after exposure

Checkpoint

  • Can explain why store ≠ vault in one minute
  • Repo audited with secret heuristics
  • Anti-patterns removed or ticketed for Day 34
  • Fake store demo understood and deleted
  • User access uses keys/hashes, not plaintext passwords in git
  • Incident rotation checklist written

Commit

git add .
git commit -m "day33: remove secret anti-patterns; document threat model"

Write three personal gotchas before continuing.

Tomorrow

Day 34 — sops-nix: encrypted-in-repo secrets with decrypt-at-activation—end-to-end one secret on the lab host.