Day 12 — Secrets: sops-nix & agenix

Updated

July 30, 2025

Day 12 — Secrets: sops-nix & agenix

Day 33 — Secret problem

Stage IV · ~4h
Goal: Internalize why secrets-as-Nix-strings fail on NixOS: the store is a world-readable object database. Find and remove secret anti-patterns from your lab flake before wiring sops-nix.

Important

Never put live passwords, API tokens, or private keys in flake.nix, modules, or HM text files. Examples below use obvious placeholders only.

Why this day exists

Stage IV builds real services. Services need credentials. The naive NixOS instinct:

# CATASTROPHIC PATTERN — do not use
services.someApp.password = "hunter2";

or:

# ALSO BAD
environment.variables.API_TOKEN = "sk-live-...";

Those values enter the Nix store, typically world-readable, often copied to binary caches if you ever push paths, and immortalized in old generations until GC.

Today is pure threat model + cleanup. Implementation tools: Days 34–35.

Theory 1 — The store is not a vault

Recall Day 2: store paths are immutable objects. Permissions are generally:

/nix/store/...  readable by all users on the machine

So:

Belief Reality
“It’s in my private git repo” Still world-readable on every machine that builds/activates
“Only root rebuilds” Any local user can cat the store path
“I’ll delete the file later” Old generations may retain the path
“I’ll push to my cache” You may distribute the secret to the cache’s consumers

Evaluation vs runtime

Nix evaluation sees strings. If a module option takes a string password, that string becomes part of a derivation input or config file in the store.

Theory 2 — Common anti-patterns catalog

Pattern Why it fails
Password options in nixpkgs modules set to plaintext Store + process listing risk
writeText "secret" "value" Explicit store file with secret
HM home.file with tokens Store-backed home links
Embedding TLS private keys in module text Private key in store
builtins.readFile ./secret.key into config File content becomes string in eval graph
Committing .env with production secrets Git history forever

builtins.readFile trap

# STILL BAD if the result is interpolated into world-readable config
services.foo.token = builtins.readFile ./token.txt;

Reading a non-store private file at eval time can also break purity and CI; when it “works,” it often still copies content into the store via generated configs.

Theory 3 — What “good” looks like (preview)

Approach Idea
sops-nix (primary this book) Encrypted files in git; decrypt on activation to root-owned paths
agenix Age-encrypted secrets; similar activation story
External secret inject Cloud SM, Vault—out of band at runtime
Password hashes only hashedPassword for users—not reversible plaintext
SSH keys Public in git; private only on disk/agent

Good secrets:

  1. Appear in git only encrypted (or not at all)
  2. Land on disk at activation/runtime with tight permissions
  3. Are not Nix string inputs to public derivations

Theory 4 — Generations and GC

Even after you remove a secret from the flake:

  • Previous system generations may still reference store paths containing it
  • nix-collect-garbage eventually drops unreachable paths—but not if a generation roots them

Incident response sketch:

  1. Rotate the credential (assume compromise)
  2. Remove from config
  3. Delete old generations once new system confirmed
  4. GC
  5. If paths were pushed to a cache: treat cache as compromised for that secret

Theory 5 — Multi-user threat on a “lab laptop”

If your lab has only one human user, risk feels abstract—until:

  • You enable a network service
  • A second local account exists
  • A container escapes
  • You copy a store path off-machine

Design as if local multi-user and backup tarballs of /nix/store exist.

Theory 6 — Distinguishing config from secret

Config (OK in git) Secret (not OK plaintext)
Port numbers DB passwords
Public hostnames API tokens
Public SSH keys Private SSH keys
Non-secret feature flags Session signing keys
ACME email (usually) ACME account private material (tool-managed)

Gray areas (document decisions): internal-only hostnames, non-production lab passwords that still teach bad habits—prefer good habits even in lab.

Lab 1 — Threat model paragraph

Write 10–15 lines:

  • Who can read /nix/store on your lab host?
  • Who can read your git remote?
  • What is the worst secret currently at risk (even if lab-only)?

Lab 2 — Audit the flake for anti-patterns

Run searches from the worked example. Open every hit. Classify:

Hit OK / Bad / Investigate

Fix Bad today by removing or replacing with placeholders and a comment // TODO day34 sops.

Lab 3 — Prove store readability (safe demo)

Create a fake secret pattern on a throwaway branch, rebuild or nix build a writeText, show:

# example educational only
nix build --impure --expr 'with import <nixpkgs> {}; writeText "fake" "not-a-real-secret"' 
# modern equivalent preferred:
nix-build -E 'with import (builtins.getFlake "nixpkgs") {}; writeText "fake" "not-a-real-secret"' 2>/dev/null || true

Simpler pure demo inside your flake temporarily:

# DO NOT MERGE — educational package
packages.x86_64-linux.fakeSecretDemo = pkgs.writeText "fake-secret" "PLACEHOLDER_ONLY";
nix build .#fakeSecretDemo
cat result
ls -la result

Then delete the demo package. Point: content is just a file in the store.

Lab 4 — User password hygiene check

If any user still has plaintext password configuration, migrate to:

  • SSH keys only for remote, and/or
  • hashedPassword generated via mkpasswd
# generate a hash interactively; paste hash only into config
mkpasswd -m yescrypt

Do not commit the interactive password itself.

Lab 5 — Remove anti-patterns and rebuild

sudo nixos-rebuild switch --flake .#lab

Confirm services you care about still run without embedding secrets yet (services that require secrets may wait for Day 34).

Lab 6 — Incident checklist card

Write a 5-step card for “secret committed to git”:

  1. Rotate
  2. Rewrite history or accept leak + rotate (policy)
  3. Remove from flake
  4. Drop generations / GC
  5. Audit caches

Common gotchas

Symptom / mistake What to do
“Lab only, nobody cares” Habits transfer to prod; practice correctly
Secret in old generation Rotate + GC old gens
HM file “mode 600” but store-backed Still world-readable store object
readFile feels private Content still often ends up in store configs
Binary cache push after leak Rotate; scrub if possible; assume leak
Focusing on encryption theater without rotation Rotation is mandatory after exposure

Checkpoint

  • Can explain why store ≠ vault in one minute
  • Repo audited with secret heuristics
  • Anti-patterns removed or ticketed for Day 34
  • Fake store demo understood and deleted
  • User access uses keys/hashes, not plaintext passwords in git
  • Incident rotation checklist written

Commit

git add .
git commit -m "day33: remove secret anti-patterns; document threat model"

Write three personal gotchas before continuing.

Tomorrow

Day 34 — sops-nix: encrypted-in-repo secrets with decrypt-at-activation—end-to-end one secret on the lab host.

Day 34 — sops-nix

Stage IV · ~4h
Goal: Wire sops-nix end-to-end: encrypt a secret in-repo, decrypt at activation, consume it from a NixOS service or timed script—without plaintext secrets in the Nix store evaluation graph.

Note

House primary: sops-nix. Day 35 compares agenix; do not run both as permanent dual systems unless you have a reason.

Important

Use placeholder values and lab-only credentials. Never paste production tokens into examples or commits.

Why this day exists

Day 33 established the threat model. Today installs the default tooling for this volume:

Property sops-nix approach
At rest in git Encrypted YAML/JSON/ENV (SOPS)
Encryption keys Age and/or GPG; age recommended
On host Decrypted at activation to a path like /run/secrets/...
Nix modules Reference secret paths, not secret strings

sops-nix flow from encrypt in git to activation secret path

sops-nix encrypt decrypt flow

Theory 1 — Components

Piece Role
SOPS Encrypts/decrypts structured files
age Modern encryption tool; identities (private) + recipients (public)
sops-nix NixOS/HM modules: decrypt during activation; systemd credentials integration patterns
.sops.yaml Creation rules: which keys encrypt which files

Flow

edit plaintext locally (editor via `sops file`)
    → encrypted file committed
    → nixos-rebuild
    → sops-nix activation decrypts with host/user age key
    → secret file on tmpfs/run with tight perms
    → service reads path

Theory 2 — Age keys (host-centric pattern)

Common NixOS pattern: host age key on the machine, public key in .sops.yaml creation rules.

Generate key (on the lab host; protect private key):

# install age if needed (temporary shell)
nix shell nixpkgs#age -c age-keygen -o "$HOME/.config/sops/age/keys.txt"
# private key file — NEVER commit
# public key printed as age1...

For unattended host decryption, sops-nix docs describe placing keys where the activation can read them (often /var/lib/sops-nix/key.txt or configured path)—follow current sops-nix README for 26.05-compatible setup. Private key stays on host disk with root-only permissions—not in git.

Tip

Also encrypt to your admin age key so you can decrypt on a workstation for editing. Creation rules can list multiple recipients.

Theory 3 — Flake input

inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
  sops-nix.url = "github:Mic92/sops-nix";
  sops-nix.inputs.nixpkgs.follows = "nixpkgs";
  # home-manager ...
};

Import module:

modules = [
  ./hosts/lab
  inputs.sops-nix.nixosModules.sops
];

Theory 4 — Minimal NixOS sops config

# modules/common/sops.nix
{ config, ... }:
{
  sops.defaultSopsFile = ../../secrets/lab.yaml;
  # sops.age.keyFile = "/var/lib/sops-nix/key.txt"; # per upstream guidance
  sops.age.sshKeyPaths = [ ]; # or use ssh host key conversion patterns if you choose that method

  sops.secrets."lab/demo_token" = {
    # owner / group / mode as needed
    mode = "0400";
    # restartUnits = [ "some-service.service" ];
  };
}

Consume path:

# example: a oneshot that proves the secret path exists
systemd.services.sops-demo-check = {
  wantedBy = [ "multi-user.target" ];
  after = [ "etc.service" ]; # adjust ordering as needed
  serviceConfig = {
    Type = "oneshot";
    RemainAfterExit = true;
    ExecStart = "/bin/sh -c 'test -f /run/secrets/lab/demo_token && echo ok > /var/lib/sops-demo-ok'";
  };
};

Exact secret path layout depends on sops-nix version and naming—inspect after activation:

sudo ls -la /run/secrets

Never do builtins.readFile config.sops.secrets."lab/demo_token".path into another world-readable derivation if that re-embeds content. Prefer systemd LoadCredential, EnvironmentFile pointing at the secret path, or scripts that read at runtime.

Theory 5 — Creating encrypted files

.sops.yaml example shape:

keys:
  - &admin age1AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
  - &host age1BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
creation_rules:
  - path_regex: secrets/.*\.yaml$
    key_groups:
      - age:
          - *admin
          - *host

Edit:

nix shell nixpkgs#sops nixpkgs#age -c sops secrets/lab.yaml

Example structure before encryption (editor buffer only):

lab:
  demo_token: PLACEHOLDER_LAB_TOKEN_ROTATE_ME

After save, file on disk is encrypted ciphertext suitable for git.

Theory 6 — HM secrets (awareness)

sops-nix also integrates with Home Manager for user secrets. Today prefer system secret for clarity; user secrets can wait until you need them.

Theory 7 — Bootstrapping chicken-and-egg

First host key setup is the awkward part:

  1. Generate age key on host
  2. Put public key in .sops.yaml
  3. Encrypt secrets to that recipient
  4. Place private key where sops-nix expects
  5. Rebuild

Document your bootstrap in docs/secrets.md. Losing the age private key without other recipients = cannot decrypt—keep admin recipient + backups of keys (offline).

Worked example — Secret for a custom script service

{ config, pkgs, ... }:
{
  sops.secrets."lab/demo_token" = {
    mode = "0400";
    owner = "root";
  };

  systemd.services.lab-token-touch = {
    description = "Demonstrate runtime read of sops secret path";
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      Type = "oneshot";
      RemainAfterExit = true;
      ExecStart = pkgs.writeShellScript "lab-token-touch" ''
        set -euo pipefail
        # path attribute is the runtime location
        token_path="${config.sops.secrets."lab/demo_token".path}"
        test -r "$token_path"
        # Do NOT echo the token to logs
        wc -c < "$token_path" > /var/lib/lab-token-bytes
      '';
    };
  };
}

Note: writeShellScript embeds the path string, not the secret content—good.

Lab 1 — Add input and module

# edit flake inputs; nix flake update sops-nix
# import sops-nix.nixosModules.sops
nix flake metadata

Lab 2 — Age keys and .sops.yaml

  1. Generate admin age identity (laptop)
  2. Generate or configure host identity per sops-nix docs
  3. Write .sops.yaml with both public keys
  4. Confirm private keys are gitignored
keys.txt
*.agekey
/var/lib/sops-nix/  # if ever copied into repo by mistake — should never be

Lab 3 — First encrypted secret file

mkdir -p secrets
# create encrypted secrets/lab.yaml via sops
git add secrets/lab.yaml .sops.yaml
# ensure plaintext never staged

Lab 4 — Declare sops.secrets and rebuild

sudo nixos-rebuild switch --flake .#lab
sudo ls -la /run/secrets
sudo wc -c /run/secrets/...   # do not cat into scrollback you paste publicly

Lab 5 — Runtime consumer

Add the oneshot that reads path without logging secret contents. Prove /var/lib/lab-token-bytes or similar.

Lab 6 — Failure drill

  1. Break key access (rename key file temporarily)
  2. Rebuild or restart activation path
  3. Observe failure mode
  4. Restore key

Journal: how the host fails closed.

Lab 7 — Docs

docs/secrets.md:

  • Key locations
  • How to add a secret
  • How to add a recipient
  • Rotation sketch

Common gotchas

Symptom / mistake What to do
Cannot decrypt on host Host public key not in creation rules / wrong private key path
Cannot edit on laptop Admin age key missing from recipients
Secret path empty Activation order; sops config; check journal
Plaintext committed Rotate; purge if needed; fix
readFile secret into store Use runtime path only
Logged secret via echo $TOKEN Never log secrets
GPG-only complexity Prefer age for new labs

Checkpoint

  • sops-nix input pinned with follows
  • .sops.yaml with host + admin recipients
  • Encrypted secret file in git
  • Private keys not in git
  • sops.secrets declared; appears under /run/secrets
  • Service/script consumes path only
  • Bootstrap docs written

Commit

git add .
git commit -m "day34: sops-nix end-to-end lab secret"

Write three personal gotchas before continuing.

Tomorrow

Day 35 — agenix comparison: implement the same secret once with agenix, then pick a house default (sops-nix for this book).

Day 35 — agenix comparison

Stage IV · ~4h
Goal: Implement the same lab secret once with agenix, compare trade-offs to sops-nix, and declare a house default (this volume: sops-nix primary).

Note

Comparison day ≠ dual-stack forever. After the lab, remove or disable the non-default tool so you maintain one system.

Why this day exists

Both sops-nix and agenix solve “encrypted secrets in git + decrypt on activation.” Teams pick based on:

Factor Why it matters
File format UX YAML multi-secret files vs one age file per secret
Tooling sops editor vs agenix CLI
Ecosystem fit Docs you read; modules you copy
Complexity Bootstrap and key distribution

You should not choose from Twitter vibes alone. Touch both once.

Theory 1 — agenix mental model

agenix (and compatible forks/community usage) roughly:

Concept Meaning
secrets.nix Maps secret files → public keys allowed to decrypt
*.age files Ciphertext in repo
age identities Private keys on machines/admins
NixOS module age.secrets.<name>.file = ./secret.age; → runtime path

Flow

echo 'PLACEHOLDER' | agenix -e secret.age
git add secret.age secrets.nix
nixos-rebuild → decrypt to /run/agenix/... (path scheme per version)
service reads path

Theory 2 — Flake input sketch

inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
  agenix.url = "github:ryantm/agenix";
  agenix.inputs.nixpkgs.follows = "nixpkgs";
};

# modules list:
# inputs.agenix.nixosModules.default

Pin and follows like any other input (Day 24 discipline).

Theory 3 — secrets.nix shape

# secrets/secrets.nix — public keys only
let
  # replace with your real age public keys
  admin = "age1AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
  host  = "age1BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB";
in
{
  "lab-demo-token.age".publicKeys = [ admin host ];
}

No private keys in this file.

Theory 4 — NixOS declaration

{ config, inputs, ... }:
{
  imports = [ inputs.agenix.nixosModules.default ];

  age.secrets.lab-demo-token = {
    file = ../../secrets/lab-demo-token.age;
    # mode / owner as needed
  };

  # config.age.secrets.lab-demo-token.path → runtime path
}

Consumer pattern matches Day 34: path at runtime, never embed content in derivations.

Theory 5 — Side-by-side comparison

Dimension sops-nix agenix
Multi-key structured file Natural (YAML tree) Usually one file per secret
Edit UX sops file in-place decrypt editor agenix -e
Creation rules .sops.yaml secrets.nix publicKeys
HM integration Strong story Available patterns
Learning transfer SOPS used outside Nix too Age-centric, Nix-flavored
House default (this book) Primary Compare once

When agenix might still win for you

  • You want the simplest possible “one secret, one file” model
  • Your team already standardized on agenix
  • You dislike SOPS YAML tooling

When sops-nix wins for this curriculum

  • Multiple secrets per environment file
  • Same SOPS skills in non-Nix contexts
  • Syllabus / house default consistency

Theory 6 — Dual-running risks

Running both modules on one host:

  • Two key bootstraps
  • Two runtime directories
  • Cognitive load on incident response

Acceptable for one lab day. Not acceptable as permanent undefined state—pick a default by end of day.

Theory 7 — Declaring house default

Write in docs/secrets.md:

## House default

**sops-nix** for all new secrets.

agenix: lab comparison only; module removed after Day 35
(or: we chose agenix because … — only if you override the book).

If you override the book, do so explicitly so future chapters’ sops examples are translated consciously.

Worked example — Same placeholder secret both ways

Tool Artifact
sops secrets/lab.yaml key lab.demo_token
agenix secrets/lab-demo-token.age

Same placeholder value during local edit; different ciphertext formats in git. Runtime paths differ; consumer scripts should use module-provided .path.

Lab 1 — Add agenix input

# flake input + lock update agenix only
nix flake update agenix
nix flake metadata

Lab 2 — Public keys and secrets.nix

Reuse public age keys from Day 34 if possible (same admin/host). Write secrets/secrets.nix with publicKeys only.

Lab 3 — Encrypt one age secret

Using current agenix/age CLI docs for your pin:

# conceptual — see agenix README for exact flags
nix shell github:ryantm/agenix#agenix nixpkgs#age
# agenix -e secrets/lab-demo-token.age

Put only a lab placeholder value inside.

Lab 4 — NixOS module + rebuild

Declare age.secrets.…, rebuild, list runtime secrets dir for agenix:

sudo nixos-rebuild switch --flake .#lab
sudo ls -la /run/agenix 2>/dev/null || sudo ls -la /run/secrets 2>/dev/null || true

Confirm path exists; do not paste secret contents into notes.

Lab 5 — Consumer parity

Point a oneshot (or temporary) at config.age.secrets.lab-demo-token.path similar to Day 34. Prove readable by the intended user.

Lab 6 — Comparison write-up

One page in the journal:

  1. Bootstrap friction: sops vs agenix
  2. Day-2 secret add friction
  3. Multi-secret ergonomics
  4. Failure modes observed
  5. Decision: house default = …

Lab 7 — Decommission the non-default

If house default is sops-nix:

  1. Remove agenix module from active host imports
  2. Optionally leave encrypted .age file as historical artifact or delete
  3. Remove agenix input or keep input unused (prefer remove to simplify lock)
  4. Rebuild clean

Opposite steps if you chose agenix (then translate later chapters).

Common gotchas

Symptom / mistake What to do
Private key in secrets.nix Remove; rotate
Wrong public key list Re-encrypt with correct recipients
Both tools half-configured Finish Lab 7
Assuming identical paths Always use .path from the module
Editing ciphertext as plaintext Use tool entrypoints only
Forgetting follows on agenix input Fix lock discipline

Checkpoint

  • agenix encrypt + decrypt-on-activate worked once
  • Comparison table completed in your words
  • House default declared in docs/secrets.md
  • Non-default tool removed or clearly disabled
  • Still no plaintext secrets in git
  • Rebuild succeeds with chosen stack only

Commit

git add .
git commit -m "day35: agenix comparison; house default documented"

Write three personal gotchas before continuing.

Tomorrow

Day 36 — Hardening baseline: SSH, sudo, disable unused services, careful kernel defaults—as a small reusable module.