Day 44 — Stage IV gate

Updated

July 30, 2026

Day 44 — Stage IV gate

Stage IV · ~4h
Goal: Prove the Stage IV exit criteria: a lab slice that is declarative, secret-safe, firewalled, and rebuildable—proxy + secrets tooling + one app/data path—with evidence that grep finds no live secrets and a fresh VM story is clear.

Important

Gates are proof days. Do not start greenfield k3s or a second database here. Close gaps from Days 33–43.

Why this day exists

Stage IV added production-shaped risk:

Risk Your control
Secret leakage via store/git sops-nix (house default)
Network exposure firewall + localhost binds
Service sprawl one proxy pattern + one DB
“Works on my SSH session” flake rebuild + docs

Stage V packaging will add more moving parts. Exit Stage IV clean.

Theory 1 — Syllabus exit criteria

grep-able repo has no live secrets; slice rebuilds on a fresh VM from flake + keys.

Break down:

  1. No live secrets in git (encrypted OK; plaintext not)
  2. Slice = proxy + secret consumption + data or app path + firewall story
  3. Fresh VM = clone flake, place age key, generate hardware, rebuild
  4. Keys = age/sops host key + admin SSH—not passwords in the repo

Theory 2 — Slice definition (minimum)

Your lab must demonstrate:

Component Evidence
Secrets sops-nix (or declared house default) decrypts at least one secret to /run/...
Proxy / TLS Front door pattern (public ACME or documented lab TLS)
Data or app path Postgres or OCI/native app behind proxy
Firewall docs/ports.md matches reality
Hardening SSH keys-only baseline still true
Layout Still hosts/modules/homes—not a relapsed monolith

Optional: observability, containers, k3s—only if already done and stable.

Theory 3 — Evidence pack

Create docs/stage-iv-evidence.md:

# Stage IV evidence

## Architecture diagram (ASCII)
[clients] -> :443 proxy -> localhost app
                |
             postgres (localhost)
secrets: sops -> /run/secrets/...

## Commands that passed
- rg secret audit
- nix flake check
- nixos-rebuild switch
- curl proxy
- psql / app check
- sudo ls /run/secrets (no values pasted)

## Pins
- nixpkgs: …
- sops-nix: …

## Fresh VM steps
1. …
2. …

## Debt carried to later stages
- …

Theory 4 — Secret audit protocol

cd /path/to/your-flake
rg -n -i 'password\s*=\s*\"[^\"]+\"' .
rg -n -i 'BEGIN (OPENSSH|RSA|EC) PRIVATE KEY' .
rg -n -i 'api[_-]?key\s*=\s*\"' .
rg -n 'AKIA[0-9A-Z]{16}' .  # cloud key shape heuristic
# encrypted sops files should look like ciphertext, not plain tokens

Classify hits. Fix before claiming the gate.

Theory 5 — Fresh VM narrative (real or dry)

Ideal: second VM completes rebuild.

Dry-run acceptable if resource-limited—must be step-complete:

  1. Clone repo
  2. Install NixOS minimal / use existing installer path
  3. Copy/generate hardware-configuration.nix
  4. Install host age private key to configured path
  5. nixos-rebuild switch --flake .#lab
  6. Verify proxy + secret path + DB

List which secrets/keys are out of band (not in git).

Theory 6 — What Stage V will assume

You can:

  • Navigate module layout
  • Update locks selectively
  • Keep secrets out of derivations
  • Run a networked service responsibly

Packaging craft should not fight a burning host config.

Lab 1 — Architecture freeze

Draw the ASCII diagram of the actual slice (not aspirational). If something is missing, implement the smallest fix or demote it from the gate claim.

Lab 2 — Secret audit

Run Theory 4 searches. Record clean result or fixes. Do not paste secret values into the evidence file.

Lab 3 — Full rebuild

sudo nixos-rebuild switch --flake .#lab
nix flake check -L

Fix failures.

Lab 4 — Functional probes

# SSH
ssh lab true

# Proxy
curl -k -sS -o /dev/null -w '%{http_code}\n' https://localhost/ 2>/dev/null || curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1/

# Secret path exists (no cat of values)
sudo test -e /run/secrets && sudo find /run/secrets -type f | head

# DB
sudo -u postgres psql -d labapp -c 'SELECT 1;' 2>/dev/null || true

Adjust to your hostnames and paths.

Lab 5 — Firewall reconcile

ss -tulpn
# compare to docs/ports.md — fix either reality or docs

Lab 6 — Fresh VM write-up

Complete the step list in evidence. If you perform a real second VM, attach notes on time taken and issues.

Lab 7 — Gate commit / tag

git add .
git commit -m "day44: stage IV gate — secret-safe lab slice"
git tag -a stage-iv-gate -m "Stage IV exit"

Lab 8 — Personal retrospective

Three questions:

  1. What almost leaked a secret?
  2. What service would you not put on this host yet?
  3. What will you simplify before Stage V?

Theory 7 — Slice dependency graph

age key (out of band)
    → sops decrypt at activation
        → app/DB credentials
            → data service
                → app/backend
                    → reverse proxy :443
                        → clients
firewall policy wraps the graph

If any node is imperative-only, the fresh VM story fails.

Theory 8 — Evidence that greps clean

# run from flake root; tune patterns to your threat model
rg -n --hidden -g '!.git' \
  -e 'BEGIN OPENSSH PRIVATE KEY' \
  -e 'BEGIN RSA PRIVATE KEY' \
  -e 'AGE-SECRET-KEY' \
  -e 'AKIA[0-9A-Z]{16}' \
  || true

Encrypted sops blobs may still match naive patterns—know what is ciphertext vs plaintext. Live passwords and private keys are the fail condition.

Worked example — Gate eval probes

nix eval .#nixosConfigurations.lab.config.services.openssh.enable
nix eval .#nixosConfigurations.lab.config.networking.firewall.enable
# proxy enable path — adjust to caddy/nginx
nix flake check -L || true

Lab 9 — Rollback confidence after secrets

Confirm you still know:

sudo nixos-rebuild switch --rollback

Rolling back config does not roll back DB rows or rotated secrets—write that sentence in the evidence pack.

Lab 10 — ports.md final reconcile

Every listening non-localhost port must appear in docs/ports.md. Delete docs rows that no longer exist. Gate fails on fiction.

Common gotchas

Symptom / mistake What to do
Gate with plaintext “lab” passwords in git Remove + rotate; fail the gate until fixed
Proxy works; secrets demo missing Wire one sops secret consumer
ports.md fantasy Reconcile Lab 5
Monolith returned Re-apply Day 23 structure
k3s half-broken eating the day Disable; skip path; gate native slice
Evidence only oral Write docs/stage-iv-evidence.md

Checkpoint — Stage IV exit

  • sops-nix (or documented house default) E2E
  • No live plaintext secrets in repo (audit clean)
  • Hardening baseline retained
  • Firewall documented and true
  • TLS/proxy front door pattern live
  • Data service or app path declarative
  • Containers/k3s either solid or out of the claim
  • Observability at least logs runbook or metrics
  • Fresh VM steps written
  • Evidence doc committed
  • nix flake check + rebuild pass

Commit

git add .
git commit -m "day44: stage IV gate"

Write three personal gotchas before continuing.

Tomorrow

Day 45 — stdenv phases: Stage V packaging craft begins—build software with the stdenv phase model, not only consume it.