Day 84 — Capstone 2/5

Updated

July 30, 2026

Day 84 — Capstone 2/5 — secrets & proxy

Stage VIII · ~4h (implementation)
Goal: Complete secrets E2E (sops-nix or agenix) and the edge reverse-proxy path so a service is reachable through the front door without plaintext secrets in git.

Important

Use lab-only tokens. Never commit production credentials. Prefer sops-nix as house default (Day 34); agenix is fine if Capstone already standardized on it—do not run both permanently.

Why this day exists

Host skeletons without secrets train bad habits. Proxy without a TLS story (or documented lab HTTP mode) is not an edge pattern. Day 84 closes Capstone A items for secrets + reverse proxy with proof commands you can paste into the checklist.

Theory 1 — Secrets E2E definition of done

Step	Done means
Encrypt	`secrets/.yaml` (or age files) committed encrypted*
Keys	Machine age/ssh key or admin key decrypts on host
NixOS wiring	`sops.secrets.` or `age.secrets.` → path on disk
Consumer	systemd `LoadCredential` / `EnvironmentFile` / path read
Negative proof	Repo clone alone cannot read secret without keys
Test story	nixosTest uses dummy secret or test key (Days 76/86)
Offline custody	Private keys exist off the lab disk (Day 87–88)

Threat model (short)

Bad outcome	Control
Secret in git plaintext	SOPS/age encrypt before commit
Secret in Nix string → store	Reference paths, never secret values in modules
World-readable decrypted file	`owner` / `mode` / service user match
Lost keys after wipe	Offline backup of age/ssh private material

Theory 2 — sops-nix sketch (preferred)

# host module fragment — follow current sops-nix README for key paths
{ config, ... }:
{
  sops.defaultSopsFile = ./secrets/secrets.yaml;
  # common patterns: age ssh host key and/or dedicated age key file
  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  # sops.age.keyFile = "/var/lib/sops-nix/key.txt";

  sops.secrets."myapp/api_token" = {
    owner = "myapp";
    group = "myapp";
    mode = "0400";
    restartUnits = [ "myapp.service" ];
  };

  systemd.services.myapp = {
    description = "Capstone sample app";
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "myapp";
      Group = "myapp";
      # Prefer credentials / EnvironmentFile pointing at the decrypted path:
      EnvironmentFile = config.sops.secrets."myapp/api_token".path;
      # Or ExecStart that reads the path—never pkgs.writeText with the token
    };
  };
}

`.sops.yaml` creation rules (idea)

Encrypt lab secrets to host recipient + admin recipient so you can edit on a workstation and the machine can decrypt at activation. Exact YAML keys follow current SOPS + sops-nix docs.

# edit encrypted file (opens $EDITOR with decrypted buffer)
sops secrets/secrets.yaml

Theory 3 — agenix sketch (alternative)

age.secrets.apiToken = {
  file = ../secrets/apiToken.age;
  owner = "myapp";
  mode = "0400";
};
# service references config.age.secrets.apiToken.path

Pick one house default for Capstone A. Dual systems double key ceremony without double safety.

Theory 4 — Proxy E2E definition of done

Step	Done means
Upstream	App listens localhost (or internal-only) port
Proxy	nginx/caddy/traefik terminates edge
Firewall	Only designed ports public (often 80/443 + SSH)
TLS	ACME or lab self-signed/internal CA or HTTP lab mode documented
Probe	`curl -f` through proxy hostname/port
Negative	Upstream port not world-open unless intentional
Regression	Test or script fails if upstream link breaks

nginx sketch

{ config, lib, ... }:
{
  services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;
    virtualHosts."lab.example.com" = {
      # forceSSL = true;
      # enableACME = true;
      locations."/" = {
        proxyPass = "http://127.0.0.1:8080";
        proxyWebsockets = true;
      };
    };
  };

  # Lab without public DNS: listen on localhost or use a self-signed cert.
  # networking.firewall.allowedTCPPorts = [ 80 443 ];
  networking.firewall.allowedTCPPorts = [ 80 ]; # lab HTTP mode example
}

TLS modes (pick and document in `docs/TLS.md`)

Mode	When
ACME / public DNS	Real hostname, ports reachable
Self-signed / internal CA	Lab realism without public DNS
HTTP-only lab	Acceptable for Capstone if prod path is written down

Do not block Capstone A forever on ACME in a NAT lab. Document the prod path honestly.

Theory 5 — Ordering and dependencies

secret decrypt (activation)
    → app service start (reads secret path)
    → proxy start / reload
    → health probe via front door

Use restartUnits on secrets, systemd After=/Requires= where needed, and in tests wait_for_unit for both app and proxy. Avoid racing proxy health before the app listens.

Store discipline reminder

# WRONG — leaks into world-readable store
environment.variables.API_TOKEN = "supersecret";

# RIGHT — path only
# EnvironmentFile = config.sops.secrets."myapp/api_token".path;

If the secret value appears in a derivation input string, assume compromise of the store path.

Theory 6 — Module shape for Capstone

Prefer typed options (Day 72) over one-off host spaghetti:

# modules/proxy.nix (sketch)
{ lib, config, ... }:
let cfg = config.roles.proxy; in
{
  options.roles.proxy = {
    enable = lib.mkEnableOption "edge reverse proxy";
    upstream = lib.mkOption {
      type = lib.types.str;
      default = "http://127.0.0.1:8080";
    };
    hostName = lib.mkOption {
      type = lib.types.str;
      default = "lab.example.com";
    };
  };
  config = lib.mkIf cfg.enable {
    # services.nginx.virtualHosts.${cfg.hostName} = …
  };
}

Same idea for roles.myapp consuming a secret path option.

Worked example — Proof commands

# After switch on lab:
sudo ls -la /run/secrets 2>/dev/null || sudo ls -la /run/secrets.d 2>/dev/null
# path conventions vary by sops-nix version—use config.sops.secrets.*.path
systemctl status myapp.service --no-pager
curl -fsS http://127.0.0.1:8080/ | head   # upstream (local)
curl -fsS -H "Host: lab.example.com" http://127.0.0.1/ | head  # front door

# Negative: plaintext token not in git
git grep -n "the-real-lab-token-value" && echo "LEAK" || echo "ok-no-plaintext"

# Firewall intent (from admin, adjust interface)
# nmap or curl to :8080 from outside should fail if designed closed

Lab 0 — Branch

mkdir -p ~/lab/90daysofx/02-nixos/day84
cd /path/to/capstone-flake
git checkout -b capstone/day84  # or continue capstone branch

Lab 1 — Secrets path E2E

Ensure age/ssh keys exist for the lab host; backup private material offline (Days 87–88 need this).
Generate a non-production random token.
Encrypt with sops-nix or agenix workflow; commit ciphertext only.
Wire module → service consumer via path.
nixos-rebuild switch / deploy; prove service healthy.

sudo systemctl restart myapp.service
systemctl is-active myapp.service
# logs should show authenticated behavior without printing the secret
journalctl -u myapp.service -n 50 --no-pager

Negative proofs:

git grep -nE 'api_token|BEGIN OPENSSH' -- ':!*.md' || true
# ensure secrets yaml in git is encrypted (sops metadata headers present)
head -n 5 secrets/secrets.yaml

Lab 2 — Proxy path E2E

Enable proxy module with typed options if still weak.
Upstream app on localhost only.
Firewall ports match design.
Choose TLS mode; write docs/TLS.md.
Prove front door:

curl -vk https://127.0.0.1/   # if TLS
curl -f http://127.0.0.1/     # if lab HTTP
# Confirm whether upstream :8080 is intentionally reachable externally

If upstream should not be public, confirm firewall denies external access to that port.

Lab 3 — Failure injection (mini)

Wrong secret path or mode → service fails → fix; note in playbook.
Wrong upstream port → 502/bad gateway → fix.
Keep host green at end of day.

# intentional break then restore — lab only
# e.g. point proxyPass at dead port, switch, observe, revert

Lab 4 — Tests (at least stub toward Day 86)

Extend or sketch Day 76 suite:

machine.wait_for_unit("myapp.service")
machine.wait_for_unit("nginx.service")  # or caddy
machine.wait_for_open_port(80)
machine.succeed("curl -f http://lab/")  # adjust hostnames

nix build .#checks.x86_64-linux.<test> -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day84/test.log

If full test must wait until Day 86, leave a failing checklist note with reason—not a silent gap.

Lab 5 — Docs & checklist

Update:

Doc	Content
`docs/ARCHITECTURE.md`	Secrets + edge data flow
`docs/SECRETS.md`	Who can decrypt; key locations; rotation sketch
`docs/TLS.md`	ACME vs lab mode
`docs/CAPSTONE-A.md`	Tick secrets + proxy with proof commands

Common gotchas

Symptom	What to do
Secret in Nix string	Forces store world-readable—use sops/age paths
ACME fail on lab	Don’t block Capstone; document offline/lab TLS
Proxy works, app ignores secret	Prove unit environment actually loads file
Permission denied on secret	owner/mode/group match unit `User=`
Tests need production keys	Dummy secrets in test module
Both sops and agenix “temporary”	Pick one
Upstream open on 0.0.0.0	Bind localhost; firewall
Encrypted file unreadable by host	Creation rules missing host recipient

Checkpoint

Encrypted secrets in git; plaintext not committed
Service consumes secret successfully (proof)
Private keys backed up offline
Proxy front door works with documented TLS/HTTP mode
Firewall matches design
Test updated or explicitly deferred to 86 with reason
CAPSTONE-A checklist updated
Three personal gotchas

Commit

git add .
git commit -m "day84: capstone secrets and reverse proxy e2e"

Write three personal gotchas before continuing.

Tomorrow

Day 85 — Capstone 3/5: deploy-rs or Colmena path solid; CI builds; preferably push binary cache.

Day 84 — Capstone 2/5 — secrets & proxy

Why this day exists

Theory 1 — Secrets E2E definition of done

Threat model (short)

Theory 2 — sops-nix sketch (preferred)

.sops.yaml creation rules (idea)

Theory 3 — agenix sketch (alternative)

Theory 4 — Proxy E2E definition of done

nginx sketch

TLS modes (pick and document in docs/TLS.md)

Theory 5 — Ordering and dependencies

Store discipline reminder

Theory 6 — Module shape for Capstone

Worked example — Proof commands

Lab 0 — Branch

Lab 1 — Secrets path E2E

Lab 2 — Proxy path E2E

Lab 3 — Failure injection (mini)

Lab 4 — Tests (at least stub toward Day 86)

Lab 5 — Docs & checklist

Common gotchas

Checkpoint

Commit

Tomorrow

`.sops.yaml` creation rules (idea)

TLS modes (pick and document in `docs/TLS.md`)