Capstone: implement

Updated

July 30, 2026

Capstone: implement

Goal: Complete secrets E2E (sops-nix or agenix) and the edge reverse-proxy path so a service is reachable through the front door without plaintext secrets in git.

Important

Use lab-only tokens. Never commit production credentials. Prefer sops-nix as house default; agenix is fine if Capstone already standardized on it—do not run both permanently.

Why this chapter exists

Host skeletons without secrets train bad habits. Proxy without a TLS story (or documented lab HTTP mode) is not an edge pattern. This chapter closes Capstone A items for secrets + reverse proxy with proof commands you can paste into the checklist.

Theory 1 — Secrets E2E definition of done

Step Done means
Encrypt secrets/*.yaml (or age files) committed encrypted
Keys Machine age/ssh key or admin key decrypts on host
NixOS wiring sops.secrets.* or age.secrets.* → path on disk
Consumer systemd LoadCredential / EnvironmentFile / path read
Negative proof Repo clone alone cannot read secret without keys
Test story nixosTest uses dummy secret or test key
Offline custody Private keys exist off the lab disk

Threat model (short)

Bad outcome Control
Secret in git plaintext SOPS/age encrypt before commit
Secret in Nix string → store Reference paths, never secret values in modules
World-readable decrypted file owner / mode / service user match
Lost keys after wipe Offline backup of age/ssh private material

Theory 2 — sops-nix sketch (preferred)

# host module fragment — follow current sops-nix README for key paths
{ config, ... }:
{
  sops.defaultSopsFile = ./secrets/secrets.yaml;
  # common patterns: age ssh host key and/or dedicated age key file
  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  # sops.age.keyFile = "/var/lib/sops-nix/key.txt";

  sops.secrets."myapp/api_token" = {
    owner = "myapp";
    group = "myapp";
    mode = "0400";
    restartUnits = [ "myapp.service" ];
  };

  systemd.services.myapp = {
    description = "Capstone sample app";
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "myapp";
      Group = "myapp";
      EnvironmentFile = config.sops.secrets."myapp/api_token".path;
    };
  };
}

.sops.yaml creation rules (idea)

Encrypt lab secrets to host recipient + admin recipient so you can edit on a workstation and the machine can decrypt at activation.

sops secrets/secrets.yaml

Theory 3 — agenix sketch (alternative)

age.secrets.apiToken = {
  file = ../secrets/apiToken.age;
  owner = "myapp";
  mode = "0400";
};
# service references config.age.secrets.apiToken.path

Pick one house default for Capstone A.

Theory 4 — Proxy E2E definition of done

Step Done means
Upstream App listens localhost (or internal-only) port
Proxy nginx/caddy/traefik terminates edge
Firewall Only designed ports public (often 80/443 + SSH)
TLS ACME or lab self-signed/internal CA or HTTP lab mode documented
Probe curl -f through proxy hostname/port
Negative Upstream port not world-open unless intentional
Regression Test or script fails if upstream link breaks

nginx sketch

{ config, lib, ... }:
{
  services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;
    virtualHosts."lab.example.com" = {
      # forceSSL = true;
      # enableACME = true;
      locations."/" = {
        proxyPass = "http://127.0.0.1:8080";
        proxyWebsockets = true;
      };
    };
  };
  networking.firewall.allowedTCPPorts = [ 80 ]; # lab HTTP mode example
}

TLS modes (pick and document in docs/TLS.md)

Mode When
ACME / public DNS Real hostname, ports reachable
Self-signed / internal CA Lab realism without public DNS
HTTP-only lab Acceptable for Capstone if prod path is written down

Theory 5 — Ordering and dependencies

secret decrypt (activation)
    → app service start (reads secret path)
    → proxy start / reload
    → health probe via front door

Use restartUnits on secrets, systemd After=/Requires= where needed, and in tests wait_for_unit for both app and proxy.

Store discipline reminder

# WRONG — leaks into world-readable store
environment.variables.API_TOKEN = "supersecret";

# RIGHT — path only
# EnvironmentFile = config.sops.secrets."myapp/api_token".path;

Theory 6 — Module shape for Capstone

# modules/proxy.nix (sketch)
{ lib, config, ... }:
let cfg = config.roles.proxy; in
{
  options.roles.proxy = {
    enable = lib.mkEnableOption "edge reverse proxy";
    upstream = lib.mkOption {
      type = lib.types.str;
      default = "http://127.0.0.1:8080";
    };
    hostName = lib.mkOption {
      type = lib.types.str;
      default = "lab.example.com";
    };
  };
  config = lib.mkIf cfg.enable {
    # services.nginx.virtualHosts.${cfg.hostName} = …
  };
}

Concrete checklists

Secrets checklist

  • Lab token generated (not production)
  • Ciphertext committed; plaintext never staged
  • Host decrypts after switch
  • Unit user can read path; others cannot
  • Offline private key backup exists
  • Negative: git grep no raw token

Proxy checklist

  • Upstream localhost-only (or documented)
  • Front door probe succeeds
  • Firewall matches design
  • docs/TLS.md written
  • Negative: external upstream port closed if required

Worked example — Proof commands

sudo ls -la /run/secrets 2>/dev/null || sudo ls -la /run/secrets.d 2>/dev/null
systemctl status myapp.service --no-pager
curl -fsS http://127.0.0.1:8080/ | head
curl -fsS -H "Host: lab.example.com" http://127.0.0.1/ | head
git grep -n "the-real-lab-token-value" && echo "LEAK" || echo "ok-no-plaintext"

Lab 0 — Branch

mkdir -p ~/lab/nixos/capstone/implement
cd /path/to/capstone-flake
git checkout capstone-a  # continue Capstone branch

Lab 1 — Secrets path E2E

  1. Ensure age/ssh keys exist for the lab host; backup private material offline.
  2. Generate a non-production random token.
  3. Encrypt with sops-nix or agenix workflow; commit ciphertext only.
  4. Wire module → service consumer via path.
  5. Switch/deploy; prove service healthy.
sudo systemctl restart myapp.service
systemctl is-active myapp.service
journalctl -u myapp.service -n 50 --no-pager
head -n 5 secrets/secrets.yaml

Acceptance: Service active; secret path exists; git has ciphertext only.

Lab 2 — Proxy path E2E

  1. Enable proxy module with typed options if still weak.
  2. Upstream app on localhost only.
  3. Firewall ports match design.
  4. Choose TLS mode; write docs/TLS.md.
  5. Prove front door with curl.

Lab 3 — Failure injection (mini)

  1. Wrong secret path or mode → service fails → fix; note in playbook.
  2. Wrong upstream port → 502 → fix.
  3. Leave host green at end.

Lab 4 — Tests stub toward prove-rebuild chapter

machine.wait_for_unit("myapp.service")
machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)
machine.succeed("curl -f http://lab/")
nix build .#checks.x86_64-linux.<test> -L \
  2>&1 | tee ~/lab/nixos/capstone/implement/test.log

If full test must wait, leave a failing checklist note with reason—not a silent gap.

Lab 5 — Docs & checklist

Doc Content
docs/ARCHITECTURE.md Secrets + edge data flow
docs/SECRETS.md Who can decrypt; key locations; rotation sketch
docs/TLS.md ACME vs lab mode
docs/CAPSTONE-A.md Tick secrets + proxy with proof commands

Lab 6 — Offline key custody proof

# Document location of offline age/ssh private material
# Do NOT commit private keys
# Verify you can open the offline backup medium

Write path (physical) in docs/SECRETS.md offline section.

Common gotchas

Symptom What to do
Secret in Nix string Use sops/age paths only
ACME fail on lab Document offline/lab TLS; don’t block Capstone
Proxy works, app ignores secret Prove unit environment loads file
Permission denied on secret owner/mode/group match unit User=
Encrypted file looks empty Confirm sops metadata headers

Checkpoint

  • Secrets E2E with negative git proof
  • Proxy front door probe
  • docs/SECRETS.md + docs/TLS.md
  • Offline keys backed up
  • CAPSTONE-A ticks for secrets + proxy with commands
  • Host green

Commit

git add .
git commit -m "capstone: secrets and reverse-proxy E2E"

Write three personal gotchas before continuing.

Next

→ Capstone: harden and document — remote deploy, CI, cache.