Day 28 — Capstone: Flake, Host & Secrets

Updated

July 30, 2025

Day 28 — Capstone: Flake, Host & Secrets

Day 83 — Capstone 1/5 implement

Stage VIII · ~4h (implementation)
Goal: Close the highest P0 gaps from Day 82 so the Capstone A skeleton is real: flake host role, module layout, HM or service users, and a live checklist that tracks proof—not vibes.

Important

Capstone A (required): flake host · HM or service users · sops/agenix · reverse proxy · deploy-rs/colmena · nixosTest · CI + cache.
Days 83–87 implement; 88 wipe/rebuild; 89 rollback+tests; 90 retro.

Why this day exists

Architecture review without commits is stationery. Day 83 turns GAPS.md P0s into code on a branch, with rebuilds you can demonstrate.

Theory — Working agreements for Stage VIII

Rule	Why
One change → build → commit	Bisectable; safer switch
Lab VM first	Protect daily driver
Checklist is law	`CAPSTONE-A.md` updated same day
No drive-by rewrites	Freeze is Day 87
Secrets never in git plaintext	Even “temporary”

Suggested repo shape (adapt, don’t dogma)

flake.nix
flake.lock
hosts/<name>/configuration.nix
modules/{proxy,common,users,…}.nix
homes/<user>.nix          # if HM
secrets/secrets.yaml      # sops-encrypted
tests/*.nix
deploy.nix or colmena hive
.github/workflows/ci.yml
docs/{ARCHITECTURE,CAPSTONE-A,RUNBOOK}.md

Capstone A checklist (live)

Copy into repo root if missing; tick as you go this week:

# CAPSTONE-A.md
- [ ] Host role rebuilds from flake
- [ ] HM user OR service user story
- [ ] sops-nix OR agenix wired; service reads secret
- [ ] Reverse proxy edge with TLS or lab HTTP mode documented
- [ ] deploy-rs OR colmena path works from admin machine
- [ ] nixosTest/check catches a real regression
- [ ] CI builds toplevel (and checks)
- [ ] CI pushes to binary cache (preferred)
- [ ] ARCHITECTURE.md current
- [ ] RUNBOOK.md for deploy/rollback
- [ ] Day 88: wipe + rebuild timed
- [ ] Day 89: rollback drill + suite green

Day 83 target scope

From GAPS.md, implement P0s that unblock structure, typically:

nixosConfigurations.<lab> evaluates and builds
Host modules split (common + role)
Users: interactive HM or system service user + data dir
Firewall baseline + SSH
Placeholder modules for proxy/secrets if not ready (interfaces only)—prefer real stubs that mkEnableOption cleanly

Defer full proxy TLS and CI cache to 84–85 if needed—but leave option shapes ready.

Worked example — Minimal host flake skeleton

# flake.nix (sketch)
{
  description = "capstone lab";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
    # home-manager.url = "github:nix-community/home-manager/release-26.05";
    # home-manager.inputs.nixpkgs.follows = "nixpkgs";
    # sops-nix.url = "github:Mic92/sops-nix";
    # sops-nix.inputs.nixpkgs.follows = "nixpkgs";
  };
  outputs = { self, nixpkgs, ... }@inputs: {
    nixosConfigurations.lab = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      specialArgs = { inherit inputs; };
      modules = [
        ./hosts/lab/configuration.nix
        ./modules/common.nix
        # inputs.sops-nix.nixosModules.sops
        # inputs.home-manager.nixosModules.home-manager
      ];
    };
  };
}

# modules/common.nix
{ lib, pkgs, ... }:
{
  time.timeZone = "UTC";
  networking.firewall.enable = true;
  services.openssh.enable = true;
  environment.systemPackages = with pkgs; [ vim git curl ];
}

# hosts/lab/configuration.nix
{ config, pkgs, ... }:
{
  imports = [ ./hardware.nix ];
  networking.hostName = "lab";
  # users / HM / roles
  system.stateVersion = "26.05"; # your real value
}

Lab 0 — Branch & checklist

cd /path/to/capstone-flake
git checkout -b capstone/day83
cp ~/lab/90daysofx/02-nixos/day82/CAPSTONE-A.md ./docs/CAPSTONE-A.md  # or merge
mkdir -p ~/lab/90daysofx/02-nixos/day83

Lab 1 — Build green host

nix flake show
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day83/build.log

Fix until green. Prefer build over switch until config is sane; then:

sudo nixos-rebuild switch --flake .#lab
# or deploy path if already wired

Lab 2 — Users story

Track HM (workstation-ish)

Enable HM as NixOS module
Manage shell + one dotfile
Prove with home-manager generations or files in place

Track service users (server-ish)

users.users.myapp = {
  isSystemUser = true;
  group = "myapp";
  home = "/var/lib/myapp";
  createHome = true;
};
users.groups.myapp = { };

Document which track in docs/CAPSTONE-A.md.

Lab 3 — Module hygiene from Day 72

Ensure at least one custom module uses:

typed options
mkEnableOption
mkIf
one assertion

Even if proxy is stubbed:

options.my.capstone.proxy.enable = lib.mkEnableOption "edge proxy";

Lab 4 — Update architecture & gaps

# edit docs/ARCHITECTURE.md paths if moved
# mark GAPS.md items closed with PR/commit refs

Write ~/lab/90daysofx/02-nixos/day83/NOTES.md: what shipped, what slipped to 84.

Lab 5 — Smoke

systemctl --failed
hostname
# ssh localhost or from client

Theory — Capstone A success definition

Day 83 is not “finish the whole series project.” It is:

Done	Not done
Green host from flake	Perfect theming
Users + SSH story	Every optional Stage IV service
Module hygiene applied	Multi-region fleet
Architecture doc updated	Final polish (later capstone days)

Scope control is part of the grade.

Theory — Branch discipline

git checkout -b capstone-a
# work
# merge only when smoke passes

Avoid committing broken main mid-capstone if you use the same repo for the lab host.

Worked example — smoke script

#!/usr/bin/env bash
set -euo pipefail
hostname
systemctl is-active sshd
curl -sf -o /dev/null -w '%{http_code}\n' http://127.0.0.1:… || true
nix flake metadata

Keep it boring; run after every significant rebuild.

Lab 6 — Gap register freeze for Day 83

Write docs/capstone-gaps.md with only items that remain for Days 84–87. Anything critical for a bootable admin host must be closed today.

Lab 7 — Teach-back outline (5 bullets)

How to rebuild
How to roll back
Where secrets live
What state to back up
What the next capstone day will add

Common gotchas

Symptom	What to do
Hardware config missing in flake	Import `hardware-configuration.nix` or disko
Infinite recursion after module split	Day 71 dependency cycles
HM + system package duplication	Pick layers deliberately
Switching daily driver	Stop; use VM
Checklist not updated	Same commit as code

Checkpoint

nix build toplevel green for lab host
HM or service user story demoable
Modules structured; ≥1 typed custom module
CAPSTONE-A.md updated honestly
GAPS.md P0s reduced or re-dated
Commits on capstone branch

Commit

git add .
git commit -m "day83: capstone host skeleton and user story"

Write three personal gotchas before continuing.

Tomorrow

Day 84 — Capstone 2/5: secrets + reverse proxy end-to-end; tests updated for the edge path.

Day 84 — Capstone 2/5 — secrets & proxy

Stage VIII · ~4h (implementation)
Goal: Complete secrets E2E (sops-nix or agenix) and the edge reverse-proxy path so a service is reachable through the front door without plaintext secrets in git.

Important

Use lab-only tokens. Never commit production credentials. Prefer sops-nix as house default (Day 34); agenix is fine if Capstone already standardized on it—do not run both permanently.

Why this day exists

Host skeletons without secrets train bad habits. Proxy without a TLS story (or documented lab HTTP mode) is not an edge pattern. Day 84 closes Capstone A items for secrets + reverse proxy with proof commands you can paste into the checklist.

Theory 1 — Secrets E2E definition of done

Step	Done means
Encrypt	`secrets/.yaml` (or age files) committed encrypted*
Keys	Machine age/ssh key or admin key decrypts on host
NixOS wiring	`sops.secrets.` or `age.secrets.` → path on disk
Consumer	systemd `LoadCredential` / `EnvironmentFile` / path read
Negative proof	Repo clone alone cannot read secret without keys
Test story	nixosTest uses dummy secret or test key (Days 76/86)
Offline custody	Private keys exist off the lab disk (Day 87–88)

Threat model (short)

Bad outcome	Control
Secret in git plaintext	SOPS/age encrypt before commit
Secret in Nix string → store	Reference paths, never secret values in modules
World-readable decrypted file	`owner` / `mode` / service user match
Lost keys after wipe	Offline backup of age/ssh private material

Theory 2 — sops-nix sketch (preferred)

# host module fragment — follow current sops-nix README for key paths
{ config, ... }:
{
  sops.defaultSopsFile = ./secrets/secrets.yaml;
  # common patterns: age ssh host key and/or dedicated age key file
  sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ];
  # sops.age.keyFile = "/var/lib/sops-nix/key.txt";

  sops.secrets."myapp/api_token" = {
    owner = "myapp";
    group = "myapp";
    mode = "0400";
    restartUnits = [ "myapp.service" ];
  };

  systemd.services.myapp = {
    description = "Capstone sample app";
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      User = "myapp";
      Group = "myapp";
      # Prefer credentials / EnvironmentFile pointing at the decrypted path:
      EnvironmentFile = config.sops.secrets."myapp/api_token".path;
      # Or ExecStart that reads the path—never pkgs.writeText with the token
    };
  };
}

`.sops.yaml` creation rules (idea)

Encrypt lab secrets to host recipient + admin recipient so you can edit on a workstation and the machine can decrypt at activation. Exact YAML keys follow current SOPS + sops-nix docs.

# edit encrypted file (opens $EDITOR with decrypted buffer)
sops secrets/secrets.yaml

Theory 3 — agenix sketch (alternative)

age.secrets.apiToken = {
  file = ../secrets/apiToken.age;
  owner = "myapp";
  mode = "0400";
};
# service references config.age.secrets.apiToken.path

Pick one house default for Capstone A. Dual systems double key ceremony without double safety.

Theory 4 — Proxy E2E definition of done

Step	Done means
Upstream	App listens localhost (or internal-only) port
Proxy	nginx/caddy/traefik terminates edge
Firewall	Only designed ports public (often 80/443 + SSH)
TLS	ACME or lab self-signed/internal CA or HTTP lab mode documented
Probe	`curl -f` through proxy hostname/port
Negative	Upstream port not world-open unless intentional
Regression	Test or script fails if upstream link breaks

nginx sketch

{ config, lib, ... }:
{
  services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;
    virtualHosts."lab.example.com" = {
      # forceSSL = true;
      # enableACME = true;
      locations."/" = {
        proxyPass = "http://127.0.0.1:8080";
        proxyWebsockets = true;
      };
    };
  };

  # Lab without public DNS: listen on localhost or use a self-signed cert.
  # networking.firewall.allowedTCPPorts = [ 80 443 ];
  networking.firewall.allowedTCPPorts = [ 80 ]; # lab HTTP mode example
}

TLS modes (pick and document in `docs/TLS.md`)

Mode	When
ACME / public DNS	Real hostname, ports reachable
Self-signed / internal CA	Lab realism without public DNS
HTTP-only lab	Acceptable for Capstone if prod path is written down

Do not block Capstone A forever on ACME in a NAT lab. Document the prod path honestly.

Theory 5 — Ordering and dependencies

secret decrypt (activation)
    → app service start (reads secret path)
    → proxy start / reload
    → health probe via front door

Use restartUnits on secrets, systemd After=/Requires= where needed, and in tests wait_for_unit for both app and proxy. Avoid racing proxy health before the app listens.

Store discipline reminder

# WRONG — leaks into world-readable store
environment.variables.API_TOKEN = "supersecret";

# RIGHT — path only
# EnvironmentFile = config.sops.secrets."myapp/api_token".path;

If the secret value appears in a derivation input string, assume compromise of the store path.

Theory 6 — Module shape for Capstone

Prefer typed options (Day 72) over one-off host spaghetti:

# modules/proxy.nix (sketch)
{ lib, config, ... }:
let cfg = config.roles.proxy; in
{
  options.roles.proxy = {
    enable = lib.mkEnableOption "edge reverse proxy";
    upstream = lib.mkOption {
      type = lib.types.str;
      default = "http://127.0.0.1:8080";
    };
    hostName = lib.mkOption {
      type = lib.types.str;
      default = "lab.example.com";
    };
  };
  config = lib.mkIf cfg.enable {
    # services.nginx.virtualHosts.${cfg.hostName} = …
  };
}

Same idea for roles.myapp consuming a secret path option.

Worked example — Proof commands

# After switch on lab:
sudo ls -la /run/secrets 2>/dev/null || sudo ls -la /run/secrets.d 2>/dev/null
# path conventions vary by sops-nix version—use config.sops.secrets.*.path
systemctl status myapp.service --no-pager
curl -fsS http://127.0.0.1:8080/ | head   # upstream (local)
curl -fsS -H "Host: lab.example.com" http://127.0.0.1/ | head  # front door

# Negative: plaintext token not in git
git grep -n "the-real-lab-token-value" && echo "LEAK" || echo "ok-no-plaintext"

# Firewall intent (from admin, adjust interface)
# nmap or curl to :8080 from outside should fail if designed closed

Lab 0 — Branch

mkdir -p ~/lab/90daysofx/02-nixos/day84
cd /path/to/capstone-flake
git checkout -b capstone/day84  # or continue capstone branch

Lab 1 — Secrets path E2E

Ensure age/ssh keys exist for the lab host; backup private material offline (Days 87–88 need this).
Generate a non-production random token.
Encrypt with sops-nix or agenix workflow; commit ciphertext only.
Wire module → service consumer via path.
nixos-rebuild switch / deploy; prove service healthy.

sudo systemctl restart myapp.service
systemctl is-active myapp.service
# logs should show authenticated behavior without printing the secret
journalctl -u myapp.service -n 50 --no-pager

Negative proofs:

git grep -nE 'api_token|BEGIN OPENSSH' -- ':!*.md' || true
# ensure secrets yaml in git is encrypted (sops metadata headers present)
head -n 5 secrets/secrets.yaml

Lab 2 — Proxy path E2E

Enable proxy module with typed options if still weak.
Upstream app on localhost only.
Firewall ports match design.
Choose TLS mode; write docs/TLS.md.
Prove front door:

curl -vk https://127.0.0.1/   # if TLS
curl -f http://127.0.0.1/     # if lab HTTP
# Confirm whether upstream :8080 is intentionally reachable externally

If upstream should not be public, confirm firewall denies external access to that port.

Lab 3 — Failure injection (mini)

Wrong secret path or mode → service fails → fix; note in playbook.
Wrong upstream port → 502/bad gateway → fix.
Keep host green at end of day.

# intentional break then restore — lab only
# e.g. point proxyPass at dead port, switch, observe, revert

Lab 4 — Tests (at least stub toward Day 86)

Extend or sketch Day 76 suite:

machine.wait_for_unit("myapp.service")
machine.wait_for_unit("nginx.service")  # or caddy
machine.wait_for_open_port(80)
machine.succeed("curl -f http://lab/")  # adjust hostnames

nix build .#checks.x86_64-linux.<test> -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day84/test.log

If full test must wait until Day 86, leave a failing checklist note with reason—not a silent gap.

Lab 5 — Docs & checklist

Update:

Doc	Content
`docs/ARCHITECTURE.md`	Secrets + edge data flow
`docs/SECRETS.md`	Who can decrypt; key locations; rotation sketch
`docs/TLS.md`	ACME vs lab mode
`docs/CAPSTONE-A.md`	Tick secrets + proxy with proof commands

Common gotchas

Symptom	What to do
Secret in Nix string	Forces store world-readable—use sops/age paths
ACME fail on lab	Don’t block Capstone; document offline/lab TLS
Proxy works, app ignores secret	Prove unit environment actually loads file
Permission denied on secret	owner/mode/group match unit `User=`
Tests need production keys	Dummy secrets in test module
Both sops and agenix “temporary”	Pick one
Upstream open on 0.0.0.0	Bind localhost; firewall
Encrypted file unreadable by host	Creation rules missing host recipient

Checkpoint

Encrypted secrets in git; plaintext not committed
Service consumes secret successfully (proof)
Private keys backed up offline
Proxy front door works with documented TLS/HTTP mode
Firewall matches design
Test updated or explicitly deferred to 86 with reason
CAPSTONE-A checklist updated
Three personal gotchas

Commit

git add .
git commit -m "day84: capstone secrets and reverse proxy e2e"

Write three personal gotchas before continuing.

Tomorrow

Day 85 — Capstone 3/5: deploy-rs or Colmena path solid; CI builds; preferably push binary cache.

Day 85 — Capstone 3/5 — deploy, CI, cache

Stage VIII · ~4h (implementation)
Goal: Make remote deploy (deploy-rs and/or Colmena) the normal path, and wire CI that builds Capstone artifacts and preferably pushes a binary cache.

Note

Baseline: flakes on nixos-26.05, secrets already on a path (Day 84). Today is remote delivery + automation—not new app features.

Why this day exists

nixos-rebuild --flake only on the box is not a deploy story. Capstone A requires a deploy path and CI that would catch breakage before you SSH in panic. Cache push turns CI minutes into laptop seconds (Days 53–54, 74).

Theory 1 — Deploy definition of done

Item	Done means
Tool chosen	deploy-rs or Colmena (both OK; one is enough for A)
Config in repo	`deploy.nodes` or Colmena hive committed
Auth	SSH key or deploy user documented
Magic	One command updates lab host from admin machine
Failure UX	Partial failure doesn’t leave mystery state; rollback known
Docs	`docs/DEPLOY.md` with command + prerequisites + rollback

deploy-rs sketch

# flake outputs fragment — follow current deploy-rs docs for 26.05-era usage
{
  inputs.deploy-rs.url = "github:serokell/deploy-rs";
  # deploy-rs.inputs.nixpkgs.follows = "nixpkgs"; # often wise

  outputs = { self, nixpkgs, deploy-rs, ... }: {
    nixosConfigurations.lab = /* … */;
    deploy.nodes.lab = {
      hostname = "lab.example.com"; # or IP
      profiles.system = {
        user = "root";
        path = deploy-rs.lib.x86_64-linux.activate.nixos
          self.nixosConfigurations.lab;
      };
    };
    # optional: checks from deploy-rs lib
  };
}

# from admin machine with SSH access
deploy .#lab
# or: nix run github:serokell/deploy-rs -- .#lab

Colmena sketch

# hive via flake colmena output or hive.nix — illustrative
{
  meta.nixpkgs = import nixpkgs { system = "x86_64-linux"; };
  lab = { name, nodes, pkgs, ... }: {
    deployment.targetHost = "lab.example.com";
    imports = [ ./hosts/lab/configuration.nix ];
  };
}

colmena apply --on lab

Rollback awareness

Mechanism	Notes
Boot generations	`nixos-rebuild --rollback` / boot menu
Tool re-deploy of known tag	Redeploy `capstone-a-freeze` later
Avoid	Manual imperative package installs on lab

Document the fast path in docs/DEPLOY.md.

Theory 2 — CI definition of done

Item	Done means
Trigger	PR + main (or forge equivalent)
Nix install	Pin installer/action reasonably
Build	`nix build` lab toplevel and/or `nix flake check`
Cache	Cachix / Attic / Harmonia / Magic Nix Cache with secrets in CI
Permissions	Least privilege tokens
Docs	`docs/CI.md` required; badge optional

GitHub Actions sketch (26.05-oriented)

name: ci
on:
  push:
    branches: [ main ]
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Nix
        uses: DeterminateSystems/nix-installer-action@main
        # prefer a pinned tag you have reviewed

      # Option A: Cachix
      # - uses: cachix/cachix-action@v15
      #   with:
      #     name: yourcache
      #     authToken: "${{ secrets.CACHIX_AUTH_TOKEN }}"

      # Option B: Magic Nix Cache (illustrative)
      # - uses: DeterminateSystems/magic-nix-cache-action@main

      - name: Build lab toplevel
        run: nix build .#nixosConfigurations.lab.config.system.build.toplevel -L

      - name: Flake check
        run: nix flake check -L

Adjust action versions to current upstream tags when you wire the repo; pin before Day 87 freeze.

Without public Cachix

Self-hosted Attic/Harmonia, or at least runner-local caching. If push is truly impossible, document the blocker in docs/CI.md and still build in CI—Capstone preferred-not-met with mitigation is better than fake green.

Theory 3 — CI vs deploy coupling

Pattern	Notes
CI build only	Minimum for Capstone A
CI build + cache push	Preferred
CI deploy to lab	Optional; needs secrets + approval gates
CD to prod on every main push	Usually not Day 85

Prefer manual deploy for lab until tests are boringly green (Days 86–89). Automatic deploys of broken configs amplify outages.

Theory 4 — Trust, signing, and CI identity

Concern	Practice
CI as `trusted-users` on your laptop	No—don’t weaken local daemon for CI
Cache signing keys	CI secrets only; never commit private signing keys
Substituters on lab	Public keys in `nix.settings.trusted-public-keys`
Private flake inputs	Deploy keys / fine-scoped tokens; not PATs in logs

# lab / admin nix settings fragment
nix.settings = {
  substituters = [
    "https://cache.nixos.org"
    "https://yourcache.cachix.org"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:…"
    "yourcache.cachix.org-1:…"
  ];
};

Theory 5 — SSH deploy prerequisites

Deploy tools are thin wrappers around copy-closure + activate. Failures are often SSH, not Nix:

Check	Command ideas
Reachability	`ssh lab hostname`
Root or sudo activate	user allowed to switch system
Host key stability	`known_hosts` / pinning policy
Agent forwarding traps	Prefer explicit IdentityFile
Same flake lock	Admin deploys what CI built

ssh -o BatchMode=yes lab 'nix --version && readlink -f /run/current-system'

Worked example — One-command deploy proof

# before
ssh lab 'readlink /run/current-system' | tee ~/lab/90daysofx/02-nixos/day85/before.txt

deploy .#lab 2>&1 | tee ~/lab/90daysofx/02-nixos/day85/deploy.log

# after
ssh lab 'readlink /run/current-system; systemctl --failed' \
  | tee ~/lab/90daysofx/02-nixos/day85/after.txt

Generation path should change (or explicitly no-op if identical). Services critical to Capstone should be active.

Lab 0 — Branch

mkdir -p ~/lab/90daysofx/02-nixos/day85
cd /path/to/capstone-flake

Lab 1 — Deploy path

Wire deploy-rs or Colmena in the flake; lock inputs.
From admin machine (or another VM), not only localhost hacks:

# deploy-rs
deploy .#lab 2>&1 | tee ~/lab/90daysofx/02-nixos/day85/deploy.log
# OR colmena
# colmena apply --on lab 2>&1 | tee ~/lab/90daysofx/02-nixos/day85/deploy.log

Verify generation advanced + services healthy.
Write docs/DEPLOY.md: exact commands, SSH requirements, rollback pointer, common errors.

Lab 2 — CI workflow

Add workflow file (GitHub Actions or GitLab).
Push branch; ensure CI runs on clean runner.
Capture URL or log snippet in day85 notes.
Fix impurities (IFD surprises, missing files, unpinned channels).

# local approximation of CI purity
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L --recreate-lock-file=false

Lab 3 — Cache push

Create/use Cachix cache, Attic, Harmonia, or Magic Nix Cache.
Add public key to substituters on lab + laptop.
CI pushes; on laptop/admin:

nix build .#nixosConfigurations.lab.config.system.build.toplevel -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day85/substitute.log
# observe substitutes from your cache in the log when possible

Document in docs/CI.md.

If blocked: section “Cache push blocked because…” with mitigation; keep CI builds.

Lab 4 — Checklist & architecture

Tick Capstone A deploy + CI (+ cache if done) with proof.
Update trust diagram: who signs cache, who can deploy.
Ensure Day 84 secrets still decrypt after remote activate (common footgun: wrong host key on rebuilt VM).

Lab 5 — Failure tabletop (write answers)

Write short answers in day85 notes:

CI green but deploy fails auth — steps?
Cache serves bad path without valid signature — what should Nix do?
How to deploy previous known-good quickly (generation vs retag)?
CI cannot eval private input — options?

Lab 6 — Optional dry-run / diff habits

# build remotely without activating if your tool supports it
# deploy-rs: check flags in current --help for dry-run / magic-rollback
deploy --help | head

Prefer understanding magic rollback / activation failure behavior before freeze.

Common gotchas

Symptom	What to do
deploy works only with local root	Fix SSH / `useLocalConnection` assumptions
CI cannot eval private inputs	Public pins or deploy keys
Cachix token in logs	Rotate; fix masking
`flake check` builds everything forever	Split/attr selection; still run critical checks
Colmena/deploy-rs nixpkgs skew	`inputs.*.follows = "nixpkgs"` where sane
After deploy, secrets missing	Host key / age key not on target
Cache not trusted	`trusted-public-keys` + substituters
Unpinned Actions	Pin before Day 87

Checkpoint

One-command deploy to lab documented and demonstrated
CI builds toplevel and/or checks on clean runner
Cache push working or explicit waiver with reason
docs/DEPLOY.md + docs/CI.md
CAPSTONE-A updated with proof
Tabletop answers written
Three personal gotchas

Commit

git add .
git commit -m "day85: capstone deploy path and CI cache"

Write three personal gotchas before continuing.

Tomorrow

Day 86 — Capstone 4/5: tests + operability docs complete (runbook, diagram, regression suite).

Day 28 — Capstone: Flake, Host & Secrets

Day 83 — Capstone 1/5 implement

Why this day exists

Theory — Working agreements for Stage VIII

Suggested repo shape (adapt, don’t dogma)

Capstone A checklist (live)

Day 83 target scope

Worked example — Minimal host flake skeleton

Lab 0 — Branch & checklist

Lab 1 — Build green host

Lab 2 — Users story

Track HM (workstation-ish)

Track service users (server-ish)

Lab 3 — Module hygiene from Day 72

Lab 4 — Update architecture & gaps

Lab 5 — Smoke

Theory — Capstone A success definition

Theory — Branch discipline

Worked example — smoke script

Lab 6 — Gap register freeze for Day 83

Lab 7 — Teach-back outline (5 bullets)

Common gotchas

Checkpoint

Commit

Tomorrow

Day 84 — Capstone 2/5 — secrets & proxy

Why this day exists

Theory 1 — Secrets E2E definition of done

Threat model (short)

Theory 2 — sops-nix sketch (preferred)

.sops.yaml creation rules (idea)

Theory 3 — agenix sketch (alternative)

Theory 4 — Proxy E2E definition of done

nginx sketch

TLS modes (pick and document in docs/TLS.md)

Theory 5 — Ordering and dependencies

Store discipline reminder

Theory 6 — Module shape for Capstone

Worked example — Proof commands

Lab 0 — Branch

Lab 1 — Secrets path E2E

Lab 2 — Proxy path E2E

Lab 3 — Failure injection (mini)

Lab 4 — Tests (at least stub toward Day 86)

Lab 5 — Docs & checklist

Common gotchas

Checkpoint

Commit

Tomorrow

Day 85 — Capstone 3/5 — deploy, CI, cache

Why this day exists

Theory 1 — Deploy definition of done

deploy-rs sketch

Colmena sketch

Rollback awareness

Theory 2 — CI definition of done

GitHub Actions sketch (26.05-oriented)

Without public Cachix

Theory 3 — CI vs deploy coupling

Theory 4 — Trust, signing, and CI identity

Theory 5 — SSH deploy prerequisites

Worked example — One-command deploy proof

Lab 0 — Branch

Lab 1 — Deploy path

Lab 2 — CI workflow

Lab 3 — Cache push

Lab 4 — Checklist & architecture

Lab 5 — Failure tabletop (write answers)

Lab 6 — Optional dry-run / diff habits

Common gotchas

Checkpoint

Commit

Tomorrow

`.sops.yaml` creation rules (idea)

TLS modes (pick and document in `docs/TLS.md`)