Capstone: harden and document

Updated

July 30, 2026

Capstone: harden and document

Goal: Make remote deploy (deploy-rs and/or Colmena) the normal path, and wire CI that builds Capstone artifacts and preferably pushes a binary cache.

Note

Baseline: flakes on nixos-26.05, secrets already on a path. This chapter is remote delivery + automation—not new app features.

Why this chapter exists

nixos-rebuild --flake only on the box is not a deploy story. Capstone A requires a deploy path and CI that would catch breakage before you SSH in panic. Cache push turns CI minutes into laptop seconds.

Theory 1 — Deploy definition of done

Item Done means
Tool chosen deploy-rs or Colmena (both OK; one is enough for A)
Config in repo deploy.nodes or Colmena hive committed
Auth SSH key or deploy user documented
Magic One command updates lab host from admin machine
Failure UX Partial failure doesn’t leave mystery state; rollback known
Docs docs/DEPLOY.md with command + prerequisites + rollback

deploy-rs sketch

# flake outputs fragment — follow current deploy-rs docs for 26.05-era usage
{
  inputs.deploy-rs.url = "github:serokell/deploy-rs";
  outputs = { self, nixpkgs, deploy-rs, ... }: {
    nixosConfigurations.lab = /* … */;
    deploy.nodes.lab = {
      hostname = "lab.local"; # or IP
      profiles.system = {
        user = "root";
        path = deploy-rs.lib.x86_64-linux.activate.nixos self.nixosConfigurations.lab;
      };
    };
  };
}
deploy .#lab
# or: nix run github:serokell/deploy-rs -- .#lab

Colmena sketch

# hive.nix idea
{
  meta.nixpkgs = import (builtins.getFlake "…").inputs.nixpkgs { };
  lab = { … imports … };
}
colmena apply --on lab

Pick one primary path; document the other as optional.

Theory 2 — CI definition of done

Item Done means
Workflow file .github/workflows/ci.yml (or Forgejo/Gitea equivalent)
Triggers PR + push to main (or Capstone branch policy)
Build nix build toplevel and/or nix flake check
Logs Failures visible in CI UI
Cache (preferred) Push to Cachix/Attic/etc. with secrets in CI vault
Pin Nix installer version documented

Minimal workflow sketch

name: ci
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: cachix/install-nix-action@v27
        with:
          nix_path: nixpkgs=channel:nixos-26.05
      # - uses: cachix/cachix-action@v15
      #   with: { name: mycache, authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}' }
      - run: nix build .#nixosConfigurations.lab.config.system.build.toplevel -L
      - run: nix flake check -L

Adapt for KVM runners if nixosTest requires nested virt; otherwise split fast checks vs nightly tests.

Theory 3 — Cache push value

Without project cache With project cache
Every agent rebuilds your overlays Shared substitutes
Laptop recompiles CI work Laptop substitutes CI outputs
Slow feedback Minutes matter less

If you cannot push a cache yet, document waiver and still build in CI. Preferred: push.

Theory 4 — SSH deploy auth checklist

Check Note
Deploy key or agent Non-interactive preferred for CI deploy (optional)
root vs sudo user Match deploy tool docs
Host key verification known_hosts managed deliberately
Network path Admin machine can reach lab
Firewall SSH open; no accidental lockout without console

Theory 5 — RUNBOOK / DEPLOY docs minimum

docs/DEPLOY.md:

  1. Prerequisites (keys, network, flake path)
  2. Happy path one-liner
  3. Dry-run / build-only variant
  4. Failure symptoms
  5. Rollback pointer (docs/RUNBOOK.md)
  6. Who is allowed to deploy

Concrete checklists

Deploy checklist

  • Tool chosen and committed
  • One-command deploy from admin machine
  • Deploy log saved
  • Rollback path written
  • docs/DEPLOY.md complete

CI checklist

  • Workflow runs on PR
  • Toplevel or checks green
  • Cache push or explicit waiver
  • Nix version noted
  • Badge or URL in CAPSTONE-A

Lab 0 — Workspace

mkdir -p ~/lab/nixos/capstone/harden
cd /path/to/capstone-flake

Snapshot lab VM before first remote deploy if deploy is new.

Lab 1 — Wire deploy tool

  1. Add deploy-rs or Colmena input.
  2. Point at lab host.
  3. Deploy once from admin machine.
  4. Capture log:
deploy .#lab 2>&1 | tee ~/lab/nixos/capstone/harden/deploy.log

Acceptance: Remote activation succeeds; host healthy after.

Lab 2 — CI workflow

  1. Add workflow file.
  2. Push branch; observe CI.
  3. Fix until green.
  4. Optionally add Cachix/Attic push.
# local parity
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L
nix flake check -L

Lab 3 — Hardening pass (light)

Do not start new apps. Optionally:

  • SSH: password auth off if not already
  • Firewall review
  • systemctl --failed clean
  • Fail2ban only if already planned

Document changes in ARCHITECTURE.

Lab 4 — Docs

Doc Update
docs/DEPLOY.md Full path
docs/RUNBOOK.md Deploy + rollback stubs
docs/CAPSTONE-A.md Tick deploy + CI
docs/TRUST.md CI cache keys / trusted-users if changed

Lab 5 — Failure tabletop

Write without executing production:

  1. Deploy fails mid-activation → what do you do?
  2. CI red on main → who unblocks?
  3. Cache token leaked → rotation steps

Lab 6 — Elitebook admin path

If admin machine is the Elitebook host and target is guest:

# from host
ping -c1 <guest>
ssh root@<guest> true
deploy .#lab   # or colmena

Acceptance: Works over libvirt network without manual nixos-rebuild on guest.

Common gotchas

Symptom Fix
SSH permission denied Keys, user, PermitRootLogin policy
deploy evaluates wrong system system/attr mismatch
CI OOM lower jobs; cache; split matrix
nixosTest needs KVM on GA nightly self-hosted or skip in PR
Cache auth fails secrets scope; trusted-public-keys on consumers

Checkpoint

  • Remote deploy one-liner works
  • CI builds Capstone artifacts
  • Cache push or waiver documented
  • DEPLOY + RUNBOOK updated
  • CAPSTONE-A ticks with proof
  • Three personal gotchas

Commit

git add .
git commit -m "capstone: deploy path and CI hardening"

Write three personal gotchas before continuing.

Next

→ Capstone: prove rebuildability — tests, runbook, diagrams.