Day 85 — Capstone 3/5

Updated

July 30, 2026

Day 85 — Capstone 3/5 — deploy, CI, cache

Stage VIII · ~4h (implementation)
Goal: Make remote deploy (deploy-rs and/or Colmena) the normal path, and wire CI that builds Capstone artifacts and preferably pushes a binary cache.

Note

Baseline: flakes on nixos-26.05, secrets already on a path (Day 84). Today is remote delivery + automation—not new app features.

Why this day exists

nixos-rebuild --flake only on the box is not a deploy story. Capstone A requires a deploy path and CI that would catch breakage before you SSH in panic. Cache push turns CI minutes into laptop seconds (Days 53–54, 74).

Theory 1 — Deploy definition of done

Item Done means
Tool chosen deploy-rs or Colmena (both OK; one is enough for A)
Config in repo deploy.nodes or Colmena hive committed
Auth SSH key or deploy user documented
Magic One command updates lab host from admin machine
Failure UX Partial failure doesn’t leave mystery state; rollback known
Docs docs/DEPLOY.md with command + prerequisites + rollback

deploy-rs sketch

# flake outputs fragment — follow current deploy-rs docs for 26.05-era usage
{
  inputs.deploy-rs.url = "github:serokell/deploy-rs";
  # deploy-rs.inputs.nixpkgs.follows = "nixpkgs"; # often wise

  outputs = { self, nixpkgs, deploy-rs, ... }: {
    nixosConfigurations.lab = /* … */;
    deploy.nodes.lab = {
      hostname = "lab.example.com"; # or IP
      profiles.system = {
        user = "root";
        path = deploy-rs.lib.x86_64-linux.activate.nixos
          self.nixosConfigurations.lab;
      };
    };
    # optional: checks from deploy-rs lib
  };
}
# from admin machine with SSH access
deploy .#lab
# or: nix run github:serokell/deploy-rs -- .#lab

Colmena sketch

# hive via flake colmena output or hive.nix — illustrative
{
  meta.nixpkgs = import nixpkgs { system = "x86_64-linux"; };
  lab = { name, nodes, pkgs, ... }: {
    deployment.targetHost = "lab.example.com";
    imports = [ ./hosts/lab/configuration.nix ];
  };
}
colmena apply --on lab

Rollback awareness

Mechanism Notes
Boot generations nixos-rebuild --rollback / boot menu
Tool re-deploy of known tag Redeploy capstone-a-freeze later
Avoid Manual imperative package installs on lab

Document the fast path in docs/DEPLOY.md.

Theory 2 — CI definition of done

Item Done means
Trigger PR + main (or forge equivalent)
Nix install Pin installer/action reasonably
Build nix build lab toplevel and/or nix flake check
Cache Cachix / Attic / Harmonia / Magic Nix Cache with secrets in CI
Permissions Least privilege tokens
Docs docs/CI.md required; badge optional

GitHub Actions sketch (26.05-oriented)

name: ci
on:
  push:
    branches: [ main ]
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Nix
        uses: DeterminateSystems/nix-installer-action@main
        # prefer a pinned tag you have reviewed

      # Option A: Cachix
      # - uses: cachix/cachix-action@v15
      #   with:
      #     name: yourcache
      #     authToken: "${{ secrets.CACHIX_AUTH_TOKEN }}"

      # Option B: Magic Nix Cache (illustrative)
      # - uses: DeterminateSystems/magic-nix-cache-action@main

      - name: Build lab toplevel
        run: nix build .#nixosConfigurations.lab.config.system.build.toplevel -L

      - name: Flake check
        run: nix flake check -L

Adjust action versions to current upstream tags when you wire the repo; pin before Day 87 freeze.

Without public Cachix

Self-hosted Attic/Harmonia, or at least runner-local caching. If push is truly impossible, document the blocker in docs/CI.md and still build in CI—Capstone preferred-not-met with mitigation is better than fake green.

Theory 3 — CI vs deploy coupling

Pattern Notes
CI build only Minimum for Capstone A
CI build + cache push Preferred
CI deploy to lab Optional; needs secrets + approval gates
CD to prod on every main push Usually not Day 85

Prefer manual deploy for lab until tests are boringly green (Days 86–89). Automatic deploys of broken configs amplify outages.

Theory 4 — Trust, signing, and CI identity

Concern Practice
CI as trusted-users on your laptop No—don’t weaken local daemon for CI
Cache signing keys CI secrets only; never commit private signing keys
Substituters on lab Public keys in nix.settings.trusted-public-keys
Private flake inputs Deploy keys / fine-scoped tokens; not PATs in logs 
# lab / admin nix settings fragment
nix.settings = {
  substituters = [
    "https://cache.nixos.org"
    "https://yourcache.cachix.org"
  ];
  trusted-public-keys = [
    "cache.nixos.org-1:…"
    "yourcache.cachix.org-1:…"
  ];
};

Theory 5 — SSH deploy prerequisites

Deploy tools are thin wrappers around copy-closure + activate. Failures are often SSH, not Nix:

Check Command ideas
Reachability ssh lab hostname
Root or sudo activate user allowed to switch system
Host key stability known_hosts / pinning policy
Agent forwarding traps Prefer explicit IdentityFile
Same flake lock Admin deploys what CI built 
ssh -o BatchMode=yes lab 'nix --version && readlink -f /run/current-system'

Worked example — One-command deploy proof

# before
ssh lab 'readlink /run/current-system' | tee ~/lab/90daysofx/02-nixos/day85/before.txt

deploy .#lab 2>&1 | tee ~/lab/90daysofx/02-nixos/day85/deploy.log

# after
ssh lab 'readlink /run/current-system; systemctl --failed' \
  | tee ~/lab/90daysofx/02-nixos/day85/after.txt

Generation path should change (or explicitly no-op if identical). Services critical to Capstone should be active.

Lab 0 — Branch

mkdir -p ~/lab/90daysofx/02-nixos/day85
cd /path/to/capstone-flake

Lab 1 — Deploy path

  1. Wire deploy-rs or Colmena in the flake; lock inputs.
  2. From admin machine (or another VM), not only localhost hacks:
# deploy-rs
deploy .#lab 2>&1 | tee ~/lab/90daysofx/02-nixos/day85/deploy.log
# OR colmena
# colmena apply --on lab 2>&1 | tee ~/lab/90daysofx/02-nixos/day85/deploy.log
  1. Verify generation advanced + services healthy.
  2. Write docs/DEPLOY.md: exact commands, SSH requirements, rollback pointer, common errors.

Lab 2 — CI workflow

  1. Add workflow file (GitHub Actions or GitLab).
  2. Push branch; ensure CI runs on clean runner.
  3. Capture URL or log snippet in day85 notes.
  4. Fix impurities (IFD surprises, missing files, unpinned channels).
# local approximation of CI purity
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L --recreate-lock-file=false

Lab 3 — Cache push

  1. Create/use Cachix cache, Attic, Harmonia, or Magic Nix Cache.
  2. Add public key to substituters on lab + laptop.
  3. CI pushes; on laptop/admin:
nix build .#nixosConfigurations.lab.config.system.build.toplevel -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day85/substitute.log
# observe substitutes from your cache in the log when possible
  1. Document in docs/CI.md.

If blocked: section “Cache push blocked because…” with mitigation; keep CI builds.

Lab 4 — Checklist & architecture

  1. Tick Capstone A deploy + CI (+ cache if done) with proof.
  2. Update trust diagram: who signs cache, who can deploy.
  3. Ensure Day 84 secrets still decrypt after remote activate (common footgun: wrong host key on rebuilt VM).

Lab 5 — Failure tabletop (write answers)

Write short answers in day85 notes:

  1. CI green but deploy fails auth — steps?
  2. Cache serves bad path without valid signature — what should Nix do?
  3. How to deploy previous known-good quickly (generation vs retag)?
  4. CI cannot eval private input — options?

Lab 6 — Optional dry-run / diff habits

# build remotely without activating if your tool supports it
# deploy-rs: check flags in current --help for dry-run / magic-rollback
deploy --help | head

Prefer understanding magic rollback / activation failure behavior before freeze.

Common gotchas

Symptom What to do
deploy works only with local root Fix SSH / useLocalConnection assumptions
CI cannot eval private inputs Public pins or deploy keys
Cachix token in logs Rotate; fix masking
flake check builds everything forever Split/attr selection; still run critical checks
Colmena/deploy-rs nixpkgs skew inputs.*.follows = "nixpkgs" where sane
After deploy, secrets missing Host key / age key not on target
Cache not trusted trusted-public-keys + substituters
Unpinned Actions Pin before Day 87

Checkpoint

  • One-command deploy to lab documented and demonstrated
  • CI builds toplevel and/or checks on clean runner
  • Cache push working or explicit waiver with reason
  • docs/DEPLOY.md + docs/CI.md
  • CAPSTONE-A updated with proof
  • Tabletop answers written
  • Three personal gotchas

Commit

git add .
git commit -m "day85: capstone deploy path and CI cache"

Write three personal gotchas before continuing.

Tomorrow

Day 86 — Capstone 4/5: tests + operability docs complete (runbook, diagram, regression suite).