Day 22 — Terraform, nixosTest & CI Pipeline

Updated

July 30, 2025

Day 22 — Terraform, nixosTest & CI Pipeline

Day 63 — Terraform boundary

Stage VI · ~4h (design + light lab)
Goal: Draw a clean boundary—Terraform (or OpenTofu) provisions cloud resources; NixOS owns system configuration—and produce a sketch you could implement without turning TF into a second module system. Images from Day 62 + deploy from Days 59–60 sit on the Nix side; pin OS stories to 26.05.

Note

Full multi-cloud automation is out of scope. A sharp sketch with one example resource graph is the bar. Implement only if you already have cloud credentials and want the stretch.

Why this day exists

Nix people overreach into provisioning; TF people template snowflake user-data forever. The 2026 happy path for many teams:

Terraform/OpenTofu: VPC, VM, disks, DNS, IAM, object storage
        │
        │  outputs: IP, IDs, SSH target
        ▼
Nix flake: Disko/image/deploy-rs/Colmena configure the OS

Without a boundary doc, every new hire reinvents the split—and usually badly.

Theory 1 — Responsibility split

Concern	Owner	Why
Cloud account / IAM	TF	Lifecycle & policy engines
VM size, subnet, SG	TF	Provider API native
Disk attach / snapshots API	TF	Cloud object model
Packages, services, users	Nix	Pure rebuild story
SSH hardening	Nix	Day 36
App releases	Nix / CI	Closures
DNS record for service	Often TF	Or Nix + provider with care
Golden AMI/GCE image build	Nix + CI	Day 62
Instance launch from image	TF	Provider resource

Smell: 500-line user_data shell scripts installing docker by curl.

Smell: Nix trying to create AWS VPCs with ad-hoc runCommand and keys in the store.

Theory 2 — Handoff patterns

A — Image + TF instance

CI builds image (Day 62) → upload AMI/GCE image
TF launches VM from that image
Optional: deploy-rs for day-2 changes

B — Generic distro image + first boot Nix

TF launches stock Ubuntu/NixOS cloud image
cloud-init runs nixos-rebuild from flake or nixos-anywhere / similar
Day-2: Colmena

C — nixos-anywhere / terraform modules community

Community modules exist that glue TF + NixOS installs. Treat as optional accelerators; still keep the boundary mental model.

Pattern	Best when
A	You control golden images
B	Fast bring-up; tolerate first-boot complexity
C	You accept community module churn

Theory 3 — Data that must cross the boundary

# terraform outputs
output "edge_ip" {
  value = aws_instance.edge.private_ip
}

output "edge_id" {
  value = aws_instance.edge.id
}

# Not required to auto-import—often human or generated inventory:
# hosts/edge/deployment.hostname = "10.0.1.20";

Approach	Pros	Cons
Manual inventory update	Simple	Drift
TF writes `inventory.json`	Automatable	Pipe discipline
cloud-init pulls flake ref	Flexible	Needs network/auth
SSM/agent	Ops heavy	Not Nix-native

Pick one handoff and write it in BOUNDARY.md. Two competing handoffs become three truths.

Theory 4 — Secrets across the boundary

Secret	Where
Cloud provider keys	TF env / CI OIDC; never Nix store
SSH deploy key	Agent / CI; public key in Nix
sops age keys	Host persist; not in TF state if avoidable
TF state	Encrypted backend; restricted IAM
Bootstrap passwords	Avoid; prefer keys

TF state can contain sensitive outputs—treat state as secret-bearing. Never commit terraform.tfstate.

Theory 5 — OpenTofu note

OpenTofu is a common FOSS Terraform fork in 2026 fleets. Examples below use generic HCL; prefer whatever your org standardizes. Concepts transfer 1:1 for this day’s boundary. Commands: tofu vs terraform—document which you used.

Theory 6 — What not to put in user_data

Temptation	Prefer
`apt install nginx`	Nix module
Curl-pipe installers	Package or image bake
Writing private keys into cloud-init	sops after boot / metadata carefully
Encoding entire NixOS config as bash	Flake deploy

Short cloud-init that only installs Nix or points nixos-anywhere is a gray zone—keep it tiny and versioned if you must.

Worked example — sketch only (AWS-shaped)

# infra/edge.tf (sketch)
resource "aws_instance" "edge" {
  ami           = var.nixos_ami # produced by image pipeline
  instance_type = "t3.small"
  subnet_id     = aws_subnet.private.id
  vpc_security_group_ids = [aws_security_group.edge.id]
  key_name      = aws_key_pair.lab.key_name

  tags = {
    Role         = "edge"
    ManagedBy    = "terraform"
    ConfiguredBy = "nix"
    NixOS        = "26.05"
  }
}

resource "aws_security_group" "edge" {
  name   = "edge-lab"
  vpc_id = aws_vpc.lab.id
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [var.admin_cidr]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# BOUNDARY.md
## Provision (TF)
- VPC, subnet, SG, instance, EIP optional
- Outputs: private_ip, instance_id

## Configure (Nix)
- roles.web, profiles.hardened
- deploy with colmena --on edge once IP known

## Forbidden
- apt install in user_data
- embedding age private keys in TF
- nix store cloud credentials

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day63

Step 1 — Draw the diagram

ASCII or markdown: resources vs modules vs deploy.

Step 2 — Write `BOUNDARY.md`

Include forbidden list and secret placement. Explicit 26.05 pin policy for OS images.

Step 3 — Inventory handoff design

Specify file format, e.g.:

{
  "edge": { "host": "10.0.1.20", "tags": ["web", "edge"] }
}

Who writes it? Who reads it (Colmena targetHost / deploy-rs hostname)?

Step 4 — Optional implement

If you have a cloud account: one VM with TF/OpenTofu + one Colmena/deploy-rs deploy. If not, stop at sketch—gate accepts sketch.

Step 5 — Failure modes table

Failure	Owner	Detection
Wrong AMI	TF/CI image	Boot fails
Wrong SG	TF	SSH timeout
Bad Nix module	Nix	activate fails
State unlock	TF	team process
Drifted inventory IP	Handoff	deploy to wrong host

Step 6 — Compare pure NixOS on metal

When TF is unnecessary (homelab bare metal), document “TF optional.” Boundary still useful for future cloud.

Step 7 — Stretch: `terraform output -json` → jq → host file

Script a generator; do not commit secrets. Dry-run only is fine.

Step 8 — Teach-back

Explain the split in 90 seconds without slides. If you mention user_data bash stacks positively, rewrite BOUNDARY.md.

Step 9 — Lifecycle matrix

Fill once for your sketch host:

Lifecycle event	TF	Nix	CI
Create VM	yes	no	optional image build first
Resize disk	yes	maybe growpart in OS	—
Change SSH policy	no	yes	—
Rotate deploy key (public)	maybe	yes	yes
Destroy VM	yes	secrets offline first	—

Step 10 — Homelab exception card

If Capstone is bare metal only, still keep BOUNDARY.md with:

## Current: no cloud
- Provision: manual / PXE / ISO (Day 62 ISO optional)
- Configure: flake + deploy-rs/Colmena
- When cloud arrives: reuse this boundary; do not invent user_data

Worked anti-patterns (recognize and refuse)

Anti-pattern A: TF file provisioners copying configs into /etc
  → day-2 drift; use Nix modules

Anti-pattern B: Nix runCommand calling aws cli with keys in env during build
  → impure builds; credentials in logs; use TF/OIDC outside the store

Anti-pattern C: Two DNS writers (Route53 TF + Nix DNS module) without owner
  → pick one writer per zone/record class

Common gotchas

Symptom / mistake	What to do
user_data megascript	Delete; bake image or nixos-anywhere
Nix store has cloud keys	Move to env/OIDC
Two sources of hostname truth	Pick TF or Nix DNS story
Applying TF as “config management”	Resist; day-2 is Nix
State file in git	Use remote backend
Sketch so vague nobody could implement	Add one concrete resource graph

Checkpoint

Boundary diagram written
BOUNDARY.md with forbidden list
Handoff inventory designed
Optional real VM or explicit sketch-only completion
Secrets placement clear

Commit

git add .
git commit -m "day63: terraform/nix boundary sketch"

Write three personal gotchas before continuing.

Tomorrow

Day 64 — nixosTest: VM tests for your modules.

Day 64 — nixosTest

Stage VI · ~4h (lab-heavy)
Goal: Write and run a nixosTest VM test that exercises your module (proxy, hardened SSH, or role from Day 61)—not only a copy-paste hello world.

Why this day exists

nixos-rebuild on a pet proves little about the next pet. nixosTest boots one or more QEMU VMs from your modules, runs shell Python/Perl test scripts, and fails CI when regressions land. It is the highest leverage test type in NixOS land for system behavior.

Theory 1 — What a nixosTest is

NixOS configs (nodes) → build VM images → boot under QEMU → testScript → pass/fail

Piece	Role
`nodes`	Attrset of NixOS module stacks
`testScript`	Python (modern) driving VMs
`name`	Test identity
Driver	`nixos/lib/testing.nix` machinery

Requires KVM (or slow emulation) and often builder feature kvm / nixos-test.

Theory 2 — Minimal single-node test

# tests/ssh-hardening.nix
{
  name = "ssh-hardening";
  nodes.machine = { pkgs, ... }: {
    imports = [ ../modules/profiles/hardened.nix ];
    users.users.alice = {
      isNormalUser = true;
      openssh.authorizedKeys.keys = [
        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKeyForTestOnly not-a-real-key"
      ];
    };
    services.openssh.enable = true;
  };

  testScript = ''
    machine.wait_for_unit("sshd.service")
    machine.succeed("grep -q 'PasswordAuthentication no' /etc/ssh/sshd_config")
    # or sshd -T output depending on version
    machine.succeed("sshd -T | grep -i 'passwordauthentication no'")
  '';
}

Wire into flake:

checks.x86_64-linux.ssh-hardening =
  nixpkgs.lib.nixos.runTest {
    imports = [ ./tests/ssh-hardening.nix ];
    hostPkgs = import nixpkgs { system = "x86_64-linux"; };
    # API: prefer current nixpkgs `runTest` / `nixosTest` helper for 26.05
  };

Note

nixpkgs has migrated helpers (pkgs.testers.runNixOSTest, lib.nixos.runTest, classic nixosTest). Use whatever your 26.05 pin documents; the shape (nodes + testScript) remains the skill.

Classic style still seen in blogs:

import nixpkgs {
  system = "x86_64-linux";
  config = { allowUnfree = false; };
} .lib.nixosSystem { } # not this

# classic:
pkgs.nixosTest {
  name = "example";
  nodes.machine = { ... };
  testScript = ''...'';
}

Theory 3 — testScript essentials

machine.wait_for_unit("multi-user.target")
machine.wait_for_open_port(22)
machine.succeed("true")
machine.fail("false")
machine.copy_from_host("file", "/tmp/file")
print(machine.succeed("uname -a"))

Multi-node:

nodes = {
  server = { ... };
  client = { ... };
};

server.wait_for_unit("caddy.service")
client.succeed("curl -f http://server/")

Network is usually a test VLAN; hostnames often equal node names.

Theory 4 — What to test (high value)

Good	Weak
Module assertions about sshd config	`true` only
Service reaches active + port open	Only that evaluation succeeds
Client→server HTTP 200	Full browser suites
Firewall rejects unexpected port	Entire Kubernetes

Test your contracts from Days 36–38 / 61.

Theory 5 — Running tests

nix build .#checks.x86_64-linux.ssh-hardening -L
# or
nix flake check -L

Interactive debug (when supported by driver):

nix build .#checks… -L
# many setups: ./result/bin/nixos-test-driver with breakpoints in testScript

machine.shell_interact()  # if available in your nixpkgs test driver

Worked example — web role smoke test

# tests/web-role.nix
{
  name = "web-role";
  nodes.server = { ... }: {
    imports = [
      ../modules/profiles/baseline.nix
      ../modules/roles/web.nix
    ];
    roles.web = {
      enable = true;
      domain = "server"; # or localhost style per caddy config
    };
  };
  nodes.client = { pkgs, ... }: {
    environment.systemPackages = [ pkgs.curl ];
  };
  testScript = ''
    server.wait_for_unit("caddy.service")
    server.wait_for_open_port(80)
    client.succeed("curl -f --retry 5 --retry-delay 1 http://server/ | grep -q ok")
  '';
}

Adapt to your actual reverse proxy module from Stage IV.

Lab — multi-step

Suggested workspace: fleet flake tests/; notes in ~/lab/90daysofx/02-nixos/day64

Step 1 — Confirm virtualization

ls /dev/kvm && echo kvm-ok || echo "need kvm or nested virt"

Step 2 — Hello test

Minimal machine with environment.etc."t".text = "ok"; and succeed("test -f /etc/t").

Step 3 — Port your module

SSH hardening or web role or custom Stage IV module.

Step 4 — Make it fail once

Break the module; observe red test; fix.

Step 5 — Wire `checks`

nix flake check runs it (may be heavy—ok).

Step 6 — Document runtime

Wall clock; note CI machine requirements (Day 66).

Step 7 — Multi-node stretch

Client/server only if single-node is green.

Theory 6 — Tests as flake checks

checks.${system}.lab-ssh = self.nixosConfigurations.lab.config.system.build.toplevel; # heavy; optional
# better: dedicated nixosTest derivation
checks.${system}.web-smoke = pkgs.nixosTest { … };

Wiring tests into checks makes nix flake check the Stage VI/CI gate. Keep at least one fast smoke and quarantine long tests behind explicit attrnames.

Theory 7 — Interactive vs pure test runs

nix build .#checks.x86_64-linux.web-smoke -L
# driver interactively (when supported by your setup):
# nix run .#checks.…driverInteractive

Interactive drivers help author tests; CI should run non-interactive builds.

Worked example — wait for unit + port

machine.wait_for_unit("caddy.service")
machine.wait_for_open_port(443)
machine.succeed("curl -k -sS https://127.0.0.1 | grep -q lab")

Prefer wait_for_unit / wait_for_open_port over sleep 30.

Lab — one negative assertion

machine.fail("curl -sS --max-time 2 http://127.0.0.1:5432 || true")
# better: assert postgres not on 0.0.0.0 if that is policy

Encode a security/policy property, not only happy path.

Lab — time the test

time nix build .#checks.x86_64-linux.web-smoke -L

Record duration. Tests that take 20+ minutes will not be run and thus do not protect you—split or slim.

Common gotchas

Symptom / mistake	What to do
No `/dev/kvm`	Nested virt; remote builder with kvm; or accept qemu slow
Hang on boot	Serial logs; simplify modules; wait_for_unit
DNS names	Use node names as hostnames
Testing wrong module version	Same flake inputs as deploy
Over-large configs	Slim test nodes; don’t import Desktop
Flaky timing	`wait_for_open_port`, retries

Checkpoint

Commit

git add .
git commit -m "day64: nixosTest for own module"

Write three personal gotchas before continuing.

Tomorrow

Day 65 — More tests: expand the suite to three meaningful checks.

Day 65 — More tests

Stage VI · ~4h
Goal: Grow a suite of at least three meaningful flake checks (including nixosTest and lighter eval/build checks) that mirror CI shape without becoming a flaky tar pit.

Note

Builds on Day 64 (nixosTest). Baseline: flakes, 26.05 pins, modules you already own. Diversify kinds of checks; do not only clone one VM test three times.

Why this day exists

One demo test rots. A small suite with clear ownership becomes the merge gate for roles and packages. Today you diversify test layers, kill flakes early, and time the suite so Day 66 CI has a budget.

Theory 1 — Test pyramid for NixOS fleets

Layer	Examples	Speed	Catches
Eval	`nix flake show`, option asserts, `nix eval`	Fast	Typos, missing options
Build	`nix build .#pkg`, host toplevel	Medium	Compile/package breaks
VM (nixosTest)	Service behavior	Slow	Integration
Deploy dry	deploy-rs checks, colmena eval	Medium	Hive wiring
Live canary	Real host probes	Manual/slow	Reality

Aim: many fast, few slow. Capstone and CI die when every PR boots three full VMs for a typo in a comment.

        /live canary\
       /  nixosTest   \
      / build / toplevel \
     / eval / lint / fmt  \

Theory 2 — Three checks minimum (template)

# flake outputs fragment — illustrative
checks = forAllSystems (system:
  let
    pkgs = import nixpkgs {
      inherit system;
      overlays = [ self.overlays.default ];
    };
  in {
    # 1) package as check (build is the test)
    cooltool-build = self.packages.${system}.cooltool;

    # 2) pure command check
    cooltool-runs = pkgs.runCommand "cooltool-runs" {
      nativeBuildInputs = [ self.packages.${system}.cooltool ];
    } ''
      cooltool --help >/dev/null
      touch $out
    '';
  } // nixpkgs.lib.optionalAttrs (system == "x86_64-linux") {
    # 3+) VM tests — Linux only
    ssh-hardening = import ./tests/ssh-hardening.nix { inherit pkgs self; };
    web-role = import ./tests/web-role.nix { inherit pkgs self; };
  }
);

nix build .#checks.x86_64-linux.cooltool-runs -L
nix build .#checks.x86_64-linux.ssh-hardening -L
nix flake check -L

Theory 3 — Lightweight non-VM checks

Format / lint

# Prefer your Stage III treefmt/nixfmt wiring if present
fmt = pkgs.runCommand "fmt-check" {
  nativeBuildInputs = [ pkgs.nixfmt-rfc-style ];
} ''
  # example only — real checks usually run treefmt --ci on src
  touch $out
'';

Eval / build host configuration

edge-toplevel =
  self.nixosConfigurations.edge.config.system.build.toplevel;

nix build .#nixosConfigurations.edge.config.system.build.toplevel -L

Strong, but heavy—cache it; don’t duplicate five slight variants.

Structured eval assertions

nix eval .#nixosConfigurations.edge.config.networking.hostName
nix eval --json .#nixosConfigurations.edge.config.services.openssh.enable

Wrap in a runCommand with jq if you want it as a formal check:

host-name-ok = pkgs.runCommand "host-name-ok" {
  nativeBuildInputs = [ pkgs.jq ];
} ''
  echo ${pkgs.lib.escapeShellArg (
    builtins.toJSON self.nixosConfigurations.edge.config.networking.hostName
  )} | jq -e '. == "edge"'
  touch $out
'';

(Prefer cleaner patterns as your flake matures; the point is fast feedback.)

Theory 4 — Anti-flake tactics

Problem	Fix
Sleep-based waits	`wait_for_unit` / `wait_for_open_port`
Order-dependent multi-test	Isolated nodes per test file
Network to internet in tests	Forbid; vendor fixtures
Shared mutable state	Each test pure
Over-broad `flake check` on Darwin	`optionalAttrs` for Linux VM tests
30-minute suite	Split “CI fast” vs “nightly full”
Secret decryption in CI	Dummy modules / test keys only

# nixosTest script — good waits
machine.wait_for_unit("nginx.service")
machine.wait_for_open_port(80)
machine.succeed("curl -f http://127.0.0.1/")

# Run one check without the world
nix build .#checks.x86_64-linux.ssh-hardening -L

Theory 5 — Mapping tests to modules

Module / concern	Check ideas
`profiles.hardened`	`sshd -T` probes, package absences
`roles.web`	curl client→server via proxy
Stage V package	help/version smoke
Disko	eval `fileSystems` keys (not full wipe)
sops-nix	assert secret options configured; decrypt with test key only
firewall	multi-node: client cannot hit closed port
deploy hive	eval nodes exist / `--dry-activate` if available

Every check should name an owner module in a README table so rotting tests get a human.

Theory 6 — Suite layout and CI shape

tests/
  ssh-hardening.nix
  web-role.nix
  cooltool.nix
  fixtures/
    test-secrets.nix
  README.md          # table: check → layer → minutes → owner

# flake fragment
checks.x86_64-linux = {
  inherit (self.packages.x86_64-linux) cooltool;
  cooltool-runs = /* … */;
  ssh-hardening = /* … */;
  web-role = /* … */;
  edge-toplevel =
    self.nixosConfigurations.edge.config.system.build.toplevel;
};

Day 66 will run these on GitHub Actions. Design attrs you can select:

nix build -L \
  .#checks.x86_64-linux.cooltool-runs \
  .#checks.x86_64-linux.ssh-hardening

Worked example — fail-inject once

# 1) break intentionally
# e.g. change expected hostname assertion or stop nginx in test config
nix build .#checks.x86_64-linux.web-role -L   # expect fail
# 2) restore
# 3) confirm green
# 4) confirm other checks still independent
nix build .#checks.x86_64-linux.cooltool-runs -L

If breaking test A always breaks unrelated B, your suite shares too much impure state.

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day65

Step 1 — Inventory today’s checks

mkdir -p ~/lab/90daysofx/02-nixos/day65
cd /path/to/your/flake
nix flake show 2>&1 | tee ~/lab/90daysofx/02-nixos/day65/flake-show.txt

List current checks and gaps.

Step 2 — Add check #1 (package)

Build + run your Stage V package (runCommand or package attr as check).

Step 3 — Add check #2 (nixosTest)

Keep/improve Day 64 test; ensure waits are unit/port based.

Step 4 — Add check #3 (config)

Either second nixosTest or toplevel build for one host or eval assertion.

Step 5 — Fail inject matrix

Break each check once; confirm isolation; restore green.

Step 6 — Timing budget

/usr/bin/time -p nix flake check -L \
  2>&1 | tee ~/lab/90daysofx/02-nixos/day65/timing.txt

If over your CI budget, document split: checks.ci vs full suite.

Step 7 — README table

Check	Layer	Minutes	Owner module

Step 8 — Stability

Rerun thrice (or twice if time-boxed); note flakiness tickets.

Step 9 — Secrets policy for tests

Write one paragraph: how CI avoids prod sops keys (fixtures, dummy tokens).

Common gotchas

Symptom / mistake	What to do
`flake check` builds everything forever	Select attrs; fix IFD explosions
Darwin CI without KVM	Gate VM tests on Linux
Secret decryption in checks	Dummy modules / mock
Duplicate heavy toplevels	Share less; accept one + cache
Tests not in merge gate	Day 66 wires CI
Sleep-only “green”	Replace with readiness waits
Three identical VM tests	Diversify layers

Checkpoint

≥3 meaningful checks green
At least one VM test and one non-VM test
Timing measured
README/table of suite with owners
Fail-inject done once per check
Flakiness notes empty or ticketed
Three personal gotchas

Commit

git add .
git commit -m "day65: expanded flake check suite"

Write three personal gotchas before continuing.

Tomorrow

Day 66 — CI pipeline: GitHub Actions + Nix + cache write.

Day 66 — CI pipeline

Stage VI · ~4h
Goal: Run your flake checks/builds on GitHub Actions (or GitLab) with a modern Nix installer, optional Cachix/Attic push, and a PR-shaped workflow that fails on regressions.

Note

If tests only run on your laptop, they are hobbies. 2026 default shape: forge CI + flakes + binary cache. Baseline nixpkgs: 26.05.

Why this day exists

Stage V packages and Stage VI modules become team infrastructure only when a clean runner reproduces them. CI also forces purity: “works on my machine” impurities die in public logs. Capstone Day 85 reuses this muscle.

Theory 1 — Minimal trustworthy pipeline

checkout → install Nix → (auth cache) → nix flake check / build → (push cache)

Job / step	Purpose
`checkout`	Source + lockfile
`install Nix`	Known installer; flakes enabled
`cache auth`	Pull/push substitutes
`check`	`nix flake check` or selected checks
`build-pkg`	Stage V package
`build-hosts`	Optional toplevel builds

Trust boundaries

Surface	Rule
PR from fork	Often no secret cache write tokens
`main`	May push cache
Logs	Never print age keys, Cachix tokens, SSH keys
`trusted-users`	Do not weaken developer laptops for CI

Theory 2 — GitHub Actions workflow (2026-shaped)

# .github/workflows/nix.yml
name: nix

on:
  push:
    branches: [ main ]
  pull_request:
  workflow_dispatch:

permissions:
  contents: read
  # id-token: write  # if using OIDC for caches later

concurrency:
  group: nix-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Nix
        uses: DeterminateSystems/nix-installer-action@main
        # pin to a reviewed tag in real repos

      - name: Magic Nix Cache (optional)
        uses: DeterminateSystems/magic-nix-cache-action@main
        # or Cachix below

      # - uses: cachix/cachix-action@v15
      #   with:
      #     name: myorg
      #     authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}

      - name: Flake metadata
        run: nix flake metadata

      - name: Selected checks
        run: |
          nix build -L \
            .#checks.x86_64-linux.cooltool-runs \
            .#checks.x86_64-linux.ssh-hardening

      - name: Build package
        run: nix build .#cooltool -L

      - name: Build edge toplevel (optional, heavy)
        run: nix build .#nixosConfigurations.edge.config.system.build.toplevel -L

Warning

Pin action versions to tags or SHAs you have reviewed. Versions above are illustrative—confirm current upstream tags when you wire the repo. Freeze pins harder by Day 87 for Capstone.

Theory 3 — Cachix (and friends) in CI

- uses: cachix/install-nix-action@v31
  with:
    nix_path: nixpkgs=channel:nixos-26.05
- uses: cachix/cachix-action@v15
  with:
    name: myorg
    authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
    # signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'  # if used
- run: nix build .#cooltool -L

House rules

Rule	Why
Push from CI (common)	Shared binaries; less laptop upload chaos
Separate PR vs main push rights	Abuse / token theft mitigation
Public keys on consumers	Lab + admin `substituters` + `trusted-public-keys`
Never commit private signing keys	Rotate immediately if leaked

Alternatives

Tool	Notes
Cachix	Hosted; simple
Attic	Self-hosted multi-tenant cache
Harmonia	Serve local store as binary cache
Magic Nix Cache	UX-oriented GitHub cache integration

Any one done well beats none.

Theory 4 — What not to run on free runners

Heavy	Strategy
Full nixosTest matrix	Linux job; fewer nodes; path filters
Multiple toplevels	Build one primary host per PR
Nested KVM	May work on GH-hosted; else self-hosted runner
Secret full decrypt	Dummy configs for CI (Day 65)
Unbounded `flake check`	Select attrs until budget known

on:
  pull_request:
    paths:
      - 'flake.nix'
      - 'flake.lock'
      - 'modules/**'
      - 'hosts/**'
      - 'pkgs/**'
      - 'tests/**'
      - '.github/workflows/nix.yml'

Timeouts and disks

jobs:
  check:
    runs-on: ubuntu-latest
    timeout-minutes: 90
    steps:
      # …

Out-of-disk: slim checks, GC mid-job only if needed, or larger runners.

Theory 5 — GitLab CI sketch

# .gitlab-ci.yml (sketch)
image: debian:bookworm
variables:
  NIX_CONFIG: "experimental-features = nix-command flakes"
before_script:
  - apt-get update && apt-get install -y curl xz-utils
  # install Nix via official/Determinate method — prefer current docs
  - nix flake metadata
check:
  script:
    - nix flake check -L
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Prefer maintained install methods over ancient nixos/nix container tags when flakes misbehave.

Theory 6 — CI vs deploy

Pattern	Day 66 stance
CI verifies build/check	Default
CI pushes cache	Preferred
CI deploys to lab	Optional later; protect with environments
CI holds prod SSH keys	High risk; avoid early

Human or gated CD deploy after green checks is enough for Stage VI. Capstone Day 85 revisits.

Worked example — selected checks job

- name: Fast checks
  run: |
    set -euo pipefail
    nix build -L \
      .#checks.x86_64-linux.cooltool-runs \
      .#checks.x86_64-linux.ssh-hardening
- name: Optional full check
  if: github.ref == 'refs/heads/main'
  run: nix flake check -L

Split fast PR signal from heavier main-branch verification when needed.

Worked example — intentional red PR

# on a branch: break a check assertion or package
git checkout -b ci/red-test
# … introduce fail …
git commit -am "ci: temporary red"
git push -u origin ci/red-test
# open PR → confirm CI red
# fix → confirm green

If you cannot get a red PR, the workflow is not protecting you.

Lab — multi-step

Suggested notes: ~/lab/90daysofx/02-nixos/day66

Step 1 — Add workflow file

Commit .github/workflows/nix.yml (or GitLab equivalent) to your lab flake repo. Use path filters if monorepo.

Step 2 — First green CI

Push branch; fix until green on a clean runner. Paste run URL into notes.

mkdir -p ~/lab/90daysofx/02-nixos/day66
echo "CI_URL=…" > ~/lab/90daysofx/02-nixos/day66/ci-url.txt

Step 3 — Wire cache

Cachix or Magic Nix Cache or Attic. Confirm second run is faster or shows substitutes.

Step 4 — PR simulation

Open PR that breaks a check; confirm CI red; fix; green.

Step 5 — Secrets hygiene

Ensure tokens only in forge secrets; scan workflow for accidental echo of secrets.

Step 6 — Consumer config

Add substituter public keys to lab/admin Nix config; rebuild or set user nix.conf.

Step 7 — Document runner needs

KVM? memory? timeout? Write docs/CI.md with:

Topic	Content
Triggers	PR/main
Commands	exact `nix build` lines
Cache	name + public key
Limits	known failures without KVM
Non-goals	deploy-from-CI policy

Step 8 — Status badge (optional)

README badge for the workflow—only if green is honest.

Step 9 — Deploy from CI? (decide)

Write the choice: verify-only vs gated deploy. Default for this day: verify-only.

Common gotchas

Symptom / mistake	What to do
Flakes disabled	Installer action enables; set `NIX_CONFIG`
Disk full on runner	Slim checks; larger runner; GC carefully
Unpinned actions	Pin majors/SHAs per org policy
Cache not used	Public key / auth; path existence; wrong cache name
VM tests fail without KVM	Self-hosted or reduce tests
Monorepo wrong cwd	`defaults.run.working-directory` / filters
Fork PRs leaking write tokens	Restrict secret access
IFD / eval forever	Fix expressions; select attrs

Checkpoint

Workflow exists and runs on PR/push
At least one Nix build/check green in CI
Intentional red PR observed
Cache strategy configured or explicitly deferred with reason
docs/CI.md notes runner limits and commands
Tokens not in git
Three personal gotchas

Commit

git add .
git commit -m "day66: GitHub Actions Nix CI pipeline"

Write three personal gotchas before continuing.

Tomorrow

Day 67 — GC & store hygiene: roots, retention, optimise.

Day 22 — Terraform, nixosTest & CI Pipeline

Day 63 — Terraform boundary

Why this day exists

Theory 1 — Responsibility split

Theory 2 — Handoff patterns

A — Image + TF instance

B — Generic distro image + first boot Nix

C — nixos-anywhere / terraform modules community

Theory 3 — Data that must cross the boundary

Theory 4 — Secrets across the boundary

Theory 5 — OpenTofu note

Theory 6 — What not to put in user_data

Worked example — sketch only (AWS-shaped)

Lab — multi-step

Step 1 — Draw the diagram

Step 2 — Write BOUNDARY.md

Step 3 — Inventory handoff design

Step 4 — Optional implement

Step 5 — Failure modes table

Step 6 — Compare pure NixOS on metal

Step 7 — Stretch: terraform output -json → jq → host file

Step 8 — Teach-back

Step 9 — Lifecycle matrix

Step 10 — Homelab exception card

Worked anti-patterns (recognize and refuse)

Common gotchas

Checkpoint

Commit

Tomorrow

Day 64 — nixosTest

Why this day exists

Theory 1 — What a nixosTest is

Theory 2 — Minimal single-node test

Theory 3 — testScript essentials

Theory 4 — What to test (high value)

Theory 5 — Running tests

Worked example — web role smoke test

Lab — multi-step

Step 1 — Confirm virtualization

Step 2 — Hello test

Step 3 — Port your module

Step 4 — Make it fail once

Step 5 — Wire checks

Step 6 — Document runtime

Step 7 — Multi-node stretch

Theory 6 — Tests as flake checks

Theory 7 — Interactive vs pure test runs

Worked example — wait for unit + port

Lab — one negative assertion

Lab — time the test

Common gotchas

Checkpoint

Commit

Tomorrow

Day 65 — More tests

Why this day exists

Theory 1 — Test pyramid for NixOS fleets

Theory 2 — Three checks minimum (template)

Theory 3 — Lightweight non-VM checks

Format / lint

Eval / build host configuration

Structured eval assertions

Theory 4 — Anti-flake tactics

Theory 5 — Mapping tests to modules

Theory 6 — Suite layout and CI shape

Worked example — fail-inject once

Lab — multi-step

Step 1 — Inventory today’s checks

Step 2 — Add check #1 (package)

Step 3 — Add check #2 (nixosTest)

Step 4 — Add check #3 (config)

Step 5 — Fail inject matrix

Step 6 — Timing budget

Step 7 — README table

Step 8 — Stability

Step 9 — Secrets policy for tests

Common gotchas

Checkpoint

Commit

Tomorrow

Step 2 — Write `BOUNDARY.md`

Step 7 — Stretch: `terraform output -json` → jq → host file

Step 5 — Wire `checks`