Day 14 — PostgreSQL, nixos-containers & Podman
Day 14 — PostgreSQL, nixos-containers & Podman
Day 39 — Data service pattern
Stage IV · ~4h
Goal: Run one data service—PostgreSQL by default—declaratively on NixOS: module options, state directories, app role/auth via secrets paths, and a written backup story (even if automation is minimal).
DB passwords go through sops-nix paths, not Nix strings. Examples use placeholders and path references only.
Why this day exists
Application servers without state are easy. Real systems need durable data:
| Concern | NixOS angle |
|---|---|
| Package + unit | services.postgresql module |
| State location | services.postgresql.dataDir / ensure on persistent disk |
| Auth | peer/scram; password files from secrets |
| Network exposure | localhost first (Day 37) |
| Backups | Not automatic just because Nix is declarative |
Nix rebuilds the software and config; it does not magically version your rows.
Theory 1 — One DB only (pattern depth)
Syllabus: PostgreSQL or another single DB. Depth > zoo.
We use PostgreSQL. If you already run another engine at work, the pattern transfers: module → state dir → credentials → firewall → backups.
Theory 2 — Minimal PostgreSQL module
# modules/services/postgres.nix
{ config, pkgs, ... }:
{
services.postgresql = {
enable = true;
package = pkgs.postgresql_16; # pick a version available on 26.05 pin
ensureDatabases = [ "labapp" ];
ensureUsers = [
{
name = "labapp";
ensureDBOwnership = true;
}
];
# authentication / settings via extra plugins or ensure* options
settings = {
# listen_addresses = "localhost"; # confirm option name on your version
};
};
# Do NOT open 5432 on the host firewall for world access in the default lab.
}Verify option names against search.nixos.org for 26.05.
Theory 3 — State directories and disk
PostgreSQL data lives outside the Nix store (mutable).
| Question | Answer you must know |
|---|---|
Where is dataDir? |
Document path |
| Is it on the disk that survives reinstall? | Stage VI Disko/impermanence later |
| What happens on rollback of system generation? | Software rolls back; data may not |
Impermanence hosts (later) need explicit persistence for DB dirs—note the footgun now.
Theory 4 — Credentials
Patterns:
- Peer auth for local system users (good for admin)
- SCRAM password for app users—password from sops at runtime
- Socket-only access for local apps
Sketch with sops (illustrative—align with module capabilities):
sops.secrets."postgres/labapp_password" = {
owner = "postgres";
mode = "0400";
};
# Prefer module-supported passwordFile style options when available.
# If you script role passwords, run a oneshot that reads the secret path
# and does not print it.Never:
# BAD
services.postgresql.ensureUsers = [{ name = "labapp"; password = "secret"; }];(If an option even allows raw password—avoid it.)
Theory 5 — App connection story
App config should receive:
- Host:
127.0.0.1or socket
- DB name:
labapp
- User:
labapp
- Password: from env file populated at activation from sops
Your Day 38 proxy does not replace DB auth. Backend talks to DB on localhost; users talk to proxy on 443.
Theory 6 — Backup thought (required cognitive work)
Even a one-node lab needs a plan:
| Element | Example |
|---|---|
| What | pg_dump of labapp |
| When | nightly timer (system or user) |
| Where | /var/backups/postgres/ on another disk or remote |
| Encryption | restic/age to remote (secrets via sops) |
| Restore test | once per quarter / before capstone |
Today: write the plan + optional manual pg_dump once. Full backup automation can be minimal.
sudo -u postgres pg_dump labapp > /tmp/labapp-dump.sql
# store off-box in real life; scrub /tmp laterTheory 7 — Upgrades
Postgres major upgrades are not “change package attr and forget.” They need dump/restore or pg_upgrade procedures. Pin a major version intentionally; schedule upgrades.
Worked example — Service + docs
modules/services/postgres.nix
docs/data-service.md
- version
- dataDir
- DB/user names
- auth method
- backup sketch
- restore sketch
Lab 1 — Enable PostgreSQL
Add module; rebuild; verify unit:
sudo systemctl status postgresql
sudo -u postgres psql -c '\l'Lab 2 — Database and role
Confirm labapp DB and user via ensure* options. Connect:
sudo -u postgres psql -d labapp -c 'SELECT 1;'Lab 3 — Wire a secret password path
Create sops secret placeholder for app password. Configure the safest supported method for your module version. Prove app user can authenticate without password living in the flake as plaintext.
Lab 4 — Network posture
ss -tulpn | grep 5432 || trueConfirm listening on localhost (or document intentional LAN bind). Ensure firewall does not expose 5432 publicly.
Update docs/ports.md.
Lab 5 — Manual dump
sudo -u postgres pg_dump labapp | head
# full dump to a safe path you control; do not commit dumps with sensitive dataWrite restore commands in docs/data-service.md (even if restore is “psql < dump”).
Lab 6 — Failure narrative
Answer:
- System generation rollback after a bad app deploy—does data revert?
- Disk failure—what is lost without backups?
- Secret rotation steps for DB password
Theory 8 — ensureDatabases vs migrations
ensureDatabases / ensureUsers create empty roles/DBs. Application schema is not Nix’s job by default.
| Layer | Owner |
|---|---|
| Package + unit + listen policy | NixOS module |
| Roles/DBs existence | ensure* options / oneshots |
| Schema / migrations | App tooling (flyway, django, …) |
| Data | Backups |
Do not expect rebuild to replay SQL migrations unless you deliberately automate that (usually outside pure module defaults).
Theory 9 — Socket vs TCP local connections
Postgres often exposes a Unix socket under /run/postgresql. Local admin via sudo -u postgres psql uses peer auth. Apps may use:
host=127.0.0.1 port=5432
# or host=/run/postgresql
Pick one, document in docs/data-service.md, match app secret env.
Worked example — Timer sketch for dumps (optional)
systemd.services.labapp-pg-dump = {
serviceConfig = {
Type = "oneshot";
User = "postgres";
ExecStart = "${pkgs.bash}/bin/bash -c '${pkgs.postgresql}/bin/pg_dump labapp | ${pkgs.gzip}/bin/gzip -c > /var/backups/labapp.sql.gz'";
};
};
# + timer + StateDirectory/tmpfiles for /var/backupsWire carefully (permissions, disk space). Even a commented sketch in the flake beats “we’ll back up later.”
Lab 7 — Schema independence proof
sudo -u postgres psql -d labapp -c "CREATE TABLE day39_demo(id int);"
sudo -u postgres psql -d labapp -c '\dt'
# rebuild switch without dropping data
sudo nixos-rebuild switch --flake .#lab
sudo -u postgres psql -d labapp -c '\dt'Table should still exist—state ≠ generation.
Lab 8 — Document restore dry-run
Write exact commands in docs (do not need to destroy data):
# restore sketch
# sudo -u postgres psql -d labapp < labapp-dump.sqlCommon gotchas
| Symptom / mistake | What to do |
|---|---|
| Data in store myth | Data is on mutable disk |
| Open 5432 to world | Bind localhost; close firewall |
| Password in nix | sops path / peer auth |
| Major version bump casual | Plan dump/restore |
| No backup thought | Lab 5–6 required |
| Proxying Postgres over HTTP | Wrong layer; don’t |
Checkpoint
- PostgreSQL enabled declaratively
- App DB + user exist
- Auth without plaintext secret in git
- Localhost exposure default
- Manual dump performed once
- Backup/restore story written
- ports.md updated
Commit
git add .
git commit -m "day39: declarative postgres data service pattern"Write three personal gotchas before continuing.
Tomorrow
Day 40 — nixos-containers / OCI: isolation options on NixOS—declarative nspawn-style containers and when OCI helps without “Docker first” culture.
Day 40 — nixos-containers / OCI
Stage IV · ~4h
Goal: Understand isolation options on NixOS—especially declarative containers.* (nspawn-style) and light OCI awareness—and run one declarative container (or clearly document why your lab uses a pure NixOS service instead).
This day is not “Docker first.” Prefer a native NixOS module when it exists (Days 38–39). Containers help for upstream images, multi-version isolation, or hostile workloads—not as a default personality. Pin remains NixOS 26.05.
Why this day exists
Ecosystem pressure says “everything in containers.” NixOS offers stronger default stories for many services: one store, one firewall story, one secrets story. Still, you need literacy so you can refuse containers for good reasons—or adopt them deliberately.
| Tool | Isolation style |
|---|---|
| NixOS systemd services | Same host, units, store |
containers.<name> |
systemd-nspawn guest with NixOS config |
| OCI (Podman/Docker) | Image-based; Day 41 modules |
| VMs / k3s | Heavier; Day 43 optional |
Skipping this day leaves a blind spot: every vendor README assumes Docker, and you will paste it without knowing what you sacrificed.
Theory 1 — When containers help on NixOS
| Use container | Prefer native module |
|---|---|
| Vendor only ships image | PostgreSQL, Caddy, Prometheus in nixpkgs |
| Need older/newer app than nixpkgs | Simple Go binary you package in Stage V |
| Separate failure domain lightly | One more systemd service is enough |
| Multi-tenant sketch | Single-user lab |
| Hostile untrusted binary | Still not a full security boundary—see Theory 4 |
Decision micro-script
Is there a maintained NixOS module?
yes → use it (Days 38–39)
no → Can I package it (Stage V)?
yes → package + native service
no → OCI (Day 41) or nspawn NixOS guest (today)
Theory 2 — containers.<name> (NixOS nspawn)
NixOS can define ephemeral/declarative containers. The guest is a nested NixOS module graph—not a Dockerfile.
# modules/services/demo-container.nix
{
containers.demo = {
autoStart = true;
privateNetwork = true;
hostAddress = "192.168.100.1";
localAddress = "192.168.100.2";
config = { config, pkgs, ... }: {
system.stateVersion = "26.05";
networking.firewall.allowedTCPPorts = [ 80 ];
services.nginx = {
enable = true;
virtualHosts."localhost" = {
root = pkgs.writeTextDir "index.html" "<h1>container</h1>";
};
};
# minimal users/sshd usually not needed for service-only guests
};
};
}| Option idea | Meaning |
|---|---|
autoStart |
Start at boot |
privateNetwork |
Separate network namespace |
hostAddress / localAddress |
veth pair addressing |
bindMounts |
Host paths into guest |
config |
Nested NixOS module system |
ephemeral |
Lose rootfs state on stop (when supported) |
Exact networking options evolve—verify against your 26.05 options docs (man configuration.nix / search.nixos.org) if attributes differ.
Pros / cons
| Pros | Cons |
|---|---|
| Declarative guest config | Nested complexity / longer eval |
| Shares host store often | Not full VM isolation |
| Good teaching tool | Debug requires container tooling |
| Rebuild updates guest like host | Nested stateVersion still required |
sudo machinectl list
sudo nixos-container status demo 2>/dev/null || true
sudo nixos-container root-login demo 2>/dev/null || trueTheory 3 — OCI awareness (without Day 41 depth)
OCI images: layers + config + entrypoint. On NixOS you may:
- Pull images with Podman/Docker (Day 41)
- Build images from Nix (
dockerTools, nixery, etc.—later packaging awareness)
- Run rootless vs rootful
Today: know that an image is not a NixOS module. It is a blob with its own update story, CVE surface, and entrypoint that may ignore your host hardening.
NixOS guest (containers.*) |
OCI image |
|---|---|
| Config is Nix | Config is image + run flags |
| Updates via host rebuild | Updates via pull/tag discipline |
| Same module options as host | Ad-hoc env vars / volumes |
| Fits flake locks naturally | Needs digest pins for honesty |
Theory 4 — Security boundaries (honest)
| Boundary | Strength |
|---|---|
| Process on host | Weak isolation |
systemd hardening (ProtectSystem, …) |
Better, still host kernel |
| nspawn container | Stronger namespaces |
| OCI with user namespaces | Better if configured; easy to misconfigure |
| VM | Stronger (hardware virt) |
| Separate machine | Operational isolation |
Do not claim “container = secure” without profile hardening and a patch process for images. Kernel exploits ignore your Docker Compose feelings.
Hardening checklist (guest)
- Minimal packages in guest
- Firewall on guest and publish policy on host
- No unnecessary bind mounts of
/or secret dirs
- No privileged flags unless justified in writing
Theory 5 — State and containers
Guest state can live in:
- Container root filesystem areas
- Bind mounts from host paths
Bind mounts must respect secrets and backup stories. Mounting /var/lib/postgresql into a random container without care is a good way to corrupt data.
| Pattern | Use |
|---|---|
| Ephemeral guest | Stateless demos, rebuild-friendly |
Bind /var/lib/app from host |
Persist data; backup host path |
| Named volume (OCI) | Day 41 territory |
| Store-only content | Ideal for pure static sites |
Theory 6 — Relation to proxy / firewall
If a container listens only on a private network, the host proxy may reverse-proxy into it—or you publish a port on the host. Every publish updates docs/ports.md.
[client] → host:443 (caddy/nginx)
↓
192.168.100.2:80 (container)
| Mistake | Fix |
|---|---|
| Open guest port on public interface | Proxy only; firewall deny direct |
Forget ports.md |
Stage IV gate will bite |
| Double NAT confusion | Draw the path once |
Theory 7 — Rebuild UX and nested eval
Changing guest config requires host evaluation of nested modules. Failures look like host rebuild failures with deeper traces.
# build only (no switch) to catch nested errors early
sudo nixos-rebuild build --flake .#lab -L| Friction vs native service | Why |
|---|---|
| Longer eval | Nested module system |
| Harder journal navigation | Machine names / nspawn units |
| Networking options maze | Worth a diagram |
| Worth it when? | Real isolation need, not fashion |
Worked example — Prefer native, demo container optional
Path A (valid for gate): skip long-lived containers; write a decision record “native modules suffice for Stage IV slice.”
Path B: run the containers.demo nginx guest and curl via host networking setup you chose.
Both paths require written judgment. Path A is not a skip of the day—it is an evidence-backed refusal.
Lab 1 — Decision record
In docs/isolation.md:
# Isolation choices
Stage IV slice services:
- proxy: native module
- postgres: native module
- container: yes/no because …
Boundary strength needed:
Threat model (1 paragraph):
When we would revisit containers:Lab 2 — Explore host tools
command -v nixos-container || true
command -v machinectl || true
systemctl list-units 'systemd-nspawn@*' 2>/dev/null | head || true
man systemd-nspawn 2>/dev/null | head || trueRecord which tools exist on your 26.05 lab host.
Lab 3 — One declarative container or dry config
Implement containers.demo or write the full Nix config in-repo under modules/services/demo-container.nix with enable false and a comment—only if you truly cannot run nspawn in your environment (e.g. restricted VM). Prefer actually running it when possible.
sudo nixos-rebuild switch --flake .#lab
sudo nixos-container list 2>/dev/null || sudo machinectl listLab 4 — Reach the guest service
Depending on networking mode, curl from host to guest IP or port. Document the path in docs/isolation.md.
# example privateNetwork style
curl -sS http://192.168.100.2/ | tee ~/lab/90daysofx/02-nixos/day40/curl.txtLab 5 — Compare rebuild UX
Change guest index.html content via Nix; rebuild; observe whether container picks up changes cleanly. Note friction vs native services.nginx on host (time to change, journal path, failure modes).
Lab 6 — When you would use OCI instead
Write three bullets for Day 41: cases where an upstream image beats nspawn NixOS guest.
Lab 7 — Firewall / ports doc
Update docs/ports.md (or create it) for any published path. If Path A (no container), document that no extra ports were opened.
Common gotchas
| Symptom / mistake | What to do |
|---|---|
Nested stateVersion missing |
Set in guest config ("26.05") |
| Networking black hole | Start with simpler network mode; read options |
| Assuming Docker API | That’s Day 41 |
| Container as default for Postgres | Prefer native module |
| Forgotten port documentation | Update ports.md |
| Nested eval errors | nixos-rebuild build; read trace from deepest file |
| Bind-mounting secrets world-readable | Fix modes; consider sops on host only |
| Expecting VM-level isolation | Adjust threat model; use real VM if needed |
Checkpoint
- Isolation decision record written
- Understand nspawn-style
containers.*vs OCI
- One container ran or justified skip with config draft
- Networking/ports documented
- Comparison note vs native services
- Preview of Day 41 use cases
Commit
git add .
git commit -m "day40: nixos-containers isolation pattern"Write three personal gotchas before continuing.
Tomorrow
Day 41 — Podman/Docker modules: OCI on NixOS with module discipline; rootless notes; compare to Day 40.
Day 41 — Podman/Docker modules
Stage IV · ~4h
Goal: Run OCI containers on NixOS via declarative modules (prefer Podman), understand rootless vs rootful trade-offs, and compare honestly to Day 40 nspawn—without turning the host into an imperative docker run snowflake.
Why this day exists
Day 40 covered NixOS-native containers. The industry default many colleagues know is Docker/Podman + images. On NixOS you should still:
- Prefer a nixpkgs service module when it exists
- When you need an image, declare it in the flake/host modules
- Track ports, volumes, and update cadence
Theory 1 — Podman vs Docker on NixOS
| Engine | Notes |
|---|---|
| Podman | Daemonless roots; good rootless story; often preferred on NixOS labs |
| Docker | Familiar Compose ecosystem; virtualisation.docker module |
House suggestion: Podman unless you must match a Docker-only workflow.
# modules/services/podman.nix
{
virtualisation.podman = {
enable = true;
dockerCompat = true; # optional: docker CLI alias — document if used
};
}Docker alternative sketch:
{
virtualisation.docker.enable = true;
# users.users.alice.extraGroups = [ "docker" ]; # rootful socket risk
}docker group membership is effectively near-root. Prefer rootless or rootful with care.
Theory 2 — Rootless vs rootful
| Mode | Pros | Cons |
|---|---|---|
| Rootless | Better default security | Port <1024 hard; storage quirks |
| Rootful | Simpler networking/ports | Higher blast radius |
Rootless Podman for user workloads; rootful only when needed for lab ports—or put proxy on host and map high ports.
Theory 3 — Declarative containers (OCI)
NixOS options vary by version; common patterns include virtualisation.oci-containers.containers (backend docker/podman).
# modules/services/oci-hello.nix
{
virtualisation.oci-containers = {
backend = "podman";
containers = {
hello-lab = {
image = "docker.io/library/nginx:alpine";
ports = [ "127.0.0.1:8081:80" ];
# volumes = [ "/var/lib/hello-lab:/usr/share/nginx/html:ro" ];
autoStart = true;
};
};
};
# 8081 localhost only — host proxy can front it later
}| Field | Discipline |
|---|---|
image |
Pin tags carefully; digests better for prod |
ports |
Prefer 127.0.0.1: binds |
volumes |
Host paths must be managed (tmpfiles/persist) |
environment |
No secrets in plaintext—use secret files / sops-generated env |
Exact option schema: confirm on search.nixos.org for 26.05.
Theory 4 — Images and supply chain (awareness)
| Practice | Why |
|---|---|
| Pin by digest | Tags move |
| Prefer known registries | Reduce surprise |
| Update deliberately | Same as flake lock discipline |
| Scan when available | Optional tooling |
Nix can build images from packages (dockerTools)—Stage V+ packaging mindset. Today: consume one official image declaratively.
Theory 5 — Compare to Day 40
| Dimension | nspawn containers.* |
OCI Podman |
|---|---|---|
| Guest config | NixOS modules | Image + env/cmd |
| Updates | nixpkgs guest | Image pulls |
| Good for | Nix-first isolation | Vendor images |
| Store integration | Strong | Looser |
Write your own two-sentence preference for the Stage IV slice.
Theory 6 — Firewall and proxy integration
clients → host :443 Caddy/Nginx → 127.0.0.1:8081 → OCI nginx
Do not publish OCI ports on 0.0.0.0 unless docs/ports.md says so and auth exists.
# proxy snippet idea
# reverse_proxy 127.0.0.1:8081Theory 7 — Stateful volumes
If the container needs state:
- Create host directory with tmpfiles or activation
- Bind mount
- Include in backup story
- Never put secrets into world-readable volume without mode care
Worked example — Full small stack slice
{ ... }:
{
imports = [ ./podman.nix ];
virtualisation.oci-containers.backend = "podman";
virtualisation.oci-containers.containers.webish = {
image = "docker.io/library/nginx:alpine";
ports = [ "127.0.0.1:8081:80" ];
autoStart = true;
};
}curl -sS http://127.0.0.1:8081/ | head
podman psLab 1 — Enable Podman (or Docker)
# add module, rebuild
podman info 2>/dev/null || docker info 2>/dev/nullDocument rootless vs rootful choice.
Lab 2 — Declarative OCI container
Define one container with localhost port publish. Rebuild. Verify:
podman ps
curl -sS http://127.0.0.1:8081/ | headLab 3 — Imperative contrast (then undo)
# optional short imperative run to feel the snowflake
podman run --rm -p 127.0.0.1:8082:80 docker.io/library/nginx:alpine
# stop/remove; do NOT leave unmanaged containers as the real setupJournal: why the flake definition wins for rebuildability.
Lab 4 — Wire to proxy (optional but valuable)
Add a reverse_proxy path or separate vhost to the OCI port. Update ports.md.
Lab 5 — Secret anti-pattern check
Confirm no environment = { PASSWORD = "..." } in nix. If the image needs a password, plan sops-backed env file mount (even if deferred).
Lab 6 — Comparison table filled
Complete Day 40 vs Day 41 table in docs/isolation.md with your results (commands, friction, restart behavior).
Theory 8 — Image pins and digests
# Prefer tags you understand; better: digest pins when available
image = "public.ecr.aws/docker/library/nginx:1.27";
# image = "nginx@sha256:…";Floating :latest is the OCI equivalent of an unlocked flake input. For lab demos tags are OK; note the production preference for digests.
Theory 9 — Rootless paths and user namespaces
Rootless Podman stores containers under the user data dir. On NixOS:
| Concern | Note |
|---|---|
| subuid/subgid | User must be mapped |
| Lingering | User services may need lingering for boot-start |
| Ports <1024 | Often need rootful or redirect |
podman info | head
id alice # check subuids if rootless failsWorked example — oci-containers service shape
virtualisation.oci-containers.containers."lab-web" = {
image = "docker.io/library/nginx:1.27";
ports = [ "127.0.0.1:8081:80" ];
autoStart = true;
};Bind published ports to 127.0.0.1 and front with Day 38 proxy—do not publish :80 on all interfaces unless intentional.
Lab 7 — Volume inventory
podman volume ls 2>/dev/null || docker volume ls 2>/dev/null || trueMap each volume to a backup/persist decision in notes (even if “ephemeral OK”).
Lab 8 — Update story one-pager
Write five lines: how you will refresh images, who tests, whether flake lock/image digest changes ship together.
Common gotchas
| Symptom / mistake | What to do |
|---|---|
| Port already in use | Conflict with Day 38 backend—choose free port |
| Image pull fails | Network/registry; mirror; pin |
| Rootless port 80 bind fails | Use high ports + host proxy |
docker group casual |
Avoid; understand privilege |
| State lost on recreate | Bind volume on host |
| Forgot autoStart | Container missing after reboot |
Checkpoint
- Podman or Docker enabled via NixOS module
- One OCI container declarative in flake
- Localhost publish; ports documented
- Imperative snowflake cleaned up
- Isolation docs updated vs nspawn
- No plaintext secrets in container env
Commit
git add .
git commit -m "day41: podman/docker OCI module pattern"Write three personal gotchas before continuing.
Tomorrow
Day 42 — Observability pattern: metrics and/or structured logs for one service—Prometheus + Grafana or a minimal logs-first approach.