Day 14 — PostgreSQL, nixos-containers & Podman

Updated

July 30, 2025

Day 14 — PostgreSQL, nixos-containers & Podman

Day 39 — Data service pattern

Stage IV · ~4h
Goal: Run one data service—PostgreSQL by default—declaratively on NixOS: module options, state directories, app role/auth via secrets paths, and a written backup story (even if automation is minimal).

Important

DB passwords go through sops-nix paths, not Nix strings. Examples use placeholders and path references only.

Why this day exists

Application servers without state are easy. Real systems need durable data:

Concern NixOS angle
Package + unit services.postgresql module
State location services.postgresql.dataDir / ensure on persistent disk
Auth peer/scram; password files from secrets
Network exposure localhost first (Day 37)
Backups Not automatic just because Nix is declarative

Nix rebuilds the software and config; it does not magically version your rows.


Theory 1 — One DB only (pattern depth)

Syllabus: PostgreSQL or another single DB. Depth > zoo.

We use PostgreSQL. If you already run another engine at work, the pattern transfers: module → state dir → credentials → firewall → backups.


Theory 2 — Minimal PostgreSQL module

# modules/services/postgres.nix
{ config, pkgs, ... }:
{
  services.postgresql = {
    enable = true;
    package = pkgs.postgresql_16; # pick a version available on 26.05 pin
    ensureDatabases = [ "labapp" ];
    ensureUsers = [
      {
        name = "labapp";
        ensureDBOwnership = true;
      }
    ];
    # authentication / settings via extra plugins or ensure* options
    settings = {
      # listen_addresses = "localhost"; # confirm option name on your version
    };
  };

  # Do NOT open 5432 on the host firewall for world access in the default lab.
}

Verify option names against search.nixos.org for 26.05.


Theory 3 — State directories and disk

PostgreSQL data lives outside the Nix store (mutable).

Question Answer you must know
Where is dataDir? Document path
Is it on the disk that survives reinstall? Stage VI Disko/impermanence later
What happens on rollback of system generation? Software rolls back; data may not

Impermanence hosts (later) need explicit persistence for DB dirs—note the footgun now.


Theory 4 — Credentials

Patterns:

  1. Peer auth for local system users (good for admin)
  2. SCRAM password for app users—password from sops at runtime
  3. Socket-only access for local apps

Sketch with sops (illustrative—align with module capabilities):

sops.secrets."postgres/labapp_password" = {
  owner = "postgres";
  mode = "0400";
};

# Prefer module-supported passwordFile style options when available.
# If you script role passwords, run a oneshot that reads the secret path
# and does not print it.

Never:

# BAD
services.postgresql.ensureUsers = [{ name = "labapp"; password = "secret"; }];

(If an option even allows raw password—avoid it.)


Theory 5 — App connection story

App config should receive:

  • Host: 127.0.0.1 or socket
  • DB name: labapp
  • User: labapp
  • Password: from env file populated at activation from sops

Your Day 38 proxy does not replace DB auth. Backend talks to DB on localhost; users talk to proxy on 443.


Theory 6 — Backup thought (required cognitive work)

Even a one-node lab needs a plan:

Element Example
What pg_dump of labapp
When nightly timer (system or user)
Where /var/backups/postgres/ on another disk or remote
Encryption restic/age to remote (secrets via sops)
Restore test once per quarter / before capstone

Today: write the plan + optional manual pg_dump once. Full backup automation can be minimal.

sudo -u postgres pg_dump labapp > /tmp/labapp-dump.sql
# store off-box in real life; scrub /tmp later

Theory 7 — Upgrades

Postgres major upgrades are not “change package attr and forget.” They need dump/restore or pg_upgrade procedures. Pin a major version intentionally; schedule upgrades.


Worked example — Service + docs

modules/services/postgres.nix
docs/data-service.md
  - version
  - dataDir
  - DB/user names
  - auth method
  - backup sketch
  - restore sketch

Lab 1 — Enable PostgreSQL

Add module; rebuild; verify unit:

sudo systemctl status postgresql
sudo -u postgres psql -c '\l'

Lab 2 — Database and role

Confirm labapp DB and user via ensure* options. Connect:

sudo -u postgres psql -d labapp -c 'SELECT 1;'

Lab 3 — Wire a secret password path

Create sops secret placeholder for app password. Configure the safest supported method for your module version. Prove app user can authenticate without password living in the flake as plaintext.


Lab 4 — Network posture

ss -tulpn | grep 5432 || true

Confirm listening on localhost (or document intentional LAN bind). Ensure firewall does not expose 5432 publicly.

Update docs/ports.md.


Lab 5 — Manual dump

sudo -u postgres pg_dump labapp | head
# full dump to a safe path you control; do not commit dumps with sensitive data

Write restore commands in docs/data-service.md (even if restore is “psql < dump”).


Lab 6 — Failure narrative

Answer:

  1. System generation rollback after a bad app deploy—does data revert?
  2. Disk failure—what is lost without backups?
  3. Secret rotation steps for DB password


Theory 8 — ensureDatabases vs migrations

ensureDatabases / ensureUsers create empty roles/DBs. Application schema is not Nix’s job by default.

Layer Owner
Package + unit + listen policy NixOS module
Roles/DBs existence ensure* options / oneshots
Schema / migrations App tooling (flyway, django, …)
Data Backups

Do not expect rebuild to replay SQL migrations unless you deliberately automate that (usually outside pure module defaults).


Theory 9 — Socket vs TCP local connections

Postgres often exposes a Unix socket under /run/postgresql. Local admin via sudo -u postgres psql uses peer auth. Apps may use:

host=127.0.0.1 port=5432
# or host=/run/postgresql

Pick one, document in docs/data-service.md, match app secret env.


Worked example — Timer sketch for dumps (optional)

systemd.services.labapp-pg-dump = {
  serviceConfig = {
    Type = "oneshot";
    User = "postgres";
    ExecStart = "${pkgs.bash}/bin/bash -c '${pkgs.postgresql}/bin/pg_dump labapp | ${pkgs.gzip}/bin/gzip -c > /var/backups/labapp.sql.gz'";
  };
};
# + timer + StateDirectory/tmpfiles for /var/backups

Wire carefully (permissions, disk space). Even a commented sketch in the flake beats “we’ll back up later.”


Lab 7 — Schema independence proof

sudo -u postgres psql -d labapp -c "CREATE TABLE day39_demo(id int);"
sudo -u postgres psql -d labapp -c '\dt'
# rebuild switch without dropping data
sudo nixos-rebuild switch --flake .#lab
sudo -u postgres psql -d labapp -c '\dt'

Table should still exist—state ≠ generation.


Lab 8 — Document restore dry-run

Write exact commands in docs (do not need to destroy data):

# restore sketch
# sudo -u postgres psql -d labapp < labapp-dump.sql

Common gotchas

Symptom / mistake What to do
Data in store myth Data is on mutable disk
Open 5432 to world Bind localhost; close firewall
Password in nix sops path / peer auth
Major version bump casual Plan dump/restore
No backup thought Lab 5–6 required
Proxying Postgres over HTTP Wrong layer; don’t

Checkpoint

  • PostgreSQL enabled declaratively
  • App DB + user exist
  • Auth without plaintext secret in git
  • Localhost exposure default
  • Manual dump performed once
  • Backup/restore story written
  • ports.md updated

Commit

git add .
git commit -m "day39: declarative postgres data service pattern"

Write three personal gotchas before continuing.


Tomorrow

Day 40 — nixos-containers / OCI: isolation options on NixOS—declarative nspawn-style containers and when OCI helps without “Docker first” culture.


Day 40 — nixos-containers / OCI

Stage IV · ~4h
Goal: Understand isolation options on NixOS—especially declarative containers.* (nspawn-style) and light OCI awareness—and run one declarative container (or clearly document why your lab uses a pure NixOS service instead).

Note

This day is not “Docker first.” Prefer a native NixOS module when it exists (Days 38–39). Containers help for upstream images, multi-version isolation, or hostile workloads—not as a default personality. Pin remains NixOS 26.05.

Why this day exists

Ecosystem pressure says “everything in containers.” NixOS offers stronger default stories for many services: one store, one firewall story, one secrets story. Still, you need literacy so you can refuse containers for good reasons—or adopt them deliberately.

Tool Isolation style
NixOS systemd services Same host, units, store
containers.<name> systemd-nspawn guest with NixOS config
OCI (Podman/Docker) Image-based; Day 41 modules
VMs / k3s Heavier; Day 43 optional

Skipping this day leaves a blind spot: every vendor README assumes Docker, and you will paste it without knowing what you sacrificed.


Theory 1 — When containers help on NixOS

Use container Prefer native module
Vendor only ships image PostgreSQL, Caddy, Prometheus in nixpkgs
Need older/newer app than nixpkgs Simple Go binary you package in Stage V
Separate failure domain lightly One more systemd service is enough
Multi-tenant sketch Single-user lab
Hostile untrusted binary Still not a full security boundary—see Theory 4

Decision micro-script

Is there a maintained NixOS module?
  yes → use it (Days 38–39)
  no  → Can I package it (Stage V)?
          yes → package + native service
          no  → OCI (Day 41) or nspawn NixOS guest (today)

Theory 2 — containers.<name> (NixOS nspawn)

NixOS can define ephemeral/declarative containers. The guest is a nested NixOS module graph—not a Dockerfile.

# modules/services/demo-container.nix
{
  containers.demo = {
    autoStart = true;
    privateNetwork = true;
    hostAddress = "192.168.100.1";
    localAddress = "192.168.100.2";
    config = { config, pkgs, ... }: {
      system.stateVersion = "26.05";
      networking.firewall.allowedTCPPorts = [ 80 ];
      services.nginx = {
        enable = true;
        virtualHosts."localhost" = {
          root = pkgs.writeTextDir "index.html" "<h1>container</h1>";
        };
      };
      # minimal users/sshd usually not needed for service-only guests
    };
  };
}
Option idea Meaning
autoStart Start at boot
privateNetwork Separate network namespace
hostAddress / localAddress veth pair addressing
bindMounts Host paths into guest
config Nested NixOS module system
ephemeral Lose rootfs state on stop (when supported)

Exact networking options evolve—verify against your 26.05 options docs (man configuration.nix / search.nixos.org) if attributes differ.

Pros / cons

Pros Cons
Declarative guest config Nested complexity / longer eval
Shares host store often Not full VM isolation
Good teaching tool Debug requires container tooling
Rebuild updates guest like host Nested stateVersion still required
sudo machinectl list
sudo nixos-container status demo 2>/dev/null || true
sudo nixos-container root-login demo 2>/dev/null || true

Theory 3 — OCI awareness (without Day 41 depth)

OCI images: layers + config + entrypoint. On NixOS you may:

  • Pull images with Podman/Docker (Day 41)
  • Build images from Nix (dockerTools, nixery, etc.—later packaging awareness)
  • Run rootless vs rootful

Today: know that an image is not a NixOS module. It is a blob with its own update story, CVE surface, and entrypoint that may ignore your host hardening.

NixOS guest (containers.*) OCI image
Config is Nix Config is image + run flags
Updates via host rebuild Updates via pull/tag discipline
Same module options as host Ad-hoc env vars / volumes
Fits flake locks naturally Needs digest pins for honesty

Theory 4 — Security boundaries (honest)

Boundary Strength
Process on host Weak isolation
systemd hardening (ProtectSystem, …) Better, still host kernel
nspawn container Stronger namespaces
OCI with user namespaces Better if configured; easy to misconfigure
VM Stronger (hardware virt)
Separate machine Operational isolation

Do not claim “container = secure” without profile hardening and a patch process for images. Kernel exploits ignore your Docker Compose feelings.

Hardening checklist (guest)

  • Minimal packages in guest
  • Firewall on guest and publish policy on host
  • No unnecessary bind mounts of / or secret dirs
  • No privileged flags unless justified in writing

Theory 5 — State and containers

Guest state can live in:

  • Container root filesystem areas
  • Bind mounts from host paths

Bind mounts must respect secrets and backup stories. Mounting /var/lib/postgresql into a random container without care is a good way to corrupt data.

Pattern Use
Ephemeral guest Stateless demos, rebuild-friendly
Bind /var/lib/app from host Persist data; backup host path
Named volume (OCI) Day 41 territory
Store-only content Ideal for pure static sites

Theory 6 — Relation to proxy / firewall

If a container listens only on a private network, the host proxy may reverse-proxy into it—or you publish a port on the host. Every publish updates docs/ports.md.

[client] → host:443 (caddy/nginx)
              ↓
         192.168.100.2:80 (container)
Mistake Fix
Open guest port on public interface Proxy only; firewall deny direct
Forget ports.md Stage IV gate will bite
Double NAT confusion Draw the path once

Theory 7 — Rebuild UX and nested eval

Changing guest config requires host evaluation of nested modules. Failures look like host rebuild failures with deeper traces.

# build only (no switch) to catch nested errors early
sudo nixos-rebuild build --flake .#lab -L
Friction vs native service Why
Longer eval Nested module system
Harder journal navigation Machine names / nspawn units
Networking options maze Worth a diagram
Worth it when? Real isolation need, not fashion

Worked example — Prefer native, demo container optional

Path A (valid for gate): skip long-lived containers; write a decision record “native modules suffice for Stage IV slice.”

Path B: run the containers.demo nginx guest and curl via host networking setup you chose.

Both paths require written judgment. Path A is not a skip of the day—it is an evidence-backed refusal.


Lab 1 — Decision record

In docs/isolation.md:

# Isolation choices

Stage IV slice services:
- proxy: native module
- postgres: native module
- container: yes/no because …

Boundary strength needed:
Threat model (1 paragraph):
When we would revisit containers:

Lab 2 — Explore host tools

command -v nixos-container || true
command -v machinectl || true
systemctl list-units 'systemd-nspawn@*' 2>/dev/null | head || true
man systemd-nspawn 2>/dev/null | head || true

Record which tools exist on your 26.05 lab host.


Lab 3 — One declarative container or dry config

Implement containers.demo or write the full Nix config in-repo under modules/services/demo-container.nix with enable false and a comment—only if you truly cannot run nspawn in your environment (e.g. restricted VM). Prefer actually running it when possible.

sudo nixos-rebuild switch --flake .#lab
sudo nixos-container list 2>/dev/null || sudo machinectl list

Lab 4 — Reach the guest service

Depending on networking mode, curl from host to guest IP or port. Document the path in docs/isolation.md.

# example privateNetwork style
curl -sS http://192.168.100.2/ | tee ~/lab/90daysofx/02-nixos/day40/curl.txt

Lab 5 — Compare rebuild UX

Change guest index.html content via Nix; rebuild; observe whether container picks up changes cleanly. Note friction vs native services.nginx on host (time to change, journal path, failure modes).


Lab 6 — When you would use OCI instead

Write three bullets for Day 41: cases where an upstream image beats nspawn NixOS guest.


Lab 7 — Firewall / ports doc

Update docs/ports.md (or create it) for any published path. If Path A (no container), document that no extra ports were opened.


Common gotchas

Symptom / mistake What to do
Nested stateVersion missing Set in guest config ("26.05")
Networking black hole Start with simpler network mode; read options
Assuming Docker API That’s Day 41
Container as default for Postgres Prefer native module
Forgotten port documentation Update ports.md
Nested eval errors nixos-rebuild build; read trace from deepest file
Bind-mounting secrets world-readable Fix modes; consider sops on host only
Expecting VM-level isolation Adjust threat model; use real VM if needed

Checkpoint

  • Isolation decision record written
  • Understand nspawn-style containers.* vs OCI
  • One container ran or justified skip with config draft
  • Networking/ports documented
  • Comparison note vs native services
  • Preview of Day 41 use cases

Commit

git add .
git commit -m "day40: nixos-containers isolation pattern"

Write three personal gotchas before continuing.


Tomorrow

Day 41 — Podman/Docker modules: OCI on NixOS with module discipline; rootless notes; compare to Day 40.


Day 41 — Podman/Docker modules

Stage IV · ~4h
Goal: Run OCI containers on NixOS via declarative modules (prefer Podman), understand rootless vs rootful trade-offs, and compare honestly to Day 40 nspawn—without turning the host into an imperative docker run snowflake.

Why this day exists

Day 40 covered NixOS-native containers. The industry default many colleagues know is Docker/Podman + images. On NixOS you should still:

  1. Prefer a nixpkgs service module when it exists
  2. When you need an image, declare it in the flake/host modules
  3. Track ports, volumes, and update cadence

Theory 1 — Podman vs Docker on NixOS

Engine Notes
Podman Daemonless roots; good rootless story; often preferred on NixOS labs
Docker Familiar Compose ecosystem; virtualisation.docker module

House suggestion: Podman unless you must match a Docker-only workflow.

# modules/services/podman.nix
{
  virtualisation.podman = {
    enable = true;
    dockerCompat = true; # optional: docker CLI alias — document if used
  };
}

Docker alternative sketch:

{
  virtualisation.docker.enable = true;
  # users.users.alice.extraGroups = [ "docker" ]; # rootful socket risk
}

docker group membership is effectively near-root. Prefer rootless or rootful with care.


Theory 2 — Rootless vs rootful

Mode Pros Cons
Rootless Better default security Port <1024 hard; storage quirks
Rootful Simpler networking/ports Higher blast radius

Rootless Podman for user workloads; rootful only when needed for lab ports—or put proxy on host and map high ports.


Theory 3 — Declarative containers (OCI)

NixOS options vary by version; common patterns include virtualisation.oci-containers.containers (backend docker/podman).

# modules/services/oci-hello.nix
{
  virtualisation.oci-containers = {
    backend = "podman";
    containers = {
      hello-lab = {
        image = "docker.io/library/nginx:alpine";
        ports = [ "127.0.0.1:8081:80" ];
        # volumes = [ "/var/lib/hello-lab:/usr/share/nginx/html:ro" ];
        autoStart = true;
      };
    };
  };

  # 8081 localhost only — host proxy can front it later
}
Field Discipline
image Pin tags carefully; digests better for prod
ports Prefer 127.0.0.1: binds
volumes Host paths must be managed (tmpfiles/persist)
environment No secrets in plaintext—use secret files / sops-generated env

Exact option schema: confirm on search.nixos.org for 26.05.


Theory 4 — Images and supply chain (awareness)

Practice Why
Pin by digest Tags move
Prefer known registries Reduce surprise
Update deliberately Same as flake lock discipline
Scan when available Optional tooling

Nix can build images from packages (dockerTools)—Stage V+ packaging mindset. Today: consume one official image declaratively.


Theory 5 — Compare to Day 40

Dimension nspawn containers.* OCI Podman
Guest config NixOS modules Image + env/cmd
Updates nixpkgs guest Image pulls
Good for Nix-first isolation Vendor images
Store integration Strong Looser

Write your own two-sentence preference for the Stage IV slice.


Theory 6 — Firewall and proxy integration

clients → host :443 Caddy/Nginx → 127.0.0.1:8081 → OCI nginx

Do not publish OCI ports on 0.0.0.0 unless docs/ports.md says so and auth exists.

# proxy snippet idea
# reverse_proxy 127.0.0.1:8081

Theory 7 — Stateful volumes

If the container needs state:

  1. Create host directory with tmpfiles or activation
  2. Bind mount
  3. Include in backup story
  4. Never put secrets into world-readable volume without mode care

Worked example — Full small stack slice

{ ... }:
{
  imports = [ ./podman.nix ];

  virtualisation.oci-containers.backend = "podman";
  virtualisation.oci-containers.containers.webish = {
    image = "docker.io/library/nginx:alpine";
    ports = [ "127.0.0.1:8081:80" ];
    autoStart = true;
  };
}
curl -sS http://127.0.0.1:8081/ | head
podman ps

Lab 1 — Enable Podman (or Docker)

# add module, rebuild
podman info 2>/dev/null || docker info 2>/dev/null

Document rootless vs rootful choice.


Lab 2 — Declarative OCI container

Define one container with localhost port publish. Rebuild. Verify:

podman ps
curl -sS http://127.0.0.1:8081/ | head

Lab 3 — Imperative contrast (then undo)

# optional short imperative run to feel the snowflake
podman run --rm -p 127.0.0.1:8082:80 docker.io/library/nginx:alpine
# stop/remove; do NOT leave unmanaged containers as the real setup

Journal: why the flake definition wins for rebuildability.


Lab 4 — Wire to proxy (optional but valuable)

Add a reverse_proxy path or separate vhost to the OCI port. Update ports.md.


Lab 5 — Secret anti-pattern check

Confirm no environment = { PASSWORD = "..." } in nix. If the image needs a password, plan sops-backed env file mount (even if deferred).


Lab 6 — Comparison table filled

Complete Day 40 vs Day 41 table in docs/isolation.md with your results (commands, friction, restart behavior).



Theory 8 — Image pins and digests

# Prefer tags you understand; better: digest pins when available
image = "public.ecr.aws/docker/library/nginx:1.27";
# image = "nginx@sha256:…";

Floating :latest is the OCI equivalent of an unlocked flake input. For lab demos tags are OK; note the production preference for digests.


Theory 9 — Rootless paths and user namespaces

Rootless Podman stores containers under the user data dir. On NixOS:

Concern Note
subuid/subgid User must be mapped
Lingering User services may need lingering for boot-start
Ports <1024 Often need rootful or redirect
podman info | head
id alice  # check subuids if rootless fails

Worked example — oci-containers service shape

virtualisation.oci-containers.containers."lab-web" = {
  image = "docker.io/library/nginx:1.27";
  ports = [ "127.0.0.1:8081:80" ];
  autoStart = true;
};

Bind published ports to 127.0.0.1 and front with Day 38 proxy—do not publish :80 on all interfaces unless intentional.


Lab 7 — Volume inventory

podman volume ls 2>/dev/null || docker volume ls 2>/dev/null || true

Map each volume to a backup/persist decision in notes (even if “ephemeral OK”).


Lab 8 — Update story one-pager

Write five lines: how you will refresh images, who tests, whether flake lock/image digest changes ship together.

Common gotchas

Symptom / mistake What to do
Port already in use Conflict with Day 38 backend—choose free port
Image pull fails Network/registry; mirror; pin
Rootless port 80 bind fails Use high ports + host proxy
docker group casual Avoid; understand privilege
State lost on recreate Bind volume on host
Forgot autoStart Container missing after reboot

Checkpoint

  • Podman or Docker enabled via NixOS module
  • One OCI container declarative in flake
  • Localhost publish; ports documented
  • Imperative snowflake cleaned up
  • Isolation docs updated vs nspawn
  • No plaintext secrets in container env

Commit

git add .
git commit -m "day41: podman/docker OCI module pattern"

Write three personal gotchas before continuing.


Tomorrow

Day 42 — Observability pattern: metrics and/or structured logs for one service—Prometheus + Grafana or a minimal logs-first approach.