Day 13 — Hardening, Firewall & TLS Reverse Proxy

Updated

July 30, 2025

Day 13 — Hardening, Firewall & TLS Reverse Proxy

Day 36 — Hardening baseline

Stage IV · ~4h
Goal: Apply a small, documented hardening module for a lab server: SSH posture, sudo policy, disable unused services, and careful sysctl/kernel defaults—without cargo-culting every CIS line item.

Warning

Hardening can lock you out of a remote VM. Keep a console/VNC path or test with nixos-rebuild test / boot generation rollback ready (Day 13).

Why this day exists

Secrets tooling protects credentials. Hardening reduces attack surface and enforces least privilege. Production-shaped labs should not run with:

Password SSH
Root SSH login
Unnecessary network services
Empty sudo policies that surprise you later

Today builds a module you can import on server roles—not a security theater checklist of 200 toggles.

Theory 1 — Threat model for this lab

Assume:

Asset	Threat
SSH on a public IP	Password spray, key theft
Local users	Accidental privilege escalation
Services you enable in Stage IV	Extra open ports
Laptop multi-boot lab	Lower remote risk, still practice

Non-goals today: full SELinux policy authoring, corporate MDM, compliance certifications.

Theory 2 — SSH baseline (NixOS)

# modules/common/hardening/ssh.nix
{
  services.openssh = {
    enable = true;
    settings = {
      PasswordAuthentication = false;
      KbdInteractiveAuthentication = false;
      PermitRootLogin = "no";
      X11Forwarding = false;
      # MaxAuthTries = 4;
    };
  };
}

Setting	Intent
No passwords	Keys only
No root login	Use `sudo` from admin user
No X11 forward	Reduce attack surface on servers

Ensure your user has openssh.authorizedKeys.keys before applying password-off if you ever had passwords.

Theory 3 — sudo and wheel

# modules/common/hardening/sudo.nix
{
  security.sudo = {
    enable = true;
    wheelNeedsPassword = true; # or false for lab convenience — document choice
    execWheelOnly = true;
  };
  # users.users.alice.extraGroups = [ "wheel" ];
}

Policy	Trade-off
`wheelNeedsPassword = true`	Safer; needs password/TTY
`= false`	Convenient lab; weaker if key stolen + local

Prefer true on anything network-exposed.

Theory 4 — Disable what you do not use

Examples (only if present/enabled):

{
  services.avahi.enable = false;
  services.printing.enable = false;
  # bluetooth on servers:
  hardware.bluetooth.enable = false;
}

Do not randomly disable things you depend on (NetworkManager on a laptop role). Use roles:

# modules/roles/server.nix
{
  imports = [ ../common/hardening ];
  # server-specific disables
}

# modules/roles/workstation.nix
{
  # lighter hardening; keep user conveniences
}

Theory 5 — Kernel / sysctl care

NixOS exposes many boot.kernel.sysctl knobs. Apply few, understand each:

{
  boot.kernel.sysctl = {
    "net.ipv4.conf.all.rp_filter" = 1;
    "net.ipv4.tcp_syncookies" = 1;
    # Do not paste a 50-line blog dump without reading
  };
}

Kernel package pinning and hardened profiles exist in the ecosystem; treat full “hardened kernel” as optional advanced work—not required for gate IV.

Theory 6 — Automatic updates? (judgment)

Unattended upgrades on NixOS are a product decision:

Flake pins mean “latest” is not automatic unless you automate lock bumps + rebuilds
Auto-reboot policies can surprise

For this book: manual, deliberate updates (Day 24) unless you later build CI-driven update PRs.

Theory 7 — Module shape

modules/common/hardening/
  default.nix    # imports ssh.nix sudo.nix misc.nix
  ssh.nix
  sudo.nix
  misc.nix

# default.nix
{
  imports = [ ./ssh.nix ./sudo.nix ./misc.nix ];
}

Import from server role or common—document if workstations share it.

Worked example — Safe application order

Confirm console access
Confirm admin key SSH works
nixos-rebuild boot (activate next reboot) or test
Open new SSH session before closing old
Only then switch if using switch

# keep existing session; test new config in another tty/ssh
sudo nixos-rebuild test --flake .#lab

Lab 1 — Inventory surface

ss -tulpn | sort
systemctl list-unit-files --state=enabled | head -80

Journal: what is listening, what you do not recognize.

Lab 2 — Write hardening module

Create modules/common/hardening/ with SSH + sudo + 2–3 sensible disables for your role.

Lab 3 — Apply carefully

sudo nixos-rebuild test --flake .#lab
# new ssh session
ssh lab
sudo -v

Then switch if satisfied.

Lab 4 — Lockout recovery drill (document)

Write exact steps to recover if SSH rejects you:

Hypervisor console
Boot previous generation
nixos-rebuild switch --rollback from console

Do not skip writing this.

Lab 5 — Assertions (optional polish)

{ config, lib, ... }:
{
  assertions = [
    {
      assertion = config.services.openssh.settings.PasswordAuthentication == false;
      message = "PasswordAuthentication must be false on this host role";
    }
  ];
}

Lab 6 — Diff and explain

# compare pre/post listening ports
ss -tulpn

Explain each remaining listener in one line (sshd, etc.).

Common gotchas

Symptom / mistake	What to do
Locked out via SSH	Console + previous generation
PasswordAuth off before keys	Fix via console; add keys
Hardening module on laptop breaks UX	Split server/workstation roles
Blind sysctl paste	Remove unknowns
`PermitRootLogin yes` “for debug” left on	Remove before network exposure
No recovery notes	Write Lab 4 before remote travel

Checkpoint

Hardening module exists and is imported on the lab role
Password SSH off; root SSH off
sudo policy explicit
Unnecessary services disabled with reasons
Recovery path documented
Rebuild + new SSH session verified

Commit

git add .
git commit -m "day36: hardening baseline module"

Write three personal gotchas before continuing.

Tomorrow

Day 37 — Firewall discipline: deny-by-default posture, per-service ports, and an audit of what is actually open.

Day 37 — Firewall discipline

Stage IV · ~4h
Goal: Treat the NixOS firewall as a deny-by-default policy: open only ports you can name, tie openings to services, and audit with tools—not hope.

Why this day exists

Day 15 introduced networking/firewall basics. Stage IV services (proxy, DB, metrics) tempt:

networking.firewall.enable = false; # "lab"

or:

networking.firewall.allowedTCPPorts = [ 22 80 443 5432 3000 9090 9100 /* … */ ];

Without a story. Today is discipline: every open port has an owner service and a reason.

Theory 1 — NixOS firewall model

With networking.firewall.enable = true (default on many installs):

Mechanism	Role
nftables/iptables backend	Implementation (prefer not hand-editing)
`allowedTCPPorts` / `UDP`	Host-wide opens
`allowedTCPPortRanges`	Ranges when unavoidable
Interface-specific options	Limit exposure to LAN interfaces
Service modules	Often open ports via `openFirewall` options

Prefer service options

services.openssh.enable = true;
# many services offer:
# services.foo.openFirewall = true;

That keeps port + service definition adjacent. Global lists become a last resort.

Theory 2 — Deny by default

{
  networking.firewall = {
    enable = true;
    allowPing = true; # or false on hardened internet hosts
    allowedTCPPorts = [ ];
    allowedUDPPorts = [ ];
  };
}

Then add ports only with services. SSH often opened by services.openssh + related settings depending on config—verify rather than assume.

Check:

sudo nixos-option networking.firewall.allowedTCPPorts
# also inspect effective rules:
sudo nft list ruleset 2>/dev/null | head -100
# or iptables-save depending on backend

Theory 3 — Per-service matrix (living document)

Maintain docs/ports.md:

Port	Proto	Service	Source scope	Module
22	TCP	sshd	admin IPs / world (lab)	hardening/ssh
443	TCP	caddy/nginx	world	reverse-proxy
5432	TCP	postgres	localhost only	data service

If Postgres is localhost-only, do not open 5432 on the firewall.

Theory 4 — Localhost vs LAN vs world

Scope	Pattern
Localhost only	Bind service to `127.0.0.1`; no firewall open
LAN only	Firewall interface restrictions / not exposing WAN NIC
World	80/443 typically; auth/TLS required

# example idea — postgres listen addresses
services.postgresql.enable = true;
# ensure listen_addresses = 'localhost' for lab default

Exact options depend on module—read search.nixos.org for your pin.

Theory 5 — Temporary debug holes

Bad habit: leave allowedTCPPorts = [ 3000 ] forever for a dev server.

Better:

SSH tunnel: ssh -L 3000:127.0.0.1:3000 lab
Or short-lived config on a branch
Or VPN/Tailscale patterns (out of scope unless you already use them)

Theory 6 — Interaction with containers (preview)

Containers and port publishes can bypass your mental model if you open host ports carelessly. Days 40–41: treat published ports as first-class firewall entries in docs/ports.md.

Worked example — Tight SSH + later HTTPS only

# modules/common/firewall.nix
{ lib, ... }:
{
  networking.firewall = {
    enable = true;
    allowPing = true;
    # Explicit list kept tiny; prefer service openFirewall flags
    allowedTCPPorts = [
      # 80 443 added on Day 38 with the proxy module
    ];
  };
}

When proxy lands:

# modules/services/proxy.nix
{
  networking.firewall.allowedTCPPorts = [ 80 443 ];
  # or services.caddy/nginx openFirewall-style options if available
}

Lab 1 — Audit current opens

ss -tulpn
sudo nft list ruleset 2>/dev/null | head -150
# document everything listening

Fill the ports table for current state.

Lab 2 — Enable firewall explicitly

Ensure networking.firewall.enable = true. Rebuild carefully (console ready).

sudo nixos-rebuild test --flake .#lab

Confirm SSH still works from a new session.

Lab 3 — Remove unjustified ports

Delete any “temporary” ports. Rebuild. Re-audit ss and ruleset.

Lab 4 — Service-tied open (dry for tomorrow)

Add a comment module or stub:

# modules/services/proxy-firewall.nix
# Day 38 will open 80/443 here when proxy is enabled

Wire the idea that proxy owns those ports.

Lab 5 — Localhost service experiment

Run a quick local listener (non-Nix, temporary) or use an existing local-only service; confirm it is not reachable from another machine without a firewall hole.

# on lab: nc -l 127.0.0.1 9999
# from another host: should fail unless you opened the port AND bound 0.0.0.0

Lab 6 — ports.md committed

Commit docs/ports.md with the matrix. Update it on Days 38–42 as services arrive.

Theory 7 — `networking.firewall` interfaces and trusted nets

networking.firewall = {
  enable = true;
  # trustedInterfaces = [ "tailscale0" ]; # example VPN iface — only if real
  interfaces."eth0".allowedTCPPorts = [ 22 ]; # shape varies; verify options
};

Prefer least exposure: admin SSH on a management interface if you have one; never open DB ports “because the app is in the same VPC narrative” without proof.

Cloud security groups

NixOS firewall is not a substitute for cloud SG/NSG rules—and vice versa. Document both layers in docs/ports.md when the lab is a VPS.

Worked example — Service `openFirewall` pattern

# Prefer when the module offers it:
services.nginx.enable = true;
# services.nginx... openFirewall-style options — check 26.05 module

# Fallback explicit ownership in the same module file:
# networking.firewall.allowedTCPPorts = [ 80 443 ];

Co-locate port opens with the service module so deleting the service deletes the hole.

Lab 7 — Diff firewall across generations (optional)

# After a known-good generation, change ports, rebuild, then:
sudo nft list ruleset > /tmp/fw-after.txt
# compare mentally to docs/ports.md — docs must match reality

Lab 8 — Ping policy decision

networking.firewall.allowPing = true; # or false

Write one sentence: who needs ICMP on this host? Internet-facing hardened hosts often disable; home LAN labs often keep ping for debugging.

Common gotchas

Symptom / mistake	What to do
Locked out of SSH	Console; previous generation; re-open 22
Firewall on but service bound wrong	Fix listen address
Docker-ish publishes confuse audit	Inventory publish flags (Day 41)
`enable = false` “for lab”	Re-enable; use tunnels for debug
Open 5432 to world	Bind localhost; proxy/app only
Ruleset empty surprise	Backend differ; verify with `ss` too

Checkpoint

Firewall enabled
Every open port documented in docs/ports.md
No unjustified temporary holes
SSH verified after firewall changes
Localhost vs world distinction understood
Plan for Day 38 ports owned by proxy module

Commit

git add .
git commit -m "day37: firewall discipline and ports inventory"

Write three personal gotchas before continuing.

Tomorrow

Day 38 — TLS front door: ACME + Caddy or Nginx—one reverse-proxy pattern with HTTPS to a backend.

Day 38 — TLS front door

Stage IV · ~4h
Goal: Put a TLS reverse proxy in front of a simple backend using ACME (Let’s Encrypt or lab alternative)—pick Caddy or Nginx, not both, as the house reverse-proxy pattern.

Note

Public ACME needs a public DNS name pointing at your lab. If you cannot, use internal TLS, mkcert, or Caddy/Nginx self-signed lab modes—and document the deviation. The pattern still counts.

Why this day exists

Stage IV services should not all expose their own TLS stacks. The production pattern:

Internet/LAN clients
    → :443 reverse proxy (TLS terminates)
        → 127.0.0.1:app-port (HTTP plain on localhost)

One certificate story, one cipher/policy place, one set of firewall holes (80/443).

Theory 1 — Choose one proxy

Proxy	Strengths	Notes
Caddy	Automatic HTTPS UX, simple config	Great default for labs
Nginx	Ubiquitous, explicit	ACME via `security.acme` + nginx module

Pick one for the rest of Stage IV. Write the choice in docs/proxy.md.

Theory 2 — ACME on NixOS

NixOS security.acme (used heavily with nginx) or Caddy’s built-in ACME:

Concern	Practice
Email	Real email for Let’s Encrypt notices
Domains	You control DNS
HTTP-01	Port 80 reachable from internet
DNS-01	Provider token via sops (not plaintext)
Staging	Use LE staging while debugging rate limits

Secrets

ACME account keys are managed on disk by the tooling—do not invent plaintext private keys in the flake. DNS provider tokens: sops-nix (Day 34).

Theory 3 — Caddy pattern (NixOS)

# modules/services/proxy-caddy.nix
{ pkgs, ... }:
{
  networking.firewall.allowedTCPPorts = [ 80 443 ];

  services.caddy = {
    enable = true;
    # Example virtual host — replace domain
    virtualHosts."app.example.invalid".extraConfig = ''
      encode gzip
      reverse_proxy 127.0.0.1:8080
    '';
  };
}

Caddy often obtains certs automatically when DNS is real. For invalid/lab domains, configure accordingly or use a local-only site block for HTTP lab testing.

Static site alternative:

services.caddy.virtualHosts."static.example.invalid".extraConfig = ''
  root * ${pkgs.writeTextDir "index.html" "<h1>lab</h1>"}
  file_server
'';

(Adjust to current Caddy module options on 26.05 if attribute names differ slightly—verify with option search.)

Theory 4 — Nginx + ACME pattern

# modules/services/proxy-nginx.nix
{
  networking.firewall.allowedTCPPorts = [ 80 443 ];

  security.acme = {
    acceptTerms = true;
    defaults.email = "you@example.invalid";
  };

  services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;
    virtualHosts."app.example.invalid" = {
      enableACME = true;
      forceSSL = true;
      locations."/" = {
        proxyPass = "http://127.0.0.1:8080";
        proxyWebsockets = true;
      };
    };
  };
}

Theory 5 — Backend for the day

Minimal backend options:

Static files via the proxy itself
python -m http.server style temporary (imperative only for proof—prefer declarative)
Tiny declarative service:

systemd.services.lab-backend = {
  wantedBy = [ "multi-user.target" ];
  serviceConfig = {
    DynamicUser = true;
    ExecStart = "${pkgs.python3}/bin/python3 -m http.server 8080 --bind 127.0.0.1";
    WorkingDirectory = "${pkgs.writeTextDir "index.html" "<h1>backend</h1>"}";
  };
};

Bind 127.0.0.1 so the firewall need not open 8080.

Theory 6 — Failure modes

Symptom	Checks
ACME fail	DNS A/AAAA, port 80, rate limits, email/terms
502 bad gateway	Backend down; wrong port; SE/socket
Cert works, app not	`proxyPass` URL
Works on host, not remote	Firewall 80/443; cloud SG

curl -vI https://app.example.invalid
journalctl -u caddy -b --no-pager | tail -50
# or nginx / acme services

Theory 7 — Update ports.md

Port	Service
80	ACME HTTP-01 / redirect
443	HTTPS proxy
8080	localhost backend only

Lab 1 — Pick proxy and document

Write docs/proxy.md with Caddy or Nginx and domain plan (public / LAN / self-signed).

Lab 2 — Localhost backend

Add declarative backend on 127.0.0.1:8080. Prove:

curl -sS http://127.0.0.1:8080/ | head

From another machine, connection should fail if not firewalled open.

Lab 3 — Enable proxy module

Import proxy module; open 80/443; rebuild.

sudo nixos-rebuild switch --flake .#lab

Lab 4 — TLS path

If public DNS: complete ACME; curl -I https://your.domain.

If not: configure self-signed or internal CA approach; curl -k for lab proof; document how production would differ.

Lab 5 — Headers and hygiene (light)

Enable recommended TLS/proxy settings (nginx flags above, or Caddy defaults). Confirm you do not open backend port publicly.

Lab 6 — Break/fix

Stop backend; observe 502; start backend; recover. Journal the unit names.

Theory 8 — Certificate storage and renewal

ACME clients store account/certs under state dirs (often under /var/lib). That state is not in the Nix store.

Event	Result
System generation rollback	Software/config may roll back; certs usually remain
Wipe `/var`	Certificates gone; re-issue (rate limits!)
Domain change	New cert path; update proxy vhost

Back up ACME state on anything beyond a disposable lab, or accept re-issuance cost.

Theory 9 — HTTP → HTTPS redirect

# nginx: forceSSL + enableACME typically redirects
# caddy: automatic HTTPS often redirects once certs work

Test both:

curl -I http://app.example.invalid
curl -I https://app.example.invalid

Expect redirect or secure connection—not a permanent plain HTTP production path.

Worked example — Internal TLS without public ACME

# Conceptual lab path — exact options differ by proxy:
# - generate mkcert/local CA on admin machine
# - place cert/key via sops-nix into /run/secrets
# - point proxy ssl_certificate paths at those files

Document production delta: “In prod we would use ACME DNS-01 / HTTP-01; lab uses internal CA.”

Lab 7 — ports.md + proxy.md dual-write

Ensure both docs agree on:

Item	Value
Proxy engine	Caddy / Nginx
Public ports	80, 443
Backend	127.0.0.1:8080
TLS mode	ACME / internal / self-signed

Lab 8 — Certificate expiry awareness

echo | openssl s_client -connect 127.0.0.1:443 -servername app.example.invalid 2>/dev/null | openssl x509 -noout -dates

Even on lab self-signed, practice reading notAfter.

Common gotchas

Symptom / mistake	What to do
LE rate limited	Use staging; wait; fix DNS first
Both Caddy and Nginx on 443	Pick one
Backend on 0.0.0.0:8080	Bind localhost
Secrets for DNS-01 in git	sops
IPv6 DNS mismatch	Align AAAA or disable
Forgot port 80 for HTTP-01	Open 80 or use DNS-01

Checkpoint

Commit

git add .
git commit -m "day38: TLS reverse proxy front door"

Write three personal gotchas before continuing.

Tomorrow

Day 39 — Data service pattern: declarative PostgreSQL (or one DB), app role, state directories, and a backup thought exercise.

Day 13 — Hardening, Firewall & TLS Reverse Proxy

Day 36 — Hardening baseline

Why this day exists

Theory 1 — Threat model for this lab

Theory 2 — SSH baseline (NixOS)

Theory 3 — sudo and wheel

Theory 4 — Disable what you do not use

Theory 5 — Kernel / sysctl care

Theory 6 — Automatic updates? (judgment)

Theory 7 — Module shape

Worked example — Safe application order

Lab 1 — Inventory surface

Lab 2 — Write hardening module

Lab 3 — Apply carefully

Lab 4 — Lockout recovery drill (document)

Lab 5 — Assertions (optional polish)

Lab 6 — Diff and explain

Common gotchas

Checkpoint

Commit

Tomorrow

Day 37 — Firewall discipline

Why this day exists

Theory 1 — NixOS firewall model

Prefer service options

Theory 2 — Deny by default

Theory 3 — Per-service matrix (living document)

Theory 4 — Localhost vs LAN vs world

Theory 5 — Temporary debug holes

Theory 6 — Interaction with containers (preview)

Worked example — Tight SSH + later HTTPS only

Lab 1 — Audit current opens

Lab 2 — Enable firewall explicitly

Lab 3 — Remove unjustified ports

Lab 4 — Service-tied open (dry for tomorrow)

Lab 5 — Localhost service experiment

Lab 6 — ports.md committed

Theory 7 — networking.firewall interfaces and trusted nets

Cloud security groups

Theory 8 — Connection tracking and related traffic

Worked example — Service openFirewall pattern

Lab 7 — Diff firewall across generations (optional)

Lab 8 — Ping policy decision

Common gotchas

Checkpoint

Commit

Tomorrow

Day 38 — TLS front door

Why this day exists

Theory 1 — Choose one proxy

Theory 2 — ACME on NixOS

Secrets

Theory 3 — Caddy pattern (NixOS)

Theory 4 — Nginx + ACME pattern

Theory 5 — Backend for the day

Theory 6 — Failure modes

Theory 7 — Update ports.md

Lab 1 — Pick proxy and document

Lab 2 — Localhost backend

Lab 3 — Enable proxy module

Lab 4 — TLS path

Lab 5 — Headers and hygiene (light)

Lab 6 — Break/fix

Theory 8 — Certificate storage and renewal

Theory 9 — HTTP → HTTPS redirect

Worked example — Internal TLS without public ACME

Lab 7 — ports.md + proxy.md dual-write

Lab 8 — Certificate expiry awareness

Common gotchas

Checkpoint

Commit

Tomorrow

Theory 7 — `networking.firewall` interfaces and trusted nets

Worked example — Service `openFirewall` pattern