Day 68 — Containers & distroless

Updated

July 30, 2026

Day 68 — Containers & distroless

Stage VII · ~3h
Goal: Multi-stage Docker builds that ship a small, non-root Go binary—preferring distroless or scratch—and document ports, health, and memory limits for the runtime.

Note

Today freezes how the process ships, not business logic. A bad Dockerfile undoes every Stage VII win: huge images, root processes, glibc surprises, and rebuilds that redownload the internet.

Why this day exists

Cross-compiled binaries (Day 63) still need a runtime environment for most services. Containers:

Freeze OS libs and layout
Make “works on my machine” less relevant
Are the unit of deploy for many teams

Go is unusually good at small images because CGO_ENABLED=0 produces a static binary. That advantage only appears if the Dockerfile is multi-stage and the runtime base is minimal.

Theory 1 — Why Go loves multi-stage builds

Typical pattern:

Builder stage: full Go toolchain, compile with CGO_ENABLED=0
Runtime stage: only the binary (+ certs, timezone data if needed)

# syntax=docker/dockerfile:1
FROM golang:1.26-bookworm AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app"]

Base	Use when
`scratch`	Truly static binary; you must add CA certs if you dial TLS
`distroless/static`	Static binary + minimal filesystem; common default
`distroless/base`	Needs libc (rare if `CGO_ENABLED=0`)
`alpine`	Shell debugging convenience; larger attack surface than distroless

What multi-stage actually removes

The builder image includes compilers, headers, package managers, and source. None of that belongs in production. The runtime image should answer only: “Can this process start, listen, and make required network calls?”

Theory 2 — Certificates and timezones

Many apps call HTTPS. Distroless static images usually include CA certs; pure scratch often does not:

FROM alpine:3.20 AS certs
RUN apk add --no-cache ca-certificates tzdata

FROM scratch
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /out/app /app
ENTRYPOINT ["/app"]

Timezone: prefer UTC in servers. If you need time.LoadLocation("America/New_York"), ensure zoneinfo is available or embed carefully (//go:embed zone files only when required).

Quick verification after run

# does outbound TLS work from inside the container?
docker run --rm day68:local /app -check-tls   # if you add a debug flag
# or exec into a debug sidecar; do not keep a shell in prod images

Theory 3 — Non-root, read-only, signals

Run as nonroot (distroless provides nonroot).
Listen on non-privileged ports (8080).
Go’s http.Server.Shutdown needs SIGTERM delivered to PID 1—containers send SIGTERM on stop; ensure your process is PID 1 (ENTRYPOINT binary, not a shell wrapper that ignores signals).

# Avoid: ENTRYPOINT ["sh", "-c", "./app"]  unless exec-form handles signals
ENTRYPOINT ["/app"]

Graceful shutdown code lands Day 78; today ensure the image can receive signals cleanly.

Read-only root filesystem (awareness)

Orchestrators often set readOnlyRootFilesystem: true. Your app must not write to / casually—use /tmp volume or emptyDir for scratch files. Design for that now:

// prefer explicit temp dir
dir := os.Getenv("TMPDIR")
if dir == "" {
    dir = os.TempDir()
}

Theory 4 — Build cache and reproducibility

COPY go.mod go.sum ./
RUN go mod download
COPY . .

Order matters: dependency layer caches until go.mod/go.sum change. Source copy invalidates later layers only.

docker build -t day68-app:dev .
docker run --rm -p 8080:8080 day68-app:dev
docker image ls day68-app

Build flags that matter

Flag	Why
`CGO_ENABLED=0`	Static binary; scratch/distroless-friendly
`-trimpath`	Remove local paths from binaries
`-ldflags="-s -w"`	Strip symbol/DWARF tables (smaller; harder to debug)
`GOOS=linux`	Cross-build from macOS host when needed

Memory alignment with Day 67

docker run --rm -p 8080:8080 -m 256m -e GOMEMLIMIT=200MiB day68-app:dev

Leave headroom under the hard cgroup limit. GOMEMLIMIT (soft) should sit below Docker’s -m (hard).

Theory 5 — `.dockerignore` and context size

Docker sends the build context to the daemon. Huge contexts slow every build:

.git
**/.git
**/_book
**/*.md
bin/
dist/
*.out
*.pb.gz
**/*_test.go
.idea
.vscode

Exclude test-only bulk when the image does not run tests. Keep go.sum in the context.

Comparing single-stage vs multi-stage

# naive (do not ship)
# FROM golang:1.26-bookworm
# COPY . /src && RUN go build -o /app ...
# → hundreds of MB, root, full toolchain

docker images --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}' | grep day68

Record sizes in CONTAINER.md. Expect multi-stage distroless to be orders of magnitude smaller than a single-stage golang image.

Theory 6 — Health, ports, and compose sketch

Document for operators:

Item	Example
Listen	`:8080` via `ADDR`
Health	`GET /healthz` → 200
Metrics	`GET /metrics` (Day 69)
Stop signal	SIGTERM; grace ≥ shutdown timeout

Optional local compose.yaml (not required for the lab):

services:
  app:
    build: .
    ports: ["8080:8080"]
    environment:
      ADDR: ":8080"
      GOMEMLIMIT: "200MiB"
    mem_limit: 256m
    read_only: true
    tmpfs: ["/tmp"]

Worked example — minimal HTTP service

// cmd/app/main.go
package main

import (
    "encoding/json"
    "log"
    "net/http"
    "os"
)

func main() {
    addr := env("ADDR", ":8080")
    mux := http.NewServeMux()
    mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        _, _ = w.Write([]byte("ok"))
    })
    mux.HandleFunc("GET /v1/hello", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        _ = json.NewEncoder(w).Encode(map[string]string{"msg": "hello"})
    })
    log.Printf("listening on %s", addr)
    log.Fatal(http.ListenAndServe(addr, mux))
}

func env(k, def string) string {
    if v := os.Getenv(k); v != "" {
        return v
    }
    return def
}

# Dockerfile
# syntax=docker/dockerfile:1
FROM golang:1.26-bookworm AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/app

FROM gcr.io/distroless/static-debian12:nonroot
WORKDIR /
COPY --from=build /out/app /app
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app"]

docker build -t day68:local .
docker run --rm -p 8080:8080 day68:local
curl -s localhost:8080/healthz
docker images day68:local --format '{{.Size}}'

Debug without polluting the prod image

# temporary shell image with the same binary
docker build --target build -t day68:build .
docker run --rm -it day68:build bash
# or copy binary out and run under local delve — keep prod tag clean

Lab

Suggested workspace: ~/lab/90daysofx/01-go/day68 (or containerize Stage VI service).

Multi-stage Dockerfile; runtime distroless or scratch (+ certs if needed).
CGO_ENABLED=0 build; non-root user.
Prove /healthz (or equivalent) via docker run + curl.
Note image size in CONTAINER.md (also note single-stage comparison if you build one once).
Run once with -m memory limit and GOMEMLIMIT set.
Add .dockerignore excluding _book, .git, bin artifacts.
Confirm docker stop sends SIGTERM to the binary (process exits; no hung shell).

`CONTAINER.md` template

# Container notes — day68
Image: day68:local
Base: distroless/static-debian12:nonroot
Size:
CGO: 0
User: nonroot
Port: 8080
Health: GET /healthz
Memory test: -m 256m GOMEMLIMIT=200MiB → result:
Signal: docker stop behavior:

Stretch

Multi-arch build: docker buildx build --platform linux/amd64,linux/arm64.
SBOM note: docker sbom / generate cyclonedx later—awareness only.
CI job that builds and smoke-tests /healthz.

Common gotchas

Gotcha	Fix
Dynamic link against glibc then scratch	`CGO_ENABLED=0` or use correct base
TLS fails in scratch	Install CA bundle
Shell needed for debug	Ephemeral debug image; keep prod distroless
Running as root	`USER nonroot`
Signal not reaching app	Exec-form ENTRYPOINT binary as PID 1
Huge context upload	`.dockerignore`
`go.sum` missing	Commit it; `go mod download` fails cleanly otherwise
Listening on `:80` as nonroot	Use `8080`+; map ports at the edge
Timezone panics	UTC or ship zoneinfo
Assuming macOS binary runs in Linux container	Build with `GOOS=linux` in Dockerfile

Checkpoint

Multi-stage Dockerfile committed
Distroless or scratch runtime
Non-root process
Health endpoint works in container
Size + memory-limit notes written
.dockerignore present
SIGTERM path understood for Day 78

Commit

git add .
git commit -m "day68: multi-stage distroless container"

Write three personal gotchas before continuing.

Tomorrow

Day 69 — Prometheus metrics: export RED/USE-style metrics from the process so load tests and production share a language.

Day 68 — Containers & distroless

Why this day exists

Theory 1 — Why Go loves multi-stage builds

What multi-stage actually removes

Theory 2 — Certificates and timezones

Quick verification after run

Theory 3 — Non-root, read-only, signals

Read-only root filesystem (awareness)

Theory 4 — Build cache and reproducibility

Build flags that matter

Memory alignment with Day 67

Theory 5 — .dockerignore and context size

Comparing single-stage vs multi-stage

Theory 6 — Health, ports, and compose sketch

Worked example — minimal HTTP service

Debug without polluting the prod image

Lab

CONTAINER.md template

Stretch

Common gotchas

Checkpoint

Commit

Tomorrow

Theory 5 — `.dockerignore` and context size

`CONTAINER.md` template