Systems Programming Overview

Systems Programming Overview

This part covers how Go programs interact with the operating system: processes, signals, files, and stream-oriented command design. The emphasis is operational correctness — tools that behave predictably in production shells, CI, containers, and long-running supervisors.

Start with Go as a systems language (chapter 140a) if you want the why: on Linux, a CGO_ENABLED=0 binary talks to the kernel itself and does not need libc. That is the property that makes the rest of this part deployable as one file.

Why Systems Programming in a Go Book?

Go is often introduced as a cloud/API language. It is also a userspace systems language: the runtime issues Linux syscalls directly, so Docker, Kubernetes, and most of the tools in this part ship as a single static binary. In practice, that same binary may:

• Run as a PID 1-ish service under systemd or Kubernetes
• Manage child processes (plugins, helpers, compilers)
• Rewrite thousands of config files safely
• Sit in a Unix pipeline next to grep and jq

If your HTTP handlers are solid but your process mishandles SIGTERM, you still page at 3 a.m.

                    +------------------+
   signals -------> |   Go process     | ----> child processes
   stdin/stdout --> |  main + runtime  | ----> filesystem
   env/config ----> |                  | ----> network (other parts)
                    +------------------+

Learning Path

Chapter	Focus	Outcome
140a Go as a systems language	No libc on Linux, CGO_ENABLED=0, Go vs C	Why a Go binary runs on Alpine, scratch, and NixOS
141 Processes & signals	Lifecycle, graceful stop, supervisors	Clean shutdown and restart policy
142 Safe filesystem IO	Atomic writes, path jails, fsync tradeoffs	Automation that does not corrupt state
143 Unix pipeline CLIs	stdin/stdout/stderr contracts, exit codes	Tools that compose in shell and CI
144 Env, user, CWD, umask	Identity and path resolution	Correct behavior under systemd/CI
145 File descriptors	FDs, pipes, inheritance, limits	No leaks; correct redirection
146 Memory-mapped files	mmap RO/RW tradeoffs	Random access without ReadAll
147 Unix domain sockets	Local IPC, HTTP over UDS	Sidecars without TCP ports
148 Locks & pidfiles	flock, single-instance tools	Safe cron/agents
149 Project: systool	Multi-command mini-tools	Capstone practice
150 Time & timers	Wall vs monotonic, ticker hygiene	Correct deadlines
151 Filesystem watching	Poll/fsnotify, debounce, reload	Hot config without races
152 Pseudo-terminals	PTY vs pipes, interactive wrap	Automate tty-only tools carefully
153 Resource limits	rlimit, GOMEMLIMIT, caps	Survive cgroup ceilings
154 Privileges	Non-root, capabilities, modes	Least privilege defaults
155 Ops logging	stderr/journal/syslog	Findable production logs
156 Project: sysops	deadline, reload, cronish, …	Ops toolkit capstone
157 Socket activation	systemd LISTEN_FDS	Privilege-separated bind
158 Linux procfs	/proc/self, cgroup peeks	Debug introspection
159 Project: minisup	multi-process supervisor	Restart + signal fanout

Core Concepts Map

Process as a state machine

starting -> running -> stopping -> exited
                |          ^
                +-> crash -+  (restart policy lives here)

Every production process needs explicit answers for: how do I start, how do I stop, what is my exit code vocabulary, and who restarts me?

The filesystem is not a database (but you can still be careful)

Partial writes and torn updates are normal under crash. Atomic rename patterns and directory fsync exist because people lost data.

CLI tools are network services for humans and scripts

Stable stdout, noisy stderr, and meaningful exit codes are the “API” of a CLI. Breaking them breaks automation.

Go Building Blocks

Area	Packages
Process / exec	os/exec, os, syscall (sparingly)
Signals	os/signal, signal.NotifyContext
Files	os, io, io/fs, path/filepath, embed
Env / identity	os.Getenv, os/user, os.UserHomeDir
FDs / pipes	os.Pipe, os.File.Fd, cmd.ExtraFiles
Local IPC	net.Listen("unix", …), unixgram
Locks / mmap	golang.org/x/sys/unix (Flock, Mmap)
Time	time, context deadlines
Watch / reload	poll Stat, optional fsnotify
PTY	creack/pty (when needed)
Limits	RLIMIT_*, GOMEMLIMIT, GOMAXPROCS
Permissions	os.FileMode, platform differences
Structured ops logs	log/slog on stderr; optional log/syslog

Suggested schedule

Day 0      140a                why Go binaries need no libc (Linux)
Days 1–2   141 → 142 → 143     core lifecycle + files + pipelines
Day 3      144 → 145           env/cwd + file descriptors
Day 4      146 → 147 → 148     mmap, UDS, locks
Day 5      149                 build systool
Day 6      150 → 151 → 152     time, watch, PTY
Day 7      153 → 154 → 155     limits, privilege, logging
Day 8      156                 build sysops toolkit
Day 9      157 → 158 → 159     activation, procfs, minisup

Modern signal handling skeleton

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

// pass ctx to servers, workers, exec.CommandContext
<-ctx.Done()
// drain with a second hard-deadline context

Prefer signal.NotifyContext over manual channel plumbing in new code (Go 1.16+).

Production Mindset

Ask for every systems feature:

1. What happens on SIGTERM? (deadline? drain? exit code?)
2. What happens if power fails mid-write?
3. Can a malicious path escape the working root?
4. Can this tool run non-interactively in CI?
5. What do metrics/logs say when a child exits 137 (OOM)?

Relationship to Other Parts

• Network systems (13) — servers need the shutdown patterns from 141.
• Distributed infra (15) — workers are processes with queues and backpressure.
• Security (11, 17) — path safety and least privilege are hardening.
• Projects (99) — many Linux-clone labs exercise these chapters.

How to Study

1. Implement a long-running worker that exits cleanly on SIGTERM within N seconds.
2. Write a config rewriter that survives kill -9 mid-write (atomic replace).
3. Build a filter CLI used as cat data.log | yourtool | jq.

Literacy Checklist

• Explain why a Linux Go binary can omit libc, and when cgo puts it back
• Explain SIGTERM vs SIGKILL
• Use CommandContext so children die with parent cancel
• Perform write-temp-fsync-rename safely
• Jail user paths under a root directory
• Design stdout/stderr/exit-code contracts for scripting
• Avoid shell injection via sh -c with untrusted input
• Reason about CWD vs binary-relative paths under systemd
• Avoid FD leaks; know stdin/stdout/stderr roles
• When to mmap vs stream reads
• Serve or dial a Unix domain socket safely (permissions)
• Single-instance exclusion with flock (not pidfile alone)
• Use monotonic-friendly timing and cancelable timers
• Hot-reload config with debounce and last-good retention
• Know when a PTY is required vs a plain pipe
• Name the limit behind EMFILE / OOMKill
• Prefer non-root + clear EPERM errors
• Log to stderr/journal with structure; no secrets

Part Warm-Up Exercises

1. On a sample service, document current signal handling (or lack of it).
2. Trace what Kubernetes sends on pod termination and the terminationGracePeriodSeconds interaction.
3. Find one script in your life that greps program stdout for errors — that program violated the stderr rule.
4. List three files your systems write that would hurt if truncated mid-write.
5. Compare os.Exit in library code vs returning errors to main.

Platform Notes (Unix-first)

These chapters assume a Unix-like environment (Linux, macOS, BSD) because that is where most Go servers and CLIs run in production. Windows supports many of the same Go APIs (os/exec, signals subset, paths) but:

• Signal semantics differ (no true SIGTERM from all supervisors)
• Atomic rename and file locking differ
• Path separators and reserved names differ

When writing cross-platform tools, isolate OS-specific code behind small files (path_unix.go / path_windows.go) and test both in CI.

Container and CI Reality

Environment	Systems concern
Kubernetes pod	SIGTERM + grace period; probes; non-root
systemd unit	Type=notify optional; restart policy
GitHub Actions	non-interactive; no TTY; pipe stdout carefully
Developer laptop	Ctrl-C = SIGINT; human-readable defaults OK

Design for the non-interactive case first; make interactive polish optional.

Mini Project Thread

Build one coherent ops utility across chapters:

1. 141 — long-running worker with graceful shutdown
   
2. 142 — atomically persists checkpoints
   
3. 143 — also usable as a filter in a pipeline
   
4. 144–145 — correct CWD/env and FD hygiene
   
5. 147–148 — optional control socket + single-instance lock
   
6. 149 — assemble as multi-command systool
   
7. 150–155 — deadlines, hot reload, limits, least privilege, logging
   
8. 156 — assemble sysops (deadline, reload, cronish, healthsock)

That combination is the skeleton of many internal ops utilities.

Failure Stories Worth Remembering

• Config rewrite crashed mid-write → empty file → mass restart loop
• CLI printed logs to stdout → jq parse failures in CI
• Parent killed without waiting → zombie children filled PID table
• Command with sh -c and user input → shell injection

Each chapter exists to prevent one of these classes of failure.

Checklist Before Shipping a Systems Tool

• main returns errors; os.Exit only at the top
• SIGTERM/SIGINT cancel a root context
• Children use CommandContext or explicit term/kill
• Durable state uses atomic write patterns
• User paths cannot escape the data root
• stdout is data; stderr is diagnostics
• Exit codes documented for automation
• Works with stdin piped (no TTY required)
• Logs never include secrets from env files

Sample Directory Layout

cmd/mytool/main.go
internal/app/run.go
internal/app/run_test.go
internal/fsutil/atomic.go
internal/fsutil/path.go

Keep OS edge cases in fsutil; keep business logic testable with io.Reader/Writer.

How This Part Connects to Network Services

Your HTTP server is a process:

• Chapter 141 shutdown patterns plug into http.Server.Shutdown
• Chapter 142 patterns protect TLS key material drops and config reloads
• Chapter 143 discipline applies to kubectl-style companion CLIs you ship beside the service

Do not treat “systems programming” as only embedded or C territory — production Go is systems software. Chapter 140a spells out where Go replaces C (portable userspace) and where it does not (kernel, hard real-time).

Glossary

• Graceful shutdown — stop accepting; drain; then exit.
• Atomic replace — rename-based file update without torn reads.
• Path jail — force paths under a root directory.
• Exit code — process-level API for scripts and supervisors.
• SIGPIPE — write to closed pipe; common in | head pipelines.
• PID 1 — init process in a container; reaps children / signal quirks.
• TOCTOU — time-of-check to time-of-use race on files.

More examples

Tour: signal context + atomic temp file

mkdir -p /tmp/go-sys-tour && cd /tmp/go-sys-tour
go mod init example.com/sys-tour

Save as main.go:

package main

import (
    "context"
    "fmt"
    "os"
    "os/signal"
    "path/filepath"
    "syscall"
    "time"
)

func atomicWrite(path, body string) error {
    dir := filepath.Dir(path)
    f, err := os.CreateTemp(dir, ".tmp-*")
    if err != nil {
        return err
    }
    tmp := f.Name()
    if _, err := f.WriteString(body); err != nil {
        f.Close()
        os.Remove(tmp)
        return err
    }
    if err := f.Close(); err != nil {
        os.Remove(tmp)
        return err
    }
    return os.Rename(tmp, path)
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()
    go func() {
        time.Sleep(20 * time.Millisecond)
        stop()
    }()
    <-ctx.Done()
    fmt.Println("signal path:", ctx.Err())

    dir, _ := os.MkdirTemp("", "sys")
    defer os.RemoveAll(dir)
    path := filepath.Join(dir, "cfg.json")
    if err := atomicWrite(path, `{"ok":true}`); err != nil {
        panic(err)
    }
    b, _ := os.ReadFile(path)
    fmt.Println("file:", string(b))
}

go run .

Expected output:

signal path: context canceled
file: {"ok":true}

Runnable example

Tour of this part: print process identity, catch a simulated shutdown via context, and write a line to stdout with a non-zero exit path—the process contract network services and CLIs share.

mkdir -p /tmp/go-sys-tour && cd /tmp/go-sys-tour
go mod init example.com/sys-tour

Save as main.go:

package main

import (
    "context"
    "fmt"
    "os"
    "time"
)

func run(ctx context.Context) error {
    fmt.Fprintf(os.Stdout, "pid=%d\n", os.Getpid())
    select {
    case <-ctx.Done():
        fmt.Fprintln(os.Stderr, "shutting down:", ctx.Err())
        return nil
    case <-time.After(50 * time.Millisecond):
        fmt.Fprintln(os.Stdout, "work done")
        return nil
    }
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
    defer cancel()
    if err := run(ctx); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Fprintln(os.Stdout, "tour: process + context + exit codes")
}

go run .

Expected output:

pid=<n>
work done
tour: process + context + exit codes

What to notice

• Stdout is for data/results; stderr for diagnostics—pipeline-friendly tools depend on that split.
• Context cancellation is the in-process form of SIGTERM handling (chapter 141).

Try next

• Wire signal.NotifyContext and send Ctrl-C.
• Continue with atomic file writes (142) and stdin filters (143).

Related in this book

• How Go reaches the kernel / static binaries: 140a Go as a Systems Language
• Low-level syscalls / eBPF / Delve: 92 Systems Programming (specialized)
  
• CLI curriculum (flags, Cobra, net tools): 23 CLI Tools
  
• Runtime deep dives: 20 Go Deep Dives

Next Chapter

Processes, Signals, and Supervisors — the lifecycle model everything else hangs on.