Systems Programming Overview

Updated

July 30, 2026

Systems Programming Overview

This part covers how Go programs interact with the operating system: processes, signals, files, and stream-oriented command design. The emphasis is operational correctness — tools that behave predictably in production shells, CI, containers, and long-running supervisors.

Why Systems Programming in a Go Book?

Go is often introduced as a cloud/API language. In practice, the same binary may:

  • Run as a PID 1-ish service under systemd or Kubernetes
  • Manage child processes (plugins, helpers, compilers)
  • Rewrite thousands of config files safely
  • Sit in a Unix pipeline next to grep and jq

If your HTTP handlers are solid but your process mishandles SIGTERM, you still page at 3 a.m.

                    +------------------+
   signals -------> |   Go process     | ----> child processes
   stdin/stdout --> |  main + runtime  | ----> filesystem
   env/config ----> |                  | ----> network (other parts)
                    +------------------+

Learning Path

Chapter Focus Outcome
141 Processes & signals Lifecycle, graceful stop, supervisors Clean shutdown and restart policy
142 Safe filesystem IO Atomic writes, path jails, fsync tradeoffs Automation that does not corrupt state
143 Unix pipeline CLIs stdin/stdout/stderr contracts, exit codes Tools that compose in shell and CI

Core Concepts Map

Process as a state machine

starting -> running -> stopping -> exited
                |          ^
                +-> crash -+  (restart policy lives here)

Every production process needs explicit answers for: how do I start, how do I stop, what is my exit code vocabulary, and who restarts me?

The filesystem is not a database (but you can still be careful)

Partial writes and torn updates are normal under crash. Atomic rename patterns and directory fsync exist because people lost data.

CLI tools are network services for humans and scripts

Stable stdout, noisy stderr, and meaningful exit codes are the “API” of a CLI. Breaking them breaks automation.

Go Building Blocks

Area Packages
Process / exec os/exec, os, syscall (sparingly)
Signals os/signal, signal.NotifyContext
Files os, io, io/fs, path/filepath, embed
Permissions os.FileMode, platform differences
Structured ops logs log/slog on stderr for CLIs

Modern signal handling skeleton

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

// pass ctx to servers, workers, exec.CommandContext
<-ctx.Done()
// drain with a second hard-deadline context

Prefer signal.NotifyContext over manual channel plumbing in new code (Go 1.16+).

Production Mindset

Ask for every systems feature:

  1. What happens on SIGTERM? (deadline? drain? exit code?)
  2. What happens if power fails mid-write?
  3. Can a malicious path escape the working root?
  4. Can this tool run non-interactively in CI?
  5. What do metrics/logs say when a child exits 137 (OOM)?

Relationship to Other Parts

  • Network systems (13) — servers need the shutdown patterns from 141.
  • Distributed infra (15) — workers are processes with queues and backpressure.
  • Security (11, 17) — path safety and least privilege are hardening.
  • Projects (99) — many Linux-clone labs exercise these chapters.

How to Study

  1. Implement a long-running worker that exits cleanly on SIGTERM within N seconds.
  2. Write a config rewriter that survives kill -9 mid-write (atomic replace).
  3. Build a filter CLI used as cat data.log | yourtool | jq.

Literacy Checklist

  • Explain SIGTERM vs SIGKILL
  • Use CommandContext so children die with parent cancel
  • Perform write-temp-fsync-rename safely
  • Jail user paths under a root directory
  • Design stdout/stderr/exit-code contracts for scripting
  • Avoid shell injection via sh -c with untrusted input

Part Warm-Up Exercises

  1. On a sample service, document current signal handling (or lack of it).
  2. Trace what Kubernetes sends on pod termination and the terminationGracePeriodSeconds interaction.
  3. Find one script in your life that greps program stdout for errors — that program violated the stderr rule.
  4. List three files your systems write that would hurt if truncated mid-write.
  5. Compare os.Exit in library code vs returning errors to main.

Platform Notes (Unix-first)

These chapters assume a Unix-like environment (Linux, macOS, BSD) because that is where most Go servers and CLIs run in production. Windows supports many of the same Go APIs (os/exec, signals subset, paths) but:

  • Signal semantics differ (no true SIGTERM from all supervisors)
  • Atomic rename and file locking differ
  • Path separators and reserved names differ

When writing cross-platform tools, isolate OS-specific code behind small files (path_unix.go / path_windows.go) and test both in CI.

Container and CI Reality

Environment Systems concern
Kubernetes pod SIGTERM + grace period; probes; non-root
systemd unit Type=notify optional; restart policy
GitHub Actions non-interactive; no TTY; pipe stdout carefully
Developer laptop Ctrl-C = SIGINT; human-readable defaults OK

Design for the non-interactive case first; make interactive polish optional.

Mini Project Thread

Across the three content chapters, you can build one coherent tool:

  1. 141 — long-running worker with graceful shutdown
  2. 142 — atomically persists checkpoints
  3. 143 — also usable as a filter in a pipeline

That combination is the skeleton of many internal ops utilities.

Failure Stories Worth Remembering

  • Config rewrite crashed mid-write → empty file → mass restart loop
  • CLI printed logs to stdout → jq parse failures in CI
  • Parent killed without waiting → zombie children filled PID table
  • Command with sh -c and user input → shell injection

Each chapter exists to prevent one of these classes of failure.

Checklist Before Shipping a Systems Tool

  • main returns errors; os.Exit only at the top
  • SIGTERM/SIGINT cancel a root context
  • Children use CommandContext or explicit term/kill
  • Durable state uses atomic write patterns
  • User paths cannot escape the data root
  • stdout is data; stderr is diagnostics
  • Exit codes documented for automation
  • Works with stdin piped (no TTY required)
  • Logs never include secrets from env files

Sample Directory Layout

cmd/mytool/main.go
internal/app/run.go
internal/app/run_test.go
internal/fsutil/atomic.go
internal/fsutil/path.go

Keep OS edge cases in fsutil; keep business logic testable with io.Reader/Writer.

How This Part Connects to Network Services

Your HTTP server is a process:

  • Chapter 141 shutdown patterns plug into http.Server.Shutdown
  • Chapter 142 patterns protect TLS key material drops and config reloads
  • Chapter 143 discipline applies to kubectl-style companion CLIs you ship beside the service

Do not treat “systems programming” as only embedded or C territory — production Go is systems software.

Glossary

  • Graceful shutdown — stop accepting; drain; then exit.
  • Atomic replace — rename-based file update without torn reads.
  • Path jail — force paths under a root directory.
  • Exit code — process-level API for scripts and supervisors.
  • SIGPIPE — write to closed pipe; common in | head pipelines.
  • PID 1 — init process in a container; reaps children / signal quirks.
  • TOCTOU — time-of-check to time-of-use race on files.

More examples

Tour: signal context + atomic temp file

mkdir -p /tmp/go-sys-tour && cd /tmp/go-sys-tour
go mod init example.com/sys-tour

Save as main.go:

package main

import (
    "context"
    "fmt"
    "os"
    "os/signal"
    "path/filepath"
    "syscall"
    "time"
)

func atomicWrite(path, body string) error {
    dir := filepath.Dir(path)
    f, err := os.CreateTemp(dir, ".tmp-*")
    if err != nil {
        return err
    }
    tmp := f.Name()
    if _, err := f.WriteString(body); err != nil {
        f.Close()
        os.Remove(tmp)
        return err
    }
    if err := f.Close(); err != nil {
        os.Remove(tmp)
        return err
    }
    return os.Rename(tmp, path)
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()
    go func() {
        time.Sleep(20 * time.Millisecond)
        stop()
    }()
    <-ctx.Done()
    fmt.Println("signal path:", ctx.Err())

    dir, _ := os.MkdirTemp("", "sys")
    defer os.RemoveAll(dir)
    path := filepath.Join(dir, "cfg.json")
    if err := atomicWrite(path, `{"ok":true}`); err != nil {
        panic(err)
    }
    b, _ := os.ReadFile(path)
    fmt.Println("file:", string(b))
}
go run .

Expected output:

signal path: context canceled
file: {"ok":true}

Runnable example

Tour of this part: print process identity, catch a simulated shutdown via context, and write a line to stdout with a non-zero exit path—the process contract network services and CLIs share.

mkdir -p /tmp/go-sys-tour && cd /tmp/go-sys-tour
go mod init example.com/sys-tour

Save as main.go:

package main

import (
    "context"
    "fmt"
    "os"
    "time"
)

func run(ctx context.Context) error {
    fmt.Fprintf(os.Stdout, "pid=%d\n", os.Getpid())
    select {
    case <-ctx.Done():
        fmt.Fprintln(os.Stderr, "shutting down:", ctx.Err())
        return nil
    case <-time.After(50 * time.Millisecond):
        fmt.Fprintln(os.Stdout, "work done")
        return nil
    }
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
    defer cancel()
    if err := run(ctx); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Fprintln(os.Stdout, "tour: process + context + exit codes")
}
go run .

Expected output:

pid=<n>
work done
tour: process + context + exit codes

What to notice

  • Stdout is for data/results; stderr for diagnostics—pipeline-friendly tools depend on that split.
  • Context cancellation is the in-process form of SIGTERM handling (chapter 141).

Try next

  • Wire signal.NotifyContext and send Ctrl-C.
  • Continue with atomic file writes (142) and stdin filters (143).

Next Chapter

Processes, Signals, and Supervisors — the lifecycle model everything else hangs on.