File Descriptors and Redirection

File Descriptors and Redirection

Overview

Unix processes talk to the world through file descriptors (FDs): integers indexing a per-process table. FD 0/1/2 are stdin/stdout/stderr. Understanding FDs explains pipes, redirection, and why “close the file” matters for long-running services.

The classic trio

FD	Go	Role
0	os.Stdin	Input
1	os.Stdout	Data output
2	os.Stderr	Diagnostics

fmt.Fprintln(os.Stdout, "data")
fmt.Fprintln(os.Stderr, "debug")

Shell: tool >out.txt 2>err.txt remaps FDs before exec.

Opening files yields FDs

f, err := os.Open("/etc/hosts")
// f.Fd() → uintptr underlying FD (Unix)
defer f.Close() // returns FD to the kernel

Leaking FDs (Open without Close) exhausts the process limit (ulimit -n).

Inheritance across exec

By default, child processes inherit open FDs unless marked close-on-exec.

cmd := exec.Command("child")
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr

os/exec sets up pipes or inheritance for the three standard streams. Extra FDs you opened in the parent may remain open in the child if not close-on-exec—usually undesirable.

f, _ := os.OpenFile("log.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
// syscall.CloseOnExec(int(f.Fd())) // low-level; prefer not sharing logs via inheritance
cmd.ExtraFiles = []*os.File{f} // intentional FD 3+ in child (Unix)

ExtraFiles maps to child FDs starting at 3—used by some protocols (e.g. systemd socket activation patterns).

Pipes as FDs

r, w, err := os.Pipe()
go func() {
    defer w.Close()
    fmt.Fprintln(w, "hello")
}()
buf := make([]byte, 64)
n, _ := r.Read(buf)
fmt.Println(string(buf[:n]))
r.Close()

io.Pipe is in-process (not OS FDs). os.Pipe is a real kernel pipe—shareable with children.

Redirection patterns in Go

Capture child stdout

var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
err := cmd.Run()

Stream child to parent stdout

cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr

Tee logs

cmd.Stdout = io.MultiWriter(os.Stdout, logFile)

Checking TTY vs pipe

fi, err := os.Stdout.Stat()
isChar := fi.Mode()&os.ModeCharDevice != 0

Color and progress bars: only when stdout is a TTY (and respect NO_COLOR).

Limits and EMFILE

too many open files → EMFILE / "too many open files"

Mitigation	How
Always Close	defer or t.Cleanup
Bound concurrency	worker pools for file openers
Raise limit carefully	container ulimit, not a substitute for leaks
Reuse readers	stream instead of open-all

/dev/null and discard

cmd.Stdout = io.Discard
cmd.Stderr = io.Discard
// or os.OpenFile(os.DevNull, os.O_WRONLY, 0)

Minimal tool: fdcount sketch

// Linux: count entries in /proc/self/fd
entries, err := os.ReadDir("/proc/self/fd")
fmt.Println(len(entries))

Use in tests to catch FD leaks (open N files, close, count should return).

Rules of thumb

Do	Don’t
Close files and pipe ends	Rely on GC finalizers for FDs
Keep stdout/stderr roles pure	Mix binary data and logs on FD 1
Use ExtraFiles only intentionally	Leak sockets into helpers
Handle EPIPE on writes	Crash on | head pipelines

Try next

1. Pipe a Go program into head -n 1 and ensure it exits cleanly on short write.
2. Spawn a child with ExtraFiles and print FD 3 in the child.
3. Open 10k files without close; observe the error; fix with a pool.