Unix Pipeline Style CLI Tools

Updated

July 30, 2026

Unix Pipeline Style CLI Tools

Good CLI tools are composable components in a data stream. Go is excellent for CLIs: single static binary, fast startup, solid bufio and encoding libraries.

Why / Overview

If your tool only works as an interactive TUI, it cannot sit in CI. If it mixes logs into stdout, it breaks jq. Pipeline culture is an API design problem.

stdin  -> transform / filter -> stdout   (data only)
                 |
                 +-> stderr  (diagnostics, progress)
exit code -> automation branch

The Pipeline Contract

  1. Read stdin when no file args (or when arg is -).
  2. Write data to stdout — nothing else.
  3. Write diagnostics to stderr.
  4. Exit non-zero on failure; document codes.
  5. Be quiet on success unless --verbose.
  6. Support non-TTY operation (no assumed terminal width unless queried).
func main() {
    if err := run(os.Args[1:]); err != nil {
        fmt.Fprintln(os.Stderr, "mytool:", err)
        os.Exit(1)
    }
}

Input Sources

func openInput(path string) (io.ReadCloser, error) {
    if path == "" || path == "-" {
        return io.NopCloser(os.Stdin), nil
    }
    return os.Open(path)
}

Multiple files: process sequentially like cat, or document parallel behavior carefully (order).

Streaming Filters

Prefer stream processing over ReadAll for unbounded input.

func filter(r io.Reader, w io.Writer, pred func(string) bool) error {
    sc := bufio.NewScanner(r)
    // Optional: sc.Buffer(make([]byte, 64*1024), 1024*1024)
    out := bufio.NewWriter(w)
    defer out.Flush()
    for sc.Scan() {
        line := sc.Text()
        if pred(line) {
            if _, err := out.WriteString(line); err != nil {
                return err
            }
            if err := out.WriteByte('\n'); err != nil {
                return err
            }
        }
    }
    return sc.Err()
}

Always check sc.Err() — a failed scan can look like EOF.

Output Modes: Human vs Machine

default (TTY):     human tables, colors optional
--json / pipe:     NDJSON or JSON array
--tsv:             stable columns for cut/awk

Detect TTY:

func stdoutIsTTY() bool {
    fi, err := os.Stdout.Stat()
    if err != nil {
        return false
    }
    return (fi.Mode() & os.ModeCharDevice) != 0
}

Rules:

  • Never emit ANSI color when not a TTY (or when NO_COLOR is set).
  • Machine formats must be stable across versions (version field if needed).
  • Do not interleave progress bars on stdout.
type row struct {
    User string `json:"user"`
    OK   bool   `json:"ok"`
}

func writeNDJSON(w io.Writer, rows []row) error {
    enc := json.NewEncoder(w)
    for _, r := range rows {
        if err := enc.Encode(r); err != nil { // Encode adds newline
            return err
        }
    }
    return nil
}

Flags and Subcommands

Use flag for simple tools; cobra/ff etc. for larger surfaces. Conventions:

  • --help works
  • Short flags where traditional (-n, -v)
  • --version
  • Options before/after args documented
fs := flag.NewFlagSet("grep-lite", flag.ExitOnError)
pattern := fs.String("e", "", "pattern")
invert := fs.Bool("v", false, "invert match")
_ = fs.Parse(args)

Exit Codes

Code Meaning
0 Success (and for grep-like: match found)
1 Generic error or grep-like: no match
2 Usage error

Document grep-like special cases; they surprise API people.

const (
    exitOK      = 0
    exitFail    = 1
    exitUsage   = 2
)

Handling SIGPIPE

When writing to a closed pipe (head closes early), writes fail with EPIPE / ErrClosed. Good tools exit quietly with SIGPIPE semantics rather than dumping a panic stack.

if _, err := os.Stdout.Write(buf); err != nil {
    if errors.Is(err, syscall.EPIPE) {
        os.Exit(0) // or 141 depending on tradition
    }
    return err
}

Context and Cancellation

Long CLIs should honor SIGINT:

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()

// pass ctx into work; on cancel, flush what is safe and exit non-zero

Structure of a Solid Small Tool

cmd/mytool/main.go     flags, exit codes
internal/app/run.go    pure-ish logic with io.Reader/Writer
internal/app/run_test.go  table tests with bytes.Buffer

Dependency inject IO for tests:

type App struct {
    In   io.Reader
    Out  io.Writer
    Err  io.Writer
}

func (a *App) Run(ctx context.Context, args []string) error {
    // ...
}

Progress and Logging

log := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo}))
log.Info("processed", "lines", n)

Never fmt.Println debug to stdout in library paths.

Composition Examples

# NDJSON pipeline
mytool --json < events.log | jq 'select(.level=="error")'

# Classic filters
cat access.log | mytool -e 'timeout' | wc -l

# Explicit files
mytool -e 'panic' app.log app.log.1

Production Checklist (CLI)

  • stdin default; - supported
  • stdout data-only
  • stderr for errors/logs
  • Documented exit codes
  • --help / usage on bad flags
  • Stream large inputs
  • Stable machine-readable mode
  • NO_COLOR / TTY-aware color
  • SIGINT cancels work
  • SIGPIPE handled
  • Tests use buffers, not real terminal

Common Pitfalls

  1. Logging to stdout — breaks pipes.
  2. Loading entire stdin into memory.
  3. Interactive prompts when stdin is a pipe (hangs CI).
  4. Unstable JSON key order / floating timestamps for golden tests — prefer normalized output.
  5. Relative paths depending on launch directory without documenting.
  6. Buffered stdout not flushed on early exit — use bufio.Writer + Flush or defer carefully.
  7. Assuming UTF-8 without documenting binary behavior.

Mini Implementation: line counter

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

func countLines(r io.Reader) (int, error) {
    sc := bufio.NewScanner(r)
    n := 0
    for sc.Scan() {
        n++
    }
    return n, sc.Err()
}

func main() {
    n, err := countLines(os.Stdin)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(n)
}

Extend with file args, -v, and JSON {"lines":N} as exercises.

Exercises

  1. Build a filter that prints lines containing a substring; support stdin and file args.
  2. Add --json NDJSON output of {line,no}; keep default plain text.
  3. Implement invert match (-v) and appropriate exit codes.
  4. Pipe through head -n 1 and ensure no ugly panic on SIGPIPE.
  5. Add slog diagnostics on stderr behind -v; prove jq still parses stdout.
  6. Refactor logic to App struct; unit-test with strings.NewReader / bytes.Buffer.
  7. Handle lines longer than 64KiB safely (configure scanner buffer).
  8. Add --fail-empty that exits 1 when zero lines matched.
  9. Benchmark counting 10M lines; compare Scanner vs raw bufio.Reader ReadBytes.
  10. Write a README section “Scripting interface” documenting flags, exit codes, and examples.

More examples

Stdin filter: prefix match

mkdir -p /tmp/go-pipe-filter && cd /tmp/go-pipe-filter
go mod init example.com/pipe-filter

Save as main.go:

package main

import (
    "bufio"
    "fmt"
    "strings"
)

func filter(in string, prefix string) []string {
    var out []string
    sc := bufio.NewScanner(strings.NewReader(in))
    for sc.Scan() {
        line := sc.Text()
        if strings.HasPrefix(line, prefix) {
            out = append(out, line)
        }
    }
    return out
}

func main() {
    in := "ERR boom\nINFO ok\nERR again\n"
    for _, line := range filter(in, "ERR") {
        fmt.Println(line)
    }
}
go run .

Expected output:

ERR boom
ERR again

Exit code when no matches (grep-like)

mkdir -p /tmp/go-pipe-exit && cd /tmp/go-pipe-exit
go mod init example.com/pipe-exit

Save as main.go:

package main

import (
    "fmt"
    "strings"
)

func matchCount(in, needle string) int {
    n := 0
    for _, line := range strings.Split(strings.TrimSuffix(in, "\n"), "\n") {
        if strings.Contains(line, needle) {
            n++
        }
    }
    return n
}

func main() {
    for _, needle := range []string{"go", "rust"} {
        n := matchCount("go tools\ngo test\n", needle)
        code := 0
        if n == 0 {
            code = 1
        }
        fmt.Printf("needle=%q matches=%d exit=%d\n", needle, n, code)
    }
}
go run .

Expected output:

needle="go" matches=2 exit=0
needle="rust" matches=0 exit=1

Runnable example

A pipeline-friendly line filter: read stdin (or an in-memory reader), match a substring, write matches to stdout, diagnostics to stderr, exit codes for scripts.

mkdir -p /tmp/go-pipeline-cli && cd /tmp/go-pipeline-cli
go mod init example.com/pipeline-cli

Save as main.go:

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
    "strings"
)

func filter(r io.Reader, w io.Writer, errW io.Writer, substr string, invert bool) (matches int, err error) {
    sc := bufio.NewScanner(r)
    // Raise token size for long log lines
    buf := make([]byte, 0, 64*1024)
    sc.Buffer(buf, 1024*1024)
    for sc.Scan() {
        line := sc.Text()
        hit := strings.Contains(line, substr)
        if invert {
            hit = !hit
        }
        if hit {
            if _, err := fmt.Fprintln(w, line); err != nil {
                return matches, err
            }
            matches++
        }
    }
    if err := sc.Err(); err != nil {
        fmt.Fprintln(errW, "scan:", err)
        return matches, err
    }
    return matches, nil
}

func main() {
    // Demo without relying on external pipes: feed a buffer, print results.
    input := "alpha\nerror: disk full\nbeta\nerror: timeout\ngamma\n"
    var out, errB strings.Builder
    n, err := filter(strings.NewReader(input), &out, &errB, "error:", false)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Fprint(os.Stdout, out.String())
    fmt.Fprintf(os.Stderr, "matches=%d\n", n)

    // Exit code convention for CLIs: 0 found, 1 not found (grep-like optional).
    if n == 0 {
        os.Exit(1)
    }
}
go run .
# real pipeline:
# printf 'a\nerror: x\nb\n' | go run .   # after wiring os.Stdin/Args

Expected output:

error: disk full
error: timeout
matches=2

(stderr line may interleave depending on buffering)

What to notice

  • Business output on stdout; counts/diagnostics on stderr so | jq keeps working.
  • Configure Scanner buffer for long lines; check sc.Err().
  • Exit codes are part of the scripting API.

Try next

  • Accept optional file args; default to stdin when none.
  • Add -v invert and --json NDJSON mode without breaking plain default.

Further Reading

  • Rob Pike / Unix philosophy notes on tools
  • Previous: safe file IO for CLIs that write state
  • Next part: Distributed Infrastructure patterns for long-running workers