io and bufio Composition

Updated

September 8, 2026

io and bufio Composition

Overview

The io package is Go’s streaming contract. Small interfaces (Reader, Writer, Closer) let files, sockets, compressors, and crypto share the same pipelines.

Deep patterns also appear in I/O and Streaming. This chapter is the stdlib composition cheatsheet.

Core contracts

type Reader interface {
    Read(p []byte) (n int, err error)
}
type Writer interface {
    Write(p []byte) (n int, err error)
}
  • Process n > 0 before checking err (including io.EOF).
  • io.EOF means end of stream, not a hard failure for Read loops.
  • Prefer io.Copy over manual loops — it uses WriterTo / ReaderFrom when available.

Helpers worth memorizing

Helper Purpose
io.Copy / CopyBuffer Stream A → B efficiently
io.ReadAll Entire stream to memory (cap it!)
io.LimitReader Hard cap bytes read
io.TeeReader Read + fork to a Writer (checksums)
io.MultiReader Concatenate readers
io.MultiWriter Fan-out writes
io.Pipe In-process streaming between goroutines
io.NopCloser Wrap Reader as ReadCloser
io.Discard Drain without storing 
const max = 1 << 20 // 1 MiB
data, err := io.ReadAll(io.LimitReader(r, max+1))
if err != nil {
    return err
}
if len(data) > max {
    return fmt.Errorf("payload exceeds %d bytes", max)
}

bufio: amortize syscalls

br := bufio.NewReader(f)
line, err := br.ReadString('\n')

bw := bufio.NewWriter(f)
fmt.Fprintln(bw, "hello")
if err := bw.Flush(); err != nil {
    return err
}
Type Role
bufio.Reader Peek, ReadBytes, ReadString
bufio.Writer Buffered writes; always Flush
bufio.Scanner Tokenize lines/words (watch token size) 
sc := bufio.NewScanner(f)
// optional: sc.Buffer(buf, 1024*1024) for long lines
for sc.Scan() {
    line := sc.Text()
    _ = line
}
return sc.Err()

Default Scanner max token is 64KiB — raise it for pathological log lines or reject them.

Composition examples

Checksum while uploading

h := sha256.New()
tr := io.TeeReader(src, h)
if _, err := io.Copy(dst, tr); err != nil {
    return err
}
sum := h.Sum(nil)

Bound + discard remainder

// Read at most n bytes; ensure caller can still Close the body
lr := io.LimitReader(resp.Body, n)
_, err := io.Copy(dst, lr)

Pitfalls

  1. Forgetting Close on resp.Body / files → connection or FD leaks.
  2. Unbounded ReadAll on user input → memory exhaustion.
  3. Nested bufio.Reader without understanding buffered leftovers.
  4. Ignoring short writes from custom Writers.

Runnable example

go mod init example
go run .
package main

import (
    "bufio"
    "bytes"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "io"
    "strings"
)

func main() {
    src := strings.NewReader("hello stream\nsecond line\n")

    // LimitReader + ReadAll
    limited, _ := io.ReadAll(io.LimitReader(src, 5))
    fmt.Printf("limited=%q\n", limited)

    // reset-like: new reader
    src = strings.NewReader("hello stream\nsecond line\n")
    h := sha256.New()
    var out bytes.Buffer
    tr := io.TeeReader(src, h)
    if _, err := io.Copy(&out, tr); err != nil {
        panic(err)
    }
    fmt.Printf("copied=%q\n", out.Bytes())
    fmt.Println("sha256", hex.EncodeToString(h.Sum(nil))[:16]+"...")

    // Scanner lines
    sc := bufio.NewScanner(strings.NewReader("a\nb\nc\n"))
    for sc.Scan() {
        fmt.Println("line:", sc.Text())
    }
    fmt.Println("scan err:", sc.Err())

    // MultiWriter
    var a, b bytes.Buffer
    mw := io.MultiWriter(&a, &b)
    io.WriteString(mw, "x")
    fmt.Printf("a=%q b=%q\n", a.String(), b.String())
}

Expected output:

limited="hello"
copied="hello stream\nsecond line\n"
sha256 f74a77476883f531...
line: a
line: b
line: c
scan err: <nil>
a="x" b="x"

What to notice: - LimitReader stops early without error when the cap is hit mid-stream. - TeeReader is the clean way to hash or log while still delivering bytes. - Scanner is for tokens; io.Copy is for bulk.

Try next: Use io.Pipe with two goroutines: one writes lines, one counts them with bufio.Scanner.