I/O and Streaming

Updated

July 30, 2026

I/O and Streaming

Overview

Go’s io package defines the small interfaces that power files, networks, compression, and crypto: Reader, Writer, Closer, and friends. Idiomatic streaming means processing data in bounded buffers instead of slurping multi-gigabyte payloads into memory.

This chapter covers the core interfaces, composition (MultiReader, TeeReader, LimitReader), bufio, pipes, practical streaming patterns, and production pitfalls.

Core Interfaces

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

type Closer interface {
    Close() error
}

type ReadCloser interface {
    Reader
    Closer
}

type WriteCloser interface {
    Writer
    Closer
}

Contracts that matter in production:

Read may return n > 0 and err == io.EOF on the final chunk—process n before handling EOF.
Write must report short writes (n < len(p)) as an error if not fully written (io.ErrShortWrite patterns).
Always Close ReadCloser values you own (HTTP bodies, files).

Reading

f, err := os.Open("file.txt")
if err != nil {
    return err
}
defer f.Close()

buf := make([]byte, 32*1024)
for {
    n, err := f.Read(buf)
    if n > 0 {
        // process buf[:n]
    }
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
}

Helpers:

data, err := io.ReadAll(f)           // entire stream into memory
chunk, err := io.ReadAll(io.LimitReader(r, 1<<20)) // cap at 1 MiB

HTTP bodies

resp, err := http.Get(url)
if err != nil {
    return err
}
defer resp.Body.Close()

const maxBody = 8 << 20 // 8 MiB
limited := io.LimitReader(resp.Body, maxBody+1)
b, err := io.ReadAll(limited)
if err != nil {
    return err
}
if len(b) > maxBody {
    return fmt.Errorf("response body exceeds %d bytes", maxBody)
}

Writing

f, err := os.Create("output.txt")
if err != nil {
    return err
}
defer f.Close()

if _, err := f.Write([]byte("Hello")); err != nil {
    return err
}
if _, err := io.WriteString(f, " World"); err != nil {
    return err
}
return f.Sync() // durability when needed

Copying Streams

// Efficient path: uses WriterTo / ReaderFrom when available
if _, err := io.Copy(dst, src); err != nil {
    return err
}

// Cap bytes
if _, err := io.Copy(dst, io.LimitReader(src, 1024)); err != nil {
    return err
}

// Fixed buffer size (Go 1.15+ has CopyBuffer)
buf := make([]byte, 64*1024)
if _, err := io.CopyBuffer(dst, src, buf); err != nil {
    return err
}

io.Copy is the default for proxying HTTP, piping files, and unpacking archives—prefer it over manual loops unless you need progress hooks.

Progress reader

type countingReader struct {
    r io.Reader
    n int64
}

func (c *countingReader) Read(p []byte) (int, error) {
    n, err := c.r.Read(p)
    c.n += int64(n)
    return n, err
}

Composition Helpers

Helper	Role
`io.MultiReader`	Concatenate readers sequentially
`io.MultiWriter`	Fan-out writes to multiple writers
`io.TeeReader`	Read while copying to a side writer (checksums, logs)
`io.LimitReader`	Hard cap bytes read
`io.SectionReader`	Window into an `ReaderAt`
`io.Pipe`	In-memory streaming between goroutines

// Checksum while uploading
h := sha256.New()
tr := io.TeeReader(file, h)
if _, err := io.Copy(dest, tr); err != nil {
    return err
}
sum := h.Sum(nil)

// Concatenate header + body
r := io.MultiReader(strings.NewReader("HDR\n"), body)

bufio

Buffering cuts syscalls for small reads/writes.

f, err := os.Open("access.log")
if err != nil {
    return err
}
defer f.Close()

sc := bufio.NewScanner(f)
// Raise token size for long lines (default ~64K)
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)

for sc.Scan() {
    line := sc.Text()
    _ = line
}
return sc.Err()

bw := bufio.NewWriterSize(f, 64*1024)
if _, err := bw.WriteString("row\n"); err != nil {
    return err
}
return bw.Flush() // never forget Flush

bytes.Buffer and strings.Reader

var buf bytes.Buffer
buf.WriteString("Hello")
buf.WriteByte(' ')
buf.Write([]byte("World"))
out := buf.String()

r := strings.NewReader("stream me")
io.Copy(os.Stdout, r)

Use bytes.Buffer for building in memory; switch to streaming files when size is unbounded.

Pipes Between Goroutines

pr, pw := io.Pipe()

go func() {
    defer pw.Close()
    // If Write fails, CloseWithError
    if _, err := io.WriteString(pw, "payload"); err != nil {
        pw.CloseWithError(err)
        return
    }
}()

data, err := io.ReadAll(pr)
if err != nil {
    return err
}
_ = data

Pipe blocks when the buffer is full—natural backpressure. Always close the writer side or readers hang forever.

Streaming HTTP Upload / Download

func upload(ctx context.Context, client *http.Client, url string, r io.Reader) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, r)
    if err != nil {
        return err
    }
    // If size known:
    // req.ContentLength = size
    res, err := client.Do(req)
    if err != nil {
        return err
    }
    defer res.Body.Close()
    if res.StatusCode >= 300 {
        b, _ := io.ReadAll(io.LimitReader(res.Body, 4096))
        return fmt.Errorf("upload: %s: %s", res.Status, b)
    }
    return nil
}

gzip Stream

func compress(dst io.Writer, src io.Reader) error {
    zw := gzip.NewWriter(dst)
    if _, err := io.Copy(zw, src); err != nil {
        _ = zw.Close()
        return err
    }
    return zw.Close()
}

func decompress(dst io.Writer, src io.Reader) error {
    zr, err := gzip.NewReader(src)
    if err != nil {
        return err
    }
    defer zr.Close()
    _, err = io.Copy(dst, zr)
    return err
}

Checklist

Bound all untrusted readers with LimitReader or max bytes
Close every ReadCloser you own; check Close errors on writers when data integrity matters
Prefer io.Copy / CopyBuffer over ad-hoc loops
Flush buffered writers before close/return
Pipe writer closed on both success and error paths
Scanner buffer sized for your longest valid line
No ReadAll on unbounded network input
Use context cancellation on HTTP and long copies (via req.Context() / wrappers)

Common Pitfalls

Ignoring n on EOF — last bytes dropped.
Forgetting Flush — data stuck in bufio.Writer.
Scanner token too long — silent bufio.ErrTooLong; raise buffer max.
Unclosed HTTP bodies — connection pool stalls.
Deadlocked pipes — writer blocked because reader never reads; always schedule both sides.
ioutil legacy — use io and os (ioutil deprecated).
Assuming Write is atomic for large buffers — loop or use helpers that handle short writes.

Exercises

Chunked SHA-256 — Stream a file with 32KiB buffers; print hex digest; compare to sha256sum.
LimitReader gate — Reject bodies over 1MiB with a clear error; test with strings.NewReader of known sizes.
TeeReader — Copy a file to another path while computing CRC32C (or SHA-256) in one pass.
Line scanner — Parse a 10MB log with long lines; set buffer max; count lines matching a prefix.
Pipe pipeline — Goroutine A writes numbers; goroutine B filters even lines; main reads result.
gzip round-trip — Compress and decompress a random 1MB payload; require byte-identical output.

More examples

TeeReader: copy + checksum one pass

mkdir -p /tmp/go-io-tee && cd /tmp/go-io-tee
go mod init example.com/io-tee

Save as main.go:

package main

import (
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "io"
    "os"
    "strings"
)

func main() {
    src := strings.NewReader("stream-me-please")
    h := sha256.New()
    dst, _ := os.CreateTemp("", "out")
    defer os.Remove(dst.Name())
    defer dst.Close()

    n, err := io.Copy(io.MultiWriter(dst, h), src)
    if err != nil {
        panic(err)
    }
    fmt.Println("bytes:", n)
    fmt.Println("sha256:", hex.EncodeToString(h.Sum(nil))[:16]+"...")
}

go run .

Expected output:

bytes: 15
sha256: <16 hex chars>...

Line scanner with buffer cap

mkdir -p /tmp/go-io-scan && cd /tmp/go-io-scan
go mod init example.com/io-scan

Save as main.go:

package main

import (
    "bufio"
    "fmt"
    "strings"
)

func main() {
    in := strings.NewReader("alpha\nbeta\ngamma-long-line\n")
    sc := bufio.NewScanner(in)
    sc.Buffer(make([]byte, 64), 64)
    var n int
    for sc.Scan() {
        if strings.HasPrefix(sc.Text(), "g") {
            n++
        }
    }
    fmt.Println("g-lines:", n, "err:", sc.Err())
}

go run .

Expected output:

g-lines: 1 err: <nil>

Runnable example

Streaming IO is chunked reads, limited readers, tee, and compressors—never ReadAll on unbounded sources. This program hashes while copying, enforces a size cap, and round-trips gzip.

mkdir -p /tmp/go-io-stream && cd /tmp/go-io-stream
go mod init example.com/io-stream

Save as main.go:

package main

import (
    "bytes"
    "compress/gzip"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "io"
    "os"
    "path/filepath"
    "strings"
)

func main() {
    dir := filepath.Join(os.TempDir(), "go-io-stream")
    _ = os.MkdirAll(dir, 0o750)
    srcPath := filepath.Join(dir, "src.bin")
    dstPath := filepath.Join(dir, "dst.bin")

    payload := bytes.Repeat([]byte("stream-me-"), 1000) // 10_000 bytes
    if err := os.WriteFile(srcPath, payload, 0o640); err != nil {
        panic(err)
    }

    // Chunked SHA-256 + copy (one pass with TeeReader).
    in, err := os.Open(srcPath)
    if err != nil {
        panic(err)
    }
    defer in.Close()
    out, err := os.Create(dstPath)
    if err != nil {
        panic(err)
    }
    h := sha256.New()
    n, err := io.Copy(out, io.TeeReader(in, h))
    out.Close()
    if err != nil {
        panic(err)
    }
    fmt.Println("copied_bytes:", n)
    fmt.Println("sha256:", hex.EncodeToString(h.Sum(nil)))

    // LimitReader gate
    limited := io.LimitReader(strings.NewReader(strings.Repeat("x", 100)), 16)
    buf, _ := io.ReadAll(limited)
    fmt.Println("limited_len:", len(buf))

    // Reject oversize: read with Max-style check
    const max = 1024
    r := strings.NewReader(strings.Repeat("y", 2000))
    data, err := io.ReadAll(io.LimitReader(r, max+1))
    if err != nil {
        panic(err)
    }
    if len(data) > max {
        fmt.Println("oversize: rejected", len(data))
    }

    // gzip round-trip
    var zbuf bytes.Buffer
    zw := gzip.NewWriter(&zbuf)
    _, _ = zw.Write(payload)
    _ = zw.Close()
    zr, err := gzip.NewReader(&zbuf)
    if err != nil {
        panic(err)
    }
    round, err := io.ReadAll(zr)
    _ = zr.Close()
    if err != nil {
        panic(err)
    }
    fmt.Println("gzip_roundtrip_ok:", bytes.Equal(round, payload))
    fmt.Println("gzip_ratio:", len(zbuf.Bytes()), "/", len(payload))
}

go run .

Expected output (hash stable for this payload):

copied_bytes: 10000
sha256: <64 hex chars>
limited_len: 16
oversize: rejected 1025
gzip_roundtrip_ok: true
gzip_ratio: <compressed> / 10000

What to notice

io.TeeReader hashes and copies without buffering the whole file twice.
io.LimitReader is the building block for body caps; check length after read or use http.MaxBytesReader on servers.
Always Close gzip writers/readers or trailers/checksums can be wrong.

Try next

Stream a file with a 32KiB buffer loop (Read into a pooled slice) and compare to io.Copy.
Build a pipe: goroutine writes lines, main bufio.Scanner counts them.

I/O and Streaming

Overview

Core Interfaces

Reading

HTTP bodies

Writing

Copying Streams

Progress reader

Composition Helpers

bufio

bytes.Buffer and strings.Reader

Pipes Between Goroutines

Streaming HTTP Upload / Download

gzip Stream

Checklist

Common Pitfalls

Exercises

More examples

TeeReader: copy + checksum one pass

Line scanner with buffer cap

Runnable example

Related Topics