bufio Scanner and Readers Advanced

Updated

September 8, 2026

bufio Scanner and Readers Advanced

Overview

bufio sits between raw os.File/net.Conn and high-level logic. Master Scanner token sizes, Reader peek, and Writer flush.

Scanner limits

Default max token ~64K. Long lines fail with bufio.ErrTooLong.

sc := bufio.NewScanner(r)
buf := make([]byte, 0, 1024*1024)
sc.Buffer(buf, 10*1024*1024) // max 10MiB tokens
sc.Split(bufio.ScanLines)
for sc.Scan() {
    _ = sc.Text()
}
return sc.Err()

Custom split

// split on NUL
sc.Split(func(data []byte, atEOF bool) (advance int, token []byte, err error) {
    if i := bytes.IndexByte(data, 0); i >= 0 {
        return i + 1, data[:i], nil
    }
    if atEOF && len(data) > 0 {
        return len(data), data, nil
    }
    return 0, nil, nil
})

Reader Peek / Discard

br := bufio.NewReader(r)
b, _ := br.Peek(4) // next bytes without consume
_, _ = br.Discard(4)

Writer

bw := bufio.NewWriter(w)
fmt.Fprintln(bw, "line")
_ = bw.Flush() // required before close for full delivery

Rules

Do Don’t
Raise buffer for long lines Ignore Scan errors
Flush writers Assume OS sees buffered data
Prefer Scanner for lines ReadAll multi-GB logs

Try next

  1. File with 1MB line; fix with Buffer.
  2. Peek magic bytes for file type.
  3. Benchmark Scanner vs ReadString.