Day 40 — bufio, bytes & strings

Day 40 — bufio, bytes & strings

Stage V · ~3h
Goal: Parse streams efficiently with bufio, manage byte slices with bytes, and build strings without quadratic concat—then ship a small record parser with tests.

Why this day exists

Raw io.Reader gives you arbitrary chunks. Real formats need lines, tokens, and delimited records. Doing that with tiny Read loops is error-prone and slow; loading everything into memory reintroduces Day 39’s failure mode.

bufio, bytes, and strings are the stdlib trio for text-ish I/O at the right level.

---

Theory 1 — Why buffer?

Each Read on a file or socket can be a syscall. bufio.Reader amortizes:

br := bufio.NewReader(r)   // default 4096-byte buffer
line, err := br.ReadString('\n')

Type	Role
bufio.Reader	Buffered reads; Peek, ReadByte, ReadSlice, ReadString
bufio.Writer	Buffered writes; must Flush
bufio.Scanner	Tokenize with split funcs (lines, words, custom)
bufio.ReadWriter	Both sides

Scanner vs manual ReadString

Prefer Scanner for line/word protocols unless you need partial control or binary data:

sc := bufio.NewScanner(r)
// Optional: raise token size for long lines (default ~64KiB)
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for sc.Scan() {
    line := sc.Text() // or sc.Bytes() — careful: Bytes is overwritten next Scan
    _ = line
}
if err := sc.Err(); err != nil {
    // handle scan error (not EOF)
}

Warning

sc.Bytes() returns a slice owned by the scanner. Copy it (append([]byte(nil), sc.Bytes()...)) if you retain it past the next Scan.

---

Theory 2 — Split functions

sc.Split(bufio.ScanLines)  // default
sc.Split(bufio.ScanWords)
sc.Split(bufio.ScanRunes)
sc.Split(bufio.ScanBytes)

Custom split for comma-separated records (simplified):

func scanComma(data []byte, atEOF bool) (advance int, token []byte, err error) {
    if i := bytes.IndexByte(data, ','); i >= 0 {
        return i + 1, data[:i], nil
    }
    if atEOF && len(data) > 0 {
        return len(data), data, nil
    }
    return 0, nil, nil // need more data
}

---

Theory 3 — bytes package

bytes mirrors many strings functions for []byte without converting to string (and back).

Need	API
Search	Contains, Index, HasPrefix, Count
Cut	Cut, CutPrefix, CutSuffix (modern, clear)
Fields	Fields, Split, SplitN
Trim	TrimSpace, TrimPrefix
Compare	Equal, Compare
Buffer	bytes.Buffer — grows, implements io.Reader/Writer

bytes.Buffer

var buf bytes.Buffer
buf.WriteString("hello")
buf.WriteByte(' ')
fmt.Fprintf(&buf, "day %d", 40)
s := buf.String() // copies; or buf.Bytes() (valid until next mutate)

Use Buffer when you need an in-memory Reader/Writer. For pure string assembly, prefer strings.Builder.

---

Theory 4 — strings package & Builder

var b strings.Builder
b.Grow(64) // optional hint
b.WriteString("id=")
b.WriteString(id)
out := b.String()

Never build large strings with s = s + piece in a loop—quadratic copies.

Useful modern helpers

before, after, ok := strings.Cut(line, "=")
prefix, ok := strings.CutPrefix(s, "Bearer ")

Cut is clearer than SplitN(..., 2) for “key/value once.”

---

Theory 5 — String vs []byte cost

• Conversion string(bs) / []byte(s) copies data.
  
• Hot parsers often stay in []byte until a final string is needed.
  
• Scanner.Text() allocates a string; Bytes() does not (but is ephemeral).

For Day 40 labs, clarity beats micro-opts—but know the cost model.

---

Worked example — NDJSON-ish record parser

Format (one record per line):

name=ada role=eng active=true
name=alan role=research active=false

package parse

import (
    "bufio"
    "fmt"
    "io"
    "strings"
)

type Record struct {
    Name   string
    Role   string
    Active bool
}

func ParseRecords(r io.Reader) ([]Record, error) {
    sc := bufio.NewScanner(r)
    var out []Record
    lineNo := 0
    for sc.Scan() {
        lineNo++
        line := strings.TrimSpace(sc.Text())
        if line == "" || strings.HasPrefix(line, "#") {
            continue
        }
        rec, err := parseLine(line)
        if err != nil {
            return nil, fmt.Errorf("line %d: %w", lineNo, err)
        }
        out = append(out, rec)
    }
    return out, sc.Err()
}

func parseLine(line string) (Record, error) {
    var rec Record
    fields := strings.Fields(line)
    for _, f := range fields {
        key, val, ok := strings.Cut(f, "=")
        if !ok {
            return rec, fmt.Errorf("bad field %q", f)
        }
        switch key {
        case "name":
            rec.Name = val
        case "role":
            rec.Role = val
        case "active":
            switch val {
            case "true":
                rec.Active = true
            case "false":
                rec.Active = false
            default:
                return rec, fmt.Errorf("active=%q", val)
            }
        default:
            return rec, fmt.Errorf("unknown key %q", key)
        }
    }
    if rec.Name == "" {
        return rec, fmt.Errorf("missing name")
    }
    return rec, nil
}

---

Lab 1 — Setup + golden parse

mkdir -p ~/lab/90daysofx/01-go/day40
cd ~/lab/90daysofx/01-go/day40
go mod init example.com/day40

Put parser in parse/parse.go and tests in parse/parse_test.go.

Sample input testdata/people.txt:

# staff
name=ada role=eng active=true
name=alan role=research active=false

Table-driven tests:

func TestParseRecords(t *testing.T) {
    in := "name=ada role=eng active=true\nname=x role=y active=maybe\n"
    _, err := ParseRecords(strings.NewReader(in))
    if err == nil {
        t.Fatal("expected error on bad active")
    }
}

Also test: empty lines, comments, missing name, unknown keys.

go test ./...

---

Lab 2 — Streaming CLI

go run . < people.txt
# print: ada    eng true

Use bufio.NewWriter(os.Stdout) and Flush in defer (or after loop). Compare wall time on a large generated file vs reading with os.ReadFile + strings.Split—observe memory with a big file if you wish (/usr/bin/time -l on macOS).

Generate:

python3 -c 'print("name=u role=r active=true\n"*200000)' > big.txt

---

Lab 3 — Custom Scanner split

Implement a split that yields paragraphs (blocks separated by blank lines). Print paragraph lengths. Edge cases: trailing content without final blank line; multiple blank lines.

---

Lab 4 — Builder report

From parsed records, emit a summary string with strings.Builder:

count=2 active=1

Benchmark (optional) Builder vs + concatenation for N=10_000 field appends:

go test -bench=Builder -benchmem ./...

---

Common gotchas

Gotcha	Fix
Forgetting sc.Err()	Always check after the loop
Keeping sc.Bytes()	Copy if storing
Token too long	sc.Buffer(buf, max)
Forgetting Writer.Flush	defer bw.Flush() and check error
strings.Split empty edge	Prefer Cut / Fields for clarity
Quadratic s +=	strings.Builder
Assuming Latin-1 lines	UTF-8 is fine; runes need ScanRunes or utf8

---

Checkpoint

• Explain why bufio exists in terms of syscalls
  
• Parser handles comments, blanks, and bad fields with line numbers
  
• Tests cover at least four failure modes
  
• CLI streams stdin → stdout without ReadAll of whole file
  
• Can state when to use bytes.Buffer vs strings.Builder
  
• Know Scanner’s default max token size risk

Commit

git add .
git commit -m "day40: bufio scanner parser + bytes/strings tools"

Write three personal gotchas before continuing.

---

Tomorrow

Day 41 — encoding/json: tags, omitempty, custom marshalers, streaming encode/decode, and awareness only of the json/v2 direction—without rewriting production code onto experimental APIs.