Chapter 9: Protocols: From Text to Binary to Custom Designs

Updated

October 31, 2026

Chapter 9: Protocols: From Text to Binary to Custom Designs

TCP provides a stream of bytes without message boundaries. If a client writes two separate 50-byte strings, the server might read all 100 bytes in a single call, or receive 20 bytes, followed by 30 bytes, followed by 50 bytes.

To transform an unstructured byte stream into discrete business messages, you must design an Application Protocol.

In this chapter, you will learn the mechanics of line-based text protocols, master length-prefixed framing, serialize binary headers using encoding/binary, evaluate protocol design principles, and build a custom high-performance RPC protocol from scratch.

Line-Based Text Protocols

The simplest way to delineate messages is using a delimiter character, most commonly a newline (\n or \r\n). Protocols like HTTP/1.1, Redis (RESP), SMTP, and POP3 use this strategy.

Message 1: "PING\n"
Message 2: "SET user:1 Alice\n"

Implementing Line-Based Framing with `bufio.Scanner`

Go’s bufio.Scanner splits incoming streams on newlines by default:

func handleLineProtocol(conn net.Conn) {
    defer conn.Close()
    scanner := bufio.NewScanner(conn)

    // You can increase the maximum allowed line buffer if necessary
    // const maxLineSize = 64 * 1024
    // buf := make([]byte, maxLineSize)
    // scanner.Buffer(buf, maxLineSize)

    for scanner.Scan() {
        commandLine := scanner.Text()
        log.Printf("Received command: %s", commandLine)

        // Process command...
        _, _ = conn.Write([]byte("OK\n"))
    }

    if err := scanner.Err(); err != nil {
        log.Printf("Scanner error: %v", err)
    }
}

Limitations of Delimiter-Based Framing

Payload Escaping: If the message payload itself contains a newline character, you must escape it (e.g., \n or URL encoding), adding CPU overhead.
Buffer Allocation Attacks: An adversary can send a continuous stream of bytes without ever transmitting a newline, exhausting server memory unless an explicit buffer limit is enforced.

Length-Prefixed Message Framing

Length-prefixed framing prepends each payload with a fixed-size integer indicating the number of payload bytes that follow.

+------------------------------------+----------------------------------------+
| Length Header (4 bytes, BigEndian) | Payload Data (N bytes)                 |
| [ 0x00, 0x00, 0x00, 0x05 ]         | [ 'H', 'e', 'l', 'l', 'o' ]            |
+------------------------------------+----------------------------------------+

Advantages of Length-Prefixed Framing

Arbitrary Payloads: The payload can contain raw binary data, null bytes, newlines, or compressed images without escaping.
Exact Memory Pre-allocation: The receiver reads the 4-byte header, verifies it is within acceptable limits, allocates a buffer of exact size \(N\), and reads exactly \(N\) bytes using io.ReadFull.

Reading Length-Prefixed Frames

func ReadMessage(r io.Reader) ([]byte, error) {
    var length uint32
    // Read 4-byte header
    if err := binary.Read(r, binary.BigEndian, &length); err != nil {
        return nil, err
    }

    // Security check: Guard against integer overflow or unbounded memory allocation
    const MaxPayloadSize = 16 * 1024 * 1024 // 16 MB limit
    if length > MaxPayloadSize {
        return nil, fmt.Errorf("payload size %d exceeds limit", length)
    }

    payload := make([]byte, length)
    // ReadFull guarantees reading exactly len(payload) bytes
    if _, err := io.ReadFull(r, payload); err != nil {
        return nil, err
    }

    return payload, nil
}

Binary Protocols with encoding/binary

When designing high-throughput services, binary protocols consume far less bandwidth and CPU than text formats. Go provides the encoding/binary package to marshal numbers to and from byte slices.

Endianness: Big-Endian vs. Little-Endian

Big-Endian (Network Byte Order): Most significant byte first. The standard convention across the Internet and RFC specifications.
Little-Endian: Least significant byte first. The native architecture of x86-64 and ARM CPUs.

val := uint32(0x12345678)

// Big-Endian wire format:   [0x12, 0x34, 0x56, 0x78]
// Little-Endian wire format: [0x78, 0x56, 0x34, 0x12]

In Go:

buf := make([]byte, 4)
binary.BigEndian.PutUint32(buf, 0x12345678)

parsed := binary.BigEndian.Uint32(buf)

Protocol Design Principles

When designing a production protocol, always include the following fields in your fixed-size frame header:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Magic Number (2B)       |  Version (1B) |   Type (1B)   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sequence ID (4B)                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Payload Length (4B)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Payload Data (...)                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      CRC32 Checksum (4B)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Magic Bytes (2–4 bytes): A constant signature (e.g., 0xCAFE) verifying that the connection is speaking your protocol and not junk data or an accidental HTTP scan.
Protocol Version (1 byte): Allows seamless protocol upgrades and backward compatibility.
Message Type (1 byte): Identifies message semantics (e.g., 1 = Ping, 2 = Request, 3 = Response, 4 = Error).
Correlation / Sequence ID (4–8 bytes): Allows clients to multiplex multiple requests over a single TCP connection and match out-of-order responses.
Payload Length (4 bytes): Tells the parser how many bytes to read for the payload.
Payload Integrity Check (Optional, 4 bytes): CRC32 or Adler32 checksum verifying data integrity against application-level memory bugs.

Building a Custom RPC Protocol from Scratch

Let’s implement a complete, production-ready frame parser and serializer for a custom binary protocol in Go.

// protocol.go
package main

import (
    "encoding/binary"
    "errors"
    "fmt"
    "hash/crc32"
    "io"
)

const (
    MagicBytes uint16 = 0x5443 // 'TC'
    Version1   uint8  = 1
    HeaderSize        = 12 // 2 (Magic) + 1 (Ver) + 1 (Type) + 4 (Seq) + 4 (Len)
)

type MessageType uint8

const (
    TypePing     MessageType = 1
    TypePong     MessageType = 2
    TypeRequest  MessageType = 3
    TypeResponse MessageType = 4
    TypeError    MessageType = 5
)

type Frame struct {
    Type     MessageType
    SeqID    uint32
    Payload  []byte
}

func EncodeFrame(w io.Writer, f Frame) error {
    payloadLen := uint32(len(f.Payload))
    buf := make([]byte, HeaderSize+len(f.Payload)+4) // +4 for CRC32

    // Pack header
    binary.BigEndian.PutUint16(buf[0:2], MagicBytes)
    buf[2] = Version1
    buf[3] = byte(f.Type)
    binary.BigEndian.PutUint32(buf[4:8], f.SeqID)
    binary.BigEndian.PutUint32(buf[8:12], payloadLen)

    // Copy payload
    copy(buf[12:12+payloadLen], f.Payload)

    // Calculate and append CRC32 checksum of payload
    checksum := crc32.ChecksumIEEE(f.Payload)
    binary.BigEndian.PutUint32(buf[12+payloadLen:], checksum)

    _, err := w.Write(buf)
    return err
}

func DecodeFrame(r io.Reader) (*Frame, error) {
    headerBuf := make([]byte, HeaderSize)
    if _, err := io.ReadFull(r, headerBuf); err != nil {
        return nil, err
    }

    magic := binary.BigEndian.Uint16(headerBuf[0:2])
    if magic != MagicBytes {
        return nil, fmt.Errorf("invalid protocol magic: 0x%X", magic)
    }

    ver := headerBuf[2]
    if ver != Version1 {
        return nil, fmt.Errorf("unsupported protocol version: %d", ver)
    }

    msgType := MessageType(headerBuf[3])
    seqID := binary.BigEndian.Uint32(headerBuf[4:8])
    payloadLen := binary.BigEndian.Uint32(headerBuf[8:12])

    const maxPayload = 8 * 1024 * 1024 // 8 MB limit
    if payloadLen > maxPayload {
        return nil, errors.New("payload length exceeds maximum allowed limit")
    }

    // Read payload + 4 bytes CRC
    bodyBuf := make([]byte, payloadLen+4)
    if _, err := io.ReadFull(r, bodyBuf); err != nil {
        return nil, err
    }

    payload := bodyBuf[:payloadLen]
    expectedChecksum := binary.BigEndian.Uint32(bodyBuf[payloadLen:])
    actualChecksum := crc32.ChecksumIEEE(payload)

    if actualChecksum != expectedChecksum {
        return nil, errors.New("payload CRC32 checksum mismatch: corrupted data")
    }

    return &Frame{
        Type:    msgType,
        SeqID:   seqID,
        Payload: payload,
    }, nil
}

Chapter Summary

TCP is an unstructured byte stream; applications must provide framing to distinguish messages.
Line-based text protocols are human-readable but require payload escaping and guardrails against unbounded lines.
Length-prefixed framing uses a fixed-size header to enable binary payloads and zero-copy pre-allocated reads via io.ReadFull.
encoding/binary serializes integers with strict endianness (use binary.BigEndian for network wire protocols).
Robust protocol designs incorporate magic numbers, version flags, sequence correlation IDs, and checksums.
In the next chapter, we will examine serialization formats—JSON, Protocol Buffers, and MessagePack—to structure payload data.

Chapter 9: Protocols: From Text to Binary to Custom Designs

Line-Based Text Protocols

Implementing Line-Based Framing with bufio.Scanner

Limitations of Delimiter-Based Framing

Length-Prefixed Message Framing

Advantages of Length-Prefixed Framing

Reading Length-Prefixed Frames

Binary Protocols with encoding/binary

Endianness: Big-Endian vs. Little-Endian

Protocol Design Principles

Building a Custom RPC Protocol from Scratch

Chapter Summary

Implementing Line-Based Framing with `bufio.Scanner`