Chapter 10: Serialization: JSON, Protobuf and Beyond

Updated

October 31, 2026

Chapter 10: Serialization: JSON, Protobuf and Beyond

Once you have framed your network messages, you must decide how to represent the data payload inside each frame. Serialization (marshalling) converts in-memory Go data structures into a sequence of bytes suitable for network transmission, and deserialization (unmarshalling) reconstructs those data structures on the receiving end.

In this chapter, you will evaluate the performance, wire efficiency, and ergonomics of JSON, Protocol Buffers, and MessagePack, learn how to choose the right format for your workload, and master techniques for streaming multi-gigabyte payloads without exhausting server memory.

JSON with encoding/json (When It Is Good Enough)

JavaScript Object Notation (JSON) is the lingua franca of web APIs and cloud microservices. Go’s standard library provides robust support via encoding/json.

Streaming JSON with json.Encoder and json.Decoder

When communicating over a network socket, avoid marshalling into intermediate []byte slices. Instead, connect json.Encoder and json.Decoder directly to the net.Conn stream:

type OrderPayload struct {
    OrderID   string  `json:"order_id"`
    UserID    int64   `json:"user_id"`
    Amount    float64 `json:"amount"`
    Timestamp int64   `json:"timestamp"`
}

// Server receiving JSON stream
func handleJSONStream(conn net.Conn) {
    defer conn.Close()
    decoder := json.NewDecoder(conn)
    encoder := json.NewEncoder(conn)

    for {
        var order OrderPayload
        if err := decoder.Decode(&order); err != nil {
            if errors.Is(err, io.EOF) {
                return
            }
            log.Printf("JSON decode error: %v", err)
            return
        }

        // Process order...
        response := map[string]string{"status": "ACCEPTED", "id": order.OrderID}
        if err := encoder.Encode(response); err != nil {
            return
        }
    }
}

When JSON is Good Enough

  • Public APIs & Web Clients: Broadest language support and browser-native compatibility.
  • Human Debuggability: Inspect payloads in Wireshark or logs without specialized tooling.
  • Low-to-Medium Throughput: Suitable when serialization CPU usage accounts for less than 5% of total request processing time.

Where JSON Fails

  • Heavy CPU Overhead: Reflection and string parsing consume significant CPU cycles.
  • Wire Bloat: Verbose field keys are repeated in every message.
  • Type Ambiguities: Floating-point numbers and 64-bit integer precision issues.

Protocol Buffers with Google’s protobuf-go

Google’s Protocol Buffers (Protobuf) is the gold standard for high-performance, internal microservice communication and gRPC.

The Protobuf Workflow

  1. Define a strict schema in a .proto file:
syntax = "proto3";
package order;
option go_package = "./proto";

message Order {
  string order_id = 1;
  int64 user_id = 2;
  double amount = 3;
  int64 timestamp = 4;
}
  1. Compile to Go code using protoc and protoc-gen-go:
protoc --go_out=. --go_opt=paths=source_relative proto/order.proto
  1. Serialize and deserialize using google.golang.org/protobuf/proto:
import (
    "google.golang.org/protobuf/proto"
    pb "myproject/proto"
)

// Marshalling to wire bytes:
order := &pb.Order{
    OrderId:   "ord-98712",
    UserId:    42100,
    Amount:    99.95,
    Timestamp: time.Now().Unix(),
}
wireBytes, err := proto.Marshal(order)

// Unmarshalling from wire bytes:
var received pb.Order
err = proto.Unmarshal(wireBytes, &received)

Why Protobuf Wins in Production

  • Compact Wire Size: Numeric field tags (1, 2, 3) replace verbose string keys; integers are encoded using varints (variable-length zigzag encoding).
  • Fast Serialization: Generated code avoids runtime reflection entirely.
  • Backward Compatibility: Fields can be added or deprecated safely without breaking older clients.

MessagePack and Other Binary Formats

When you want binary compactness and speed without the overhead of maintaining .proto schema files and compilation steps, MessagePack (msgpack) is an excellent compromise.

MessagePack encodes JSON-like structures directly into compact binary representations:

import "github.com/vmihailenco/msgpack/v5"

type Metric struct {
    Name  string  `msgpack:"name"`
    Value float64 `msgpack:"value"`
}

// Marshalling:
b, err := msgpack.Marshal(&Metric{Name: "cpu_usage", Value: 78.4})

// Unmarshalling:
var m Metric
err = msgpack.Unmarshal(b, &m)

Other zero-copy alternatives (such as FlatBuffers or Cap’n Proto) allow reading fields directly from raw network buffers without any deserialization step, ideal for gaming and high-frequency trading engines.

Choosing a Serialization Format

Criterion JSON MessagePack Protocol Buffers FlatBuffers
Human Readable Yes No No (requires schema) No
Strict Schema Optional (JSON Schema) No Enforced (.proto) Enforced
Serialization Speed Moderate Fast Very Fast Instant (Zero-Copy)
Payload Size Largest (100%) Compact (~60%) Highly Compact (~35%) Moderate
Tooling & Ecosystem Universal Broad Extensive (gRPC) Specialized
Best Used For Web APIs, Configs Dynamic binary RPC Microservices, gRPC HFT, Game Physics

Streaming Large Payloads Without Loading Everything Into Memory

If a client sends a 5 GB database backup or video file over a socket, reading the whole payload into memory (io.ReadAll) will trigger an Out-Of-Memory (OOM) crash:

// CATASTROPHIC BUG: OOM on 5GB payload!
data, err := io.ReadAll(conn)

1. Chunked Streaming with io.Copy

Use io.Copy or io.CopyBuffer to stream data directly from the network socket into disk storage or another network pipe in constant memory (\(O(1)\) space):

func receiveFile(conn net.Conn, destPath string) error {
    defer conn.Close()

    file, err := os.Create(destPath)
    if err != nil {
        return err
    }
    defer file.Close()

    // io.Copy transfers data in 32KB internal buffers,
    // never consuming more than 32KB of RAM regardless of file size!
    written, err := io.Copy(file, conn)
    log.Printf("Successfully saved %d bytes to %s", written, destPath)
    return err
}

2. Using io.Pipe for Streaming Transformations

To encrypt, compress, or hash data on the fly as it streams from a socket, connect an io.Pipe:

pr, pw := io.Pipe()

// Goroutine 1: Read from socket and write into pipe
go func() {
    defer pw.Close()
    _, _ = io.Copy(pw, conn)
}()

// Goroutine 2: Read from pipe and write to decompression or hash
hash := sha256.New()
_, _ = io.Copy(hash, pr)

Chapter Summary

  • encoding/json with json.NewDecoder and json.NewEncoder enables straightforward streaming of JSON messages directly over sockets.
  • Protocol Buffers provide strict schema enforcement, compact binary wire formats, and fast execution without runtime reflection.
  • MessagePack offers compact binary efficiency without requiring code generation or schemas.
  • Never use io.ReadAll for unbounded network payloads; use io.Copy and io.Pipe to stream arbitrary payload sizes in constant \(O(1)\) memory.
  • In the next chapter, we will learn how to secure socket communication using Transport Layer Security (TLS 1.3) and mutual TLS (mTLS).