Chapter 11: The Networking Stack and Performance

Updated

October 31, 2026

Chapter 11: The Networking Stack and Performance

Modern data center networks operate at line rates of 25Gbps, 100Gbps, and 400Gbps. At 100Gbps, a server can receive over 14 million packets per second (Mpps) for standard 1500-byte frames, or up to 148 Mpps for 64-byte minimum packets. Processing that packet volume through the kernel requires deep understanding of hardware NIC queues, interrupt mitigation, softirqs, socket buffers, and TCP congestion algorithms.

This chapter details the packet journey through the Linux network stack and examines the architectural levers used to maximize network throughput and minimize packet latency.

The Networking Data Path

The Linux network receive path moves packets through hardware, driver, and protocol layers:

┌────────────────────────────────────────────────────────────────────────┐
│                        NETWORK RECEIVE PATH (RX)                       │
│                                                                        │
│   1. Physical Wire / Fiber ──► NIC MAC/PHY                             │
│                                │                                       │
│                                ▼                                       │
│   2. NIC DMA Engine ──────────► Copies frame to Host RAM RX Ring Buffer│
│                                │                                       │
│                                ▼                                       │
│   3. Hardware MSI-X IRQ ──────► Hard IRQ Handler schedules NAPI       │
│                                │                                       │
│                                ▼                                       │
│   4. NET_RX_SOFTIRQ ──────────► napi_poll() drains RX ring budget      │
│                                │ Allocates struct sk_buff (skb)        │
│                                ▼                                       │
│   5. IP Layer (ip_rcv) ───────► Route lookup, checksum verification    │
│                                │                                       │
│                                ▼                                       │
│   6. TCP Layer (tcp_v4_rcv) ──► Reassembly, flow control, ACKs         │
│                                │ Copies payload into Socket Buffer     │
│                                ▼                                       │
│   7. Userspace App ───────────► Woken up via epoll / reads data        │
└────────────────────────────────────────────────────────────────────────┘
  • Socket Buffer (struct sk_buff): The core network data structure in Linux. An sk_buff (or skb) wraps packet headers and data buffers, traveling through protocol layers without copying the underlying packet payload.

Socket Buffering and Memory

Every network socket allocates dedicated memory buffers for receiving and transmitting data:

  • Receive Buffer (SO_RCVBUF): Holds incoming data that has been acknowledged by TCP but not yet read by userspace.
  • Send Buffer (SO_SNDBUF): Holds outbound data queued for transmission or awaiting peer acknowledgment for retransmission.
  • TCP Autotuning: Modern Linux automatically adjusts socket buffer sizes dynamically based on the current connection’s Bandwidth-Delay Product (BDP):
# Display min, default, and max TCP buffer sizes in bytes
cat /proc/sys/net/ipv4/tcp_rmem   # Read buffer (e.g. 4096  131072  6291456)
cat /proc/sys/net/ipv4/tcp_wmem   # Write buffer (e.g. 4096  16384   4194304)

NIC Interrupts and Softirq Processing

Generating a hardware interrupt for every incoming packet at 100Gbps would bring any multi-core CPU to an immediate halt. Linux solves this with the NAPI (New API) subsystem:

  1. The first incoming packet triggers a hardware interrupt.
  2. The interrupt handler disables further device interrupts and schedules NET_RX_SOFTIRQ.
  3. The softirq daemon polls the NIC’s ring buffer in batches (controlled by netdev_budget, default 300 packets) using high-speed polling loops.
  4. Once the ring buffer is empty, interrupts are re-enabled.
# Monitor softirq saturation on CPU cores
cat /proc/softirqs | grep NET_RX

If a single core shows 100% CPU utilization in si (software interrupt) on top, that core is saturated by network packet processing.

RSS, RPS, XPS: Distributing Network Load

To scale packet processing across dozens of CPU cores, Linux employs hardware and software steering mechanisms:

┌────────────────────────────────────────────────────────────────────────┐
│                   PACKET STEERING MECHANISMS                           │
│                                                                        │
│   Incoming Packet ──► [ NIC Hardware Hash (Toeplitz) ]                 │
│                              │                                         │
│                              ▼                                         │
│   Receive Side Scaling (RSS) ──► Dispatches to Hardware Queue 0..N     │
│                                 Each bound to dedicated CPU via MSI-X  │
│                                                                        │
│   Receive Packet Steering (RPS) ──► Software fallback: hashes packet   │
│                                     and forwards skb to target CPU     │
│                                                                        │
│   Transmit Packet Steering (XPS) ──► Maps CPU to dedicated TX queue    │
│                                     for lockless outbound transmission │
└────────────────────────────────────────────────────────────────────────┘
  • RSS (Receive Side Scaling): Hardware-level distribution. Modern multi-queue NICs hash packet 4-tuples (IP/Port pairs) and route packets directly to distinct hardware queues, each tied to an independent CPU core via dedicated MSI-X interrupts.
  • RPS (Receive Packet Steering): Software RSS emulation. Distributes packets across CPUs when NIC hardware lacks multi-queue support.
  • XPS (Transmit Packet Steering): Assigns dedicated transmission queues per CPU core, eliminating lock contention on outbound traffic.

TCP Congestion Control and Buffer Tuning

TCP congestion control algorithms govern how aggressively a sender increases its transmission window (cwnd) and how it reacts to packet loss or delay:

  • CUBIC: Default on Linux for years. Uses a cubic function of time since the last loss event to scale cwnd. Performs exceptionally well in high-bandwidth, low-latency data center environments, but can suffer bufferbloat on lossy wide-area links.
  • BBR (Bottleneck Bandwidth and RTT): Developed by Google. Models the physical delivery rate and round-trip time of the network path directly, transmitting at the bottleneck bandwidth rather than filling network buffers until packet drops occur. BBR delivers dramatically higher throughput over lossy or high-latency WAN links.
# Check available and active congestion control algorithms
sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control

UDP and Raw Sockets Performance

Unlike TCP, UDP has no flow control, acknowledgments, or window retransmissions:

  • Packet Drops: If the receiving application reads from the UDP socket more slowly than incoming packets arrive, the socket receive buffer (SO_RCVBUF) overflows, and incoming packets are silently discarded by the kernel (RcvbufErrors in /proc/net/snmp).
  • Syscall Overhead: Sending or receiving millions of UDP packets using traditional single-packet recv() / send() syscalls burns massive CPU cycles.
  • Batching Syscalls: Modern applications use recvmmsg() and sendmmsg() to transmit or receive hundreds of packets in a single system call, reducing boundary crossing overhead by up to 80%.