Chapter 16: Debugging and Diagnosing Network Problems

Updated

October 31, 2026

Chapter 16: Debugging and Diagnosing Network Problems

When a distributed service slows down or drops connections in production, application logs and high-level metrics are often insufficient. Packets drop inside router queues, firewalls silently inject RST packets, and application code stalls inside un-instrumented system calls.

To diagnose deep network failures, you must look beneath the Go runtime and examine raw packets, kernel socket buffers, and system calls.

In this chapter, you will master essential network diagnostic tooling: packet capture with tcpdump, protocol analysis with Wireshark, system call tracing with strace, socket inspection with ss, and runtime goroutine debugging with delve.

When Profiling Is Not Enough

Standard profiling (pprof) tells you what your Go code is doing while CPU is actively running or memory is being allocated. But what if your server is completely idle, CPU usage is 0%, yet all client requests are timing out?

In these scenarios, goroutines are blocked in kernel sleep states (_Gwaiting), packets are being dropped by intermediate switches, or the kernel socket receive buffer is overflowing. Solving these issues requires inspecting the boundary between the Go runtime and the Linux kernel.

Packet Capture with tcpdump

tcpdump is the premier command-line packet analyzer for Linux and Unix systems. It captures raw frames directly from network interfaces using the kernel’s packet filter (BPF).

Essential tcpdump Command Recipes

# Capture packets on loopback interface for port 8080 without resolving DNS
sudo tcpdump -i lo -nn port 8080

# Capture with ASCII and Hex payload inspection (-X)
sudo tcpdump -i any -nn -X -s 0 'tcp port 9090'

# Save packets to a pcap file for offline Wireshark analysis
sudo tcpdump -i eth0 -nn -w capture.pcap 'tcp port 443'

# Filter specifically for TCP RST (reset) packets
sudo tcpdump -i any 'tcp[tcpflags] & (tcp-rst) != 0'

# Filter for TCP SYN or FIN packets (connection lifecycle events)
sudo tcpdump -i any 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0'

Analyzing Packets with Wireshark

Wireshark provides a graphical interface for analyzing .pcap files captured by tcpdump.

[ Frame List: No., Time, Source, Destination, Protocol, Length, Info ]
┌───────────────────────────────────────────────────────────────────┐
│ 1  0.000000  127.0.0.1  127.0.0.1  TCP  74  54321 → 8080 [SYN]    │
│ 2  0.000042  127.0.0.1  127.0.0.1  TCP  74  8080 → 54321 [SYN,ACK]│
│ 3  0.000065  127.0.0.1  127.0.0.1  TCP  66  54321 → 8080 [ACK]    │
│ 4  0.001200  127.0.0.1  127.0.0.1  TCP  85  54321 → 8080 [PSH,ACK]│
└───────────────────────────────────────────────────────────────────┘

Key Wireshark Workflows for Go Engineers

  1. Follow TCP Stream: Right-click any packet and select Follow \(\to\) TCP Stream to reconstruct the exact bidirectional byte conversation between client and server.
  2. Detecting Retransmissions: In the display filter, enter tcp.analysis.retransmission or tcp.analysis.lost_segment to instantly identify packet loss and network congestion.
  3. Round-Trip Time (RTT) Graphing: Go to Statistics \(\to\) TCP Stream Graphs \(\to\) Round Trip Time to see if latency spikes correlate with packet retransmissions.

System Call Tracing with strace

strace intercepts and records the system calls made by a process and the signals received by it.

Attaching strace to a Running Go Service

# Trace all threads (-f) for a running Go PID, filtering network calls
sudo strace -f -p <PID> -e trace=network,epoll_wait -s 128

Understanding strace Output

// Goroutine accepts an inbound connection:
[pid 12345] accept4(3, {sa_family=AF_INET, sin_port=htons(54321), sin_addr=inet_addr("127.0.0.1")}, [112], SOCK_CLOEXEC|SOCK_NONBLOCK) = 6

// Setting socket options:
[pid 12345] setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0

// Netpoller registers the new descriptor with epoll:
[pid 12345] epoll_ctl(4, EPOLL_CTL_ADD, 6, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1432, u64=1432}}) = 0

// Non-blocking read returning data:
[pid 12345] read(6, "GET / HTTP/1.1\r\nHost: ...", 4096) = 28

If you see read(6, ...) returning EAGAIN repeatedly, you know the Go runtime netpoller is properly handling non-blocking states. If you see a file descriptor remaining open without subsequent reads or closes, you have found a connection leak.

Connection State Inspection with ss

The ss (Socket Statistics) command replaces legacy netstat, querying socket details directly from the Linux kernel tcp_diag subsystem.

# Display all listening and established TCP sockets with process info:
ss -t -a -p

# Display internal TCP socket timers and memory buffers:
ss -t -i -e -a

Interpreting Recv-Q and Send-Q

State      Recv-Q Send-Q  Local Address:Port   Peer Address:Port
LISTEN     128    0       0.0.0.0:8080         0.0.0.0:*
ESTAB      0      34560   10.0.0.5:8080        198.51.100.2:45210
ESTAB      16384  0       10.0.0.5:8080        198.51.100.9:51234
  • For Listening Sockets:
    • Send-Q: The maximum backlog queue size (somaxconn).
    • Recv-Q: The number of completed connections currently waiting to be accepted by ln.Accept(). If Recv-Q > 0 on a listening socket, your application’s accept loop is falling behind!
  • For Established Sockets:
    • Send-Q: Bytes buffered in kernel memory waiting to be sent or acknowledged by peer. High values indicate a congested network or stalled client.
    • Recv-Q: Bytes received by kernel but not yet read by application via conn.Read(). High values indicate your Go worker goroutine is bottlenecked or stuck.

Debugging Goroutine Hangs with delve

When your Go application freezes, attach the Delve (dlv) debugger to inspect all live goroutines without restarting the process.

# Attach delve to the running process
sudo dlv attach <PID>

Delve Commands

# List all goroutines and their current wait state
(dlv) goroutines

# Inspect a specific stuck goroutine
(dlv) goroutine 42

# View stack trace
(dlv) stack

Look for goroutines stuck in runtime.netpollblock or internal/poll.(*FD).Read. If a goroutine has been in that state for hours, you have identified a missing read deadline!

Real-World Debugging Walkthroughs

Case 1: The Accumulating CLOSE_WAIT Sockets

Symptom: Running ss -s reveals 15,000 sockets in CLOSE_WAIT state, eventually exhausting file descriptors. Root Cause: CLOSE_WAIT means the remote client has sent a FIN (closed their side), but the local Go application has never called conn.Close(). Fix: Search the codebase for early return statements that bypass defer conn.Close().

Case 2: Sudden Connection Reset by Peer (RST)

Symptom: Clients report intermittent read: connection reset by peer errors. Root Cause Captured in tcpdump: The Go server called conn.Close() while there was still unread data sitting in the socket’s kernel receive buffer. By default, closing a socket with unread data causes the Linux TCP stack to immediately send an abortive RST instead of a graceful FIN. Fix: Ensure all client input is drained before terminating the connection, or use CloseWrite() to signal half-close.

Chapter Summary

  • When high-level metrics fail, inspect kernel sockets and raw frames using tcpdump, ss, strace, and delve.
  • tcpdump -i any -nn captures raw packets with BPF filters for offline analysis in Wireshark.
  • strace -f -e trace=network reveals exact non-blocking system calls, epoll registrations, and file descriptor leaks.
  • On listening sockets, a non-zero Recv-Q in ss proves the application accept loop cannot keep pace with inbound connection arrival rates.
  • Sockets stuck in CLOSE_WAIT indicate the local application forgot to call conn.Close().
  • In the final chapter, we will synthesize our learnings into the guiding principles of reliable network programming in Go.