Chapter 3: CPU Architecture and Performance Counters

Chapter 3: CPU Architecture and Performance Counters

Modern CPUs are extraordinarily complex superscalar, out-of-order execution engines. Understanding clock frequency alone tells you almost nothing about how effectively software utilizes the processor. Two workloads consuming 100% CPU on top can deliver radically different actual performance depending on hardware pipeline stalls, cache misses, branch mispredictions, and memory bus saturation.

This chapter breaks down modern processor microarchitecture and demonstrates how to measure hardware-level efficiency using Performance Monitoring Units (PMUs) and hardware counters.

Modern CPU Microarchitecture: Pipelines and Parallelism

A modern processor core does not execute instructions sequentially one at a time. Instead, it utilizes deep pipelines and superscalar execution:

┌────────────────────────────────────────────────────────────────────────┐
│                   SUPERSCALAR OUT-OF-ORDER PIPELINE                    │
│                                                                        │
│   Instruction Fetch ──► Decode & Micro-op Translation                  │
│                                │                                       │
│                                ▼                                       │
│                   Register Renaming & Allocation                       │
│                                │                                       │
│                                ▼                                       │
│                   Reservation Station / Reorder Buffer                 │
│                                │                                       │
│          ┌─────────────────────┼─────────────────────┐                 │
│          ▼                     ▼                     ▼                 │
│   [ ALU / Int Port ]    [ Vector / SIMD ]     [ Load / Store Queue ]   │
│          │                     │                     │                 │
│          └─────────────────────┼─────────────────────┘                 │
│                                ▼                                       │
│                     Commit / Retirement Stage                          │
└────────────────────────────────────────────────────────────────────────┘

• Superscalar Execution: Modern cores can decode, dispatch, and retire multiple instructions per clock cycle (often 4 to 8 micro-ops per cycle).
• Out-of-Order (OoO) Execution: If an instruction is waiting for data to load from main memory, the core dynamically finds and executes subsequent independent instructions to keep execution units saturated.
• Retirement: Instructions execute speculatively and out of order, but results are committed to architectural registers in strict original program order.

Instructions Per Cycle (IPC)

The ultimate metric of CPU execution efficiency is IPC (Instructions Per Cycle): \[\text{IPC} = \frac{\text{Instructions Retired}}{\text{CPU Clock Cycles}}\]

• IPC > 1.5 - 2.5: Excellent pipeline utilization. Workload is compute-bound and cache-friendly.
• IPC < 0.7: Poor execution efficiency. The CPU is spending the majority of its clock cycles stalled waiting for memory lines or branch resolution.

Cache Hierarchy: L1, L2, L3, TLB and Cache Misses

Memory access latency is the primary bottleneck in modern computing. Accessing physical RAM requires hundreds of clock cycles, while L1 cache takes only a handful:

Level	Typical Capacity	Access Latency	Scope
L1 Data / Instruction (L1d/L1i)	32 KB - 64 KB per core	~4 - 5 clock cycles (~1 ns)	Private to core
L2 Cache	512 KB - 1 MB per core	~12 - 14 clock cycles (~3 - 4 ns)	Private to core
L3 Cache (Last Level Cache - LLC)	32 MB - 256 MB per socket	~40 - 60 clock cycles (~12 - 18 ns)	Shared across cores in CCX/Socket
Main Memory (DRAM)	64 GB - 2 TB	~200 - 300 clock cycles (~60 - 90 ns)	Shared via memory controllers
Remote NUMA DRAM	Multi-socket nodes	~400 - 600 clock cycles (~120 - 180 ns)	Traverses UPI/Infinity Fabric

# Inspect hardware cache topology
lscpu -C

Cache Lines and False Sharing

Memory moves between DRAM and CPU caches in fixed 64-byte chunks called cache lines.

• True Sharing: Multiple cores read and write shared data structures protected by locks.
• False Sharing: Two independent threads running on separate cores modify completely distinct variables that happen to reside within the same 64-byte cache line. The cache coherence protocol (MESI/MOESI) invalidates the cache line across cores on every write, causing severe bus contention and latency spikes.

Branch Prediction and Speculative Execution

Because pipelines are 14 to 20 stages deep, the processor cannot wait for conditional branches (if/else, loops) to evaluate before fetching subsequent instructions.

• Branch Predictor: Sophisticated neural-network and pattern-history tables predict which branch path will be taken with >95% accuracy.
• Speculative Execution: The CPU executes the predicted path ahead of time. If the prediction was correct, the results commit instantly.
• Branch Misprediction Penalty: If the prediction was wrong, the entire speculative pipeline must be flushed and refetched, burning 15 to 20 clock cycles of pure stall overhead.

Cores, Hyperthreading and SMT

Simultaneous Multithreading (SMT), known as Hyper-Threading on Intel, exposes two logical processors per physical core:

• SMT duplicates architectural register state (instruction pointers, general registers) while sharing the underlying execution units, pipelines, and L1/L2 caches.
• Throughput vs Latency Trade-off: SMT increases total system throughput by 20%–35% by keeping execution units busy during single-thread memory stalls. However, SMT increases tail latency jitter for latency-critical workloads due to resource contention between co-located threads.

In high-frequency trading and hard real-time environments, engineers often disable SMT via /sys/devices/system/cpu/smt/control to eliminate noisy neighbor effects between hyperthreads.

CPU Frequency Scaling and Power States

Processors dynamically adjust clock frequencies and power consumption to balance performance and thermal envelopes:

• C-States (Idle States): \(C0\) is fully active. Deeper states (\(C1, C6, C8\)) power down internal core circuitry and flush caches. Waking from deep C-states takes 10 to 100 microseconds, introducing latency spikes on bursty workloads.
• P-States (Performance States): Dynamic Voltage and Frequency Scaling (DVFS). P0 is maximum turbo frequency; higher numbers indicate lower clock speeds.
• Linux Governors:
  • powersave: Minimizes power draw (often default on Intel systems running intel_pstate).
  • performance: Locks the CPU at its maximum non-turbo or turbo frequency.
  • schedutil: Adjusts frequency dynamically based on CFS scheduler runqueue utilization.

# Check current CPU frequency and active governor
cpupower frequency-info

# Lock all cores to maximum performance mode
sudo cpupower frequency-set -g performance

Performance Counters and Hardware Events

Every modern enterprise CPU includes a dedicated Performance Monitoring Unit (PMU) capable of counting hardware events with zero impact on application behavior:

# Measure high-level hardware efficiency of a command
perf stat -B -e \
  cycles,instructions,cache-references,cache-misses,branches,branch-misses \
  stress-ng --cpu 4 --timeout 5s

Example output analysis:

       4,120,490,120      cycles                    #    3.210 GHz
       5,892,102,400      instructions              #    1.43  insn per cycle
          18,490,120      cache-references          #   14.400 M/sec
           2,104,890      cache-misses              #   11.38% of all L1D hits
         840,290,100      branches                  #  654.400 M/sec
          12,400,200      branch-misses             #    1.48% of all branches

By profiling IPC, LLC cache miss rates, and branch miss percentages, performance engineers can immediately pinpoint whether an application bottleneck is bound by compute pipelines, memory bandwidth, or unpredictable control flow.