Chapter 2: Linux Kernel Architecture and Performance

Updated

October 31, 2026

Chapter 2: Linux Kernel Architecture and Performance

To extract maximum performance from Linux, an engineer must understand the operating system not as an opaque black box, but as a concurrent supervisor managing hardware privilege boundaries. Every interaction between an application and the outside world—allocating memory, sending packets, writing to disks—incurs architectural overhead across the hardware privilege boundary.

This chapter analyzes the internal mechanics and latency costs of kernel boundaries, system calls, task structures, context switches, interrupt handling, and kernel synchronization primitives.

Kernel Space and User Space: The Boundary Cost

Modern processors enforce security and memory isolation using hardware privilege levels:

  • Ring 0 / EL1 (Kernel Mode): Unrestricted execution privileges, direct hardware access, full page table manipulation.
  • Ring 3 / EL0 (User Mode): Restricted execution environment, isolated virtual address spaces, hardware memory protection faults on unauthorized accesses.
┌────────────────────────────────────────────────────────────────────────┐
│                   PRIVILEGE TRANSITION OVERHEAD                        │
│                                                                        │
│   Userspace (Ring 3)                                                   │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │ Application Code ──► libc wrapper ──► SYSCALL instruction     │   │
│   └──────────────────────────────────────────────┬─────────────────┘   │
│                                                  │ Privilege Transition│
│   Hardware Boundary                              ▼ (~50 - 150 ns)      │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │ Switch stack pointer (RSP -> Kernel Stack)                     │   │
│   │ Save user registers (struct pt_regs)                           │   │
│   │ Switch Page Table Isolation / KPTI (if active)                 │   │
│   │ Clear speculative execution buffers (Spectre / Meltdown)       │   │
│   └──────────────────────────────────────────────┬─────────────────┘   │
│                                                  │                     │
│   Kernelspace (Ring 0)                           ▼                     │
│   ┌────────────────────────────────────────────────────────────────┐   │
│   │ Syscall Dispatch Table ──► sys_read() / sys_write()            │   │
│   └────────────────────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────────────────────┘

The transition between Ring 3 and Ring 0 is not free. Modern mitigation mechanisms (Kernel Page Table Isolation - KPTI, retpolines, speculative store bypass disabling) increased syscall entry/exit costs from ~50ns to over 150ns on affected x86 architectures. High-frequency syscall loops (such as calling gettimeofday or tiny read() calls in a tight loop) can burn massive CPU cycles purely on boundary crossing.

Fast Userspace Alternatives: vDSO

To eliminate boundary overhead for frequent read-only queries, the kernel exports the vDSO (virtual Dynamic Shared Object). Memory pages containing current time values (clock_gettime, gettimeofday) and CPU identity (getcpu) are mapped directly into userspace address spaces, allowing libc to return hardware-accurate results without entering Ring 0.

System Calls: Entry, Exit and Overhead

On x86_64, system calls enter via the SYSCALL instruction and return via SYSRET. The kernel saves caller-saved registers into a struct pt_regs structure on the task’s kernel stack:

// Simplified representation of pt_regs saved during syscall entry
struct pt_regs {
    unsigned long r15, r14, r13, r12, rbp, rbx;
    unsigned long r11, r10, r9, r8, ax, cx, dx, si, di;
    unsigned long orig_ax;
    unsigned long ip, cs, flags, sp, ss;
};
  1. Parameter Passing: On x86_64, system call arguments pass in registers rdi, rsi, rdx, r10, r8, r9, and the syscall number passes in rax.
  2. Dispatch Overhead: The kernel indexes into the sys_call_table array, executes the handler, and verifies return values.
  3. Copy Overhead: For read/write operations, data must be copied between user buffers and kernel page cache buffers via copy_to_user() and copy_from_user(), validating memory bounds on each byte.

Processes, Threads and the Task Structure

In the Linux kernel, there is no fundamental architectural difference between a “process” and a “thread”. Both are represented by instances of struct task_struct:

  • A Process is a task that owns a unique virtual memory descriptor (struct mm_struct), unique file descriptor table (struct files_struct), and unique signal handlers.
  • A Thread is a task created via clone() with shared memory and descriptor flags (CLONE_VM, CLONE_FILES, CLONE_FS, CLONE_SIGHAND), sharing its parent’s mm_struct.

Because tasks are the universal schedulable unit, the kernel scheduler treats threads and processes identically when placing them on CPU runqueues.

Context Switching: Mechanics and Cost

A context switch occurs when the CPU halts execution of one task and resumes execution of another. Context switches are either:

  • Voluntary: The task voluntarily yields the CPU by sleeping, waiting for I/O completion, or blocking on a lock (schedule()).
  • Involuntary: The scheduler preempts the task because its allocated time quantum expired, or a higher-priority task became runnable.
# View context switch rates system-wide (csw/s)
vmstat 1 3

# View voluntary vs involuntary context switches per process
pidstat -w -p <PID> 1 3

The Two Costs of Context Switches

  1. Direct Cost (Registers & Stacks): Saving CPU registers, swapping the kernel stack pointer, and switching architecture-specific hardware state (e.g., FPU/AVX vector registers). This takes ~1 to 3 microseconds.
  2. Indirect Cost (Cache Eviction): Swapping the CR3 register to point to a new page table invalidates the Translation Lookaside Buffer (TLB). As the new task runs, it suffers cold L1/L2/L3 cache misses as its working set replaces the old task’s cache lines. Indirect cache pollution costs often exceed 10–30 microseconds of degraded execution speed.

Interrupts, Softirqs and Bottom Halves

Hardware devices notify the CPU of events (e.g., incoming network packets, completed NVMe blocks) via hardware interrupts (IRQs). To prevent long-running interrupt handlers from freezing the system, Linux splits interrupt processing into two tiers:

  1. Top Half (Hard IRQ): Executes immediately with interrupts disabled on the receiving CPU. It acknowledges the hardware controller, acknowledges the interrupt, moves data into a temporary buffer, schedules the bottom half, and exits in microseconds.
  2. Bottom Half (Softirq / Tasklet / Workqueue): Executes asynchronously with interrupts enabled:
    • Softirqs: Fixed in-kernel vectors (NET_RX_SOFTIRQ, BLOCK_SOFTIRQ, SCHED_SOFTIRQ, RCU_SOFTIRQ). High throughput, runs in interrupt context.
    • ksoftirqd: Dedicated per-CPU kernel threads (ksoftirqd/N) that execute when softirq load is so heavy that running them in-line would starve userspace tasks.
# Inspect softirq activity across all CPUs
cat /proc/softirqs

If /proc/softirqs shows NET_RX concentrated entirely on CPU 0, that core will suffer softirq starvation, dropping packets while other cores remain idle.

Synchronization Primitives: Locks, Spinlocks and Contention

Concurrent kernel execution requires synchronization across multiple CPU cores:

Primitive Execution Context Mechanism on Contention Overhead / Use Case
Spinlock (spinlock_t) Atomic / Interrupt context Busy-waits (spins in a tight CPU loop) Ultra-fast (<10ns) for brief critical sections. Must never sleep. High CPU burn if contested.
Mutex (struct mutex) Sleepable / Process context Enqueues task onto a wait queue and context switches Used for long critical sections. Incurs context switch overhead on contention.
Futex (Fast Userspace Mutex) Userspace / Kernel hybrid Uncontested: atomic CAS in userspace. Contested: kernel syscall to sleep. The foundation of pthread mutexes. Zero syscall cost when uncontested.
Read-Copy Update (RCU) Read-mostly data structures Readers execute lock-free without atomics. Writers stage copies and defer reclamation. Extreme read scalability across thousands of cores (routing tables, VFS dentries).

Contention on spinlocks or mutexes destroys horizontal CPU scalability. As core counts scale from 16 to 128 cores, lock convoys and cache coherency bounce can cause a 10x degradation in throughput. Profiling tools like perf lock and bpftrace are essential for identifying contested lock addresses in production.