Inference-Serving Kernels

Inference-Serving Kernels

LLM inference splits into two distinct execution regimes with radically different performance characteristics: 1. Prefill Phase: The prompt is processed in parallel. High operational intensity, strictly compute-bound. 2. Decode Phase: Tokens are generated autoregressively one by one. The model loads billions of weights to compute a single new vector. Strictly memory-bound!

PagedAttention (vLLM)

In standard inference, KV caches must be allocated as contiguous memory buffers. This leads to 60-80% memory waste due to internal and external memory fragmentation. PagedAttention treats KV cache memory like virtual memory pages in an operating system: - KV vectors are stored in non-contiguous 16-token page blocks. - A page table maps logical token positions to physical GPU memory addresses. - Result: Memory waste drops to near 0%, enabling 2-4x higher serving concurrency on the same hardware.