Writing Kernel with AI
Writing Kernel with AI
CPU to AI-Written Kernels
A comprehensive, production-grade guide to high-performance kernel engineering: from the fundamentals of CPU instruction parallelism to hand-tuned CUDA, FlashAttention, modern Hopper/Blackwell hardware architectures, and AI-driven kernel optimization loops.
Welcome to Kernel Engineering
Accelerated computing and modern artificial intelligence rest on a single engineering discipline: extracting peak theoretical throughput from hardware.
Whether training frontier foundation models or serving millions of concurrent inference tokens, the software stack converges onto specialized computational kernels. Writing fast kernels is not a matter of guesswork or trial-and-error—it is an empirical science governed by the physics of computation: arithmetic intensity, memory bandwidth, latency hiding, and hardware execution hierarchies.
The Trajectory of This Book
This curriculum spans six rigorous parts designed to take you from first principles to the cutting edge:
- Part I: Parallelism, from the CPU up: Understand how modern CPUs achieve massive throughput via SIMD vectorization and multi-threading before stepping into the GPU execution hierarchy: warps, streaming multiprocessors, memory coalescing, and the Roofline model.
- Part II: The GEMM Worklog: Naive → cuBLAS: Master General Matrix Multiply (GEMM)—the operation that comprises over 90% of deep learning FLOPs. Step by step, elevate a naive kernel achieving 1.3% of GPU capacity to a hierarchical warp-tiled and Tensor Core (WMMA) implementation rivaling NVIDIA’s cuBLAS library.
- Part III: Attention & Profiling: Diagnose hardware bottlenecks using NVIDIA Nsight Compute like a compiler engineer. Construct FlashAttention v1 from scratch, utilizing the online softmax algorithm to eliminate high-bandwidth memory bottlenecks.
- Part IV: The Modern Frontier: Explore modern production kernels running on Hopper (H100) and Blackwell (B200) architectures: FlashAttention-2/3/4, TMA asynchronous copy, WGMMA, Triton, CUTLASS, CuTe-DSL, PagedAttention, and NVFP4.
- Part V: AI-Written Kernels: Analyze DeepSeek’s ultra-compact, high-performance kernels (FlashMLA and DeepGEMM). Step into the future of kernel development: using autonomous LLM agents (e.g., DeepMind AlphaEvolve, KernelBench) with profiler-in-the-loop guidance.
- Part VI: Kernel Engineering Capstone Projects: Put your skills into practice with industry-grade capstone specifications developed with Crusoe, complete with verification harnesses, performance benchmarks, and demo-day technical presentations.
Who This Is For
- ML Infrastructure & Systems Engineers: Developers optimizing training clusters and inference backends (vLLM, SGLang, TensorRT-LLM).
- GPU Programmers & Performance Engineers: Engineers seeking mastery of modern GPU architectures (Hopper, Blackwell), memory coalescing, and async execution pipelines.
- Software Engineers Transitioning to Accelerated Computing: Developers wanting deep understanding of what happens when high-level PyTorch or JAX code hits the silicon.
Formats
This book is available in multiple formats via the library portal: - HTML — responsive interactive reading experience with syntax highlighting and light/dark theme toggle. - PDF — complete typeset document for offline review and reference. - EPUB — eBook format optimized for tablets and mobile e-readers.