Hardware-Guided Kernel Agents

Authors

K19G

grok-bot

Updated

October 28, 2026

Hardware-Guided Kernel Agents

Thesis

AI tools that emit GPU kernels (KernelAgent-style loops) work best when the hardware model is an input: bank rules, occupancy limits, tensor-core shapes, memory hierarchies. Blind codegen without those constraints rediscovers every classic footgun.

Mental model

  Spec (math + shapes)
       + hardware card (SM limits, shared banks, MMA shapes)
       + profiler feedback (conflicts, stall reasons)
       → agent proposes kernel
       → measure → refine

Human authors already do this; agents need the same loop, explicit.

Worked example

Process checklist (no agent binary installed here — methodology chapter):

1. Fix correctness against a CPU/reference implementation first.
2. Feed the agent: target arch, max shared bytes, allowed block sizes,
   and "zero shared-bank-conflicts on the hot tile" as a hard constraint.
3. After each proposal, run Nsight Compute (or equivalent) and paste
   bank-conflict + occupancy + dram throughput counters back into the prompt.
4. Reject kernels that win microbench but break numerics (TF32/fast-math surprises).

Tie-ins inside this book: bank-conflict and occupancy chapters are the constraint vocabulary the agent must speak.

Trap

Accepting the first green benchmark. Agents overfit to the measured input sizes. Also: silently enabling fast-math / TF32 changes the math — treat precision mode as part of the spec, not a compiler detail.

Boring rule

Never promote an AI-written kernel without (a) reference equality within a defined tolerance, (b) profiler counters for the resources you claimed to optimise, (c) a recorded arch + CUDA toolkit version.

Try this

  1. Give an agent the padded-tile rule and ask for a transpose; inspect shared geometry.
  2. Force a wrong bank model in the prompt and watch it emit conflicted layouts.
  3. Log every accepted kernel with Nsight counters next to the source.

Sources

  • CUDA Best Practices + Programming Guide (hardware constraints)
  • Nsight Compute / cupti metrics as the feedback channel
  • Emerging KernelAgent / AI-kernel papers and vendor blog posts (methodology evolves quickly — verify against current toolkit docs)