FlashAttention-1 (2022): Proved that fusing attention and computing online softmax in SRAM yields 2-4x wall-clock speedups while cutting memory footprint from \(O(N^2)\) to \(O(N)\).
FlashAttention-2 (2023):
Swapped the outer and inner loops (looping over blocks of \(Q\) in the outer loop, streaming \(K, V\) in the inner loop).
Reduced non-matmul FLOPs by deferring output scaling until the very end.
Introduced sequence parallel work distribution across thread blocks.
FlashAttention-3 (2024):
Architected specifically for NVIDIA Hopper (H100) hardware.