FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Machine learning's explosive growth has produced a proliferation of accelerator architectures, yet no framework maps a workload's computational character to the hardware that serves it best. We build that framework: decomposing ML workloads — transformers, CNNs, GNNs, diffusion and state-space models — into core compute primitives characterized by arithmetic intensity, memory-access pattern, and parallelism, and evaluating how GPUs, TPUs, systolic arrays, dataflow processors, wafer-scale engines, and neuromorphic chips serve them. Using hierarchical roofline and utilization modelling, we show the binding constraint has shifted from peak FLOPS to memory bandwidth and data-movement energy — which quantization only sharpens — and consolidate eight persistent, unsolved problems. These motivate the central contribution: the Fused Memory-Compute Tile (FMCT, codename Habanero), an SRAM-first, HBM-free accelerator combining a three-mode (GEMM / bandwidth / fused) phase-adaptive tile switched by compiler directive, a non-linear unit fused into the systolic datapath, compute-in-memory as a selectable mode rather than the whole architecture, and an inline KV-cache compression engine. A validated component-level energy model shows ~4-7× batch-1 efficiency gains over HBM GPUs. We position FMCT against its closest SRAM-first, no-HBM analogs — d-Matrix Corsair and Tenstorrent — and OpenAI's HBM-retaining Jalapeño ASIC, closing with principles for memory-centric, workload-adaptive accelerators.