Challenges and Research Directions for Large Language Model Inference Hardware

We highlight four promising research opportunities to improve large language model inference for datacenter AI: high bandwidth flash for 10X memory capacity with HBM-like bandwidth, processing-near-memory and 3D memory-logic stacking for high memory bandwidth, and low-latency interconnect to speedup communication. We also review their applicability for mobile devices.