H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

Hyde-IKV is a dynamic management system for the Key-Value (KV) cache in Small Language Models (SLMs). It tackles the memory bottleneck of long-context inference by intelligently prioritizing and compressing the stored context. Instead of abruptly deleting tokens, Hyde-IKV uses a hybrid strategy of positional decay (gradually reducing the influence of older tokens) and importance-based modulation (preserving tokens with high attention scores or large key vector magnitudes). This non-invasive approach, implemented via hooks into the transformer architecture, significantly reduces GPU memory usage and speeds up inference for long sequences while largely preserving the quality of the generated text.