MiniCache: KV Cache Compression in Depth Dimension for Large Language Models

Summary

This paper proposes a KV cache compression method by merging keys and values of consecutive layers. Based on the empirical observation that keys and values of consecutive layers after the mid-depth layer have high cosine similarity, the authors propose a merging strategy using angular interpolation. Additionally, they emphasize the importance of retention tokens based on their empirical findings. Extensive experiments across multiple state-of-the-art models and various evaluation benchmarks demonstrate the effectiveness of the proposed compression method.

Strengths

- The proposed approach of merging keys and values can have a significant practical impact. - The method has low computational overhead while providing meaningful efficiency improvements. - The proposed approach is training-free. - Experiments conducted on various models and benchmarks demonstrate the effectiveness of the proposed method. - The writing is overall clear.

Weaknesses

- Some methodological design choices seem ad-hoc. For example, the method starts merging after the mid-layer and only merges two consecutive layers. I believe the paper could be technically improved by addressing these design choices. - The authors compare the efficiency of the method after applying quantization (MiniCache 4-bit) to the FP16 baseline. This comparison overstates the method's effectiveness because the contribution of the paper does not involve quantization. The paper should measure efficiency improvement without using quantization. Currently, lines L22-24, L302-303, etc., can mislead readers. - A comparison to early layer-exiting approaches seems more relevant than comparisons to KV quantization methods, which the current draft focuses on. - e.g., "Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding", EMNLP, 2023 **minor comments** - Figure 1, index -> layer index - Line 140, Figure 1 (b) -> Figure 1 (a) - For related works, it would be informative to include token compression methods: - "Compressed Context Memory for Online Language Model Interaction", ICLR 2024 - "Recurrent memory transformer", NeurIPS 2022

Questions

- What will be the results when applying the merging function to more than two layers? - How about applying the merging function based on cosine similarity? For instance, only apply merging when the cosine similarity is higher than a specified threshold. This might address the issues mentioned in the first weakness above. - Could you compare the proposed method with early layer-exiting methods? - In Eq (4), is the retention index defined per layer? If not, how do you measure d_i, d_{min}, and d_{max}? - Could you provide an intuitive analysis of why some tokens have severely low cosine similarity in Figure 2 (b)? Are there any common characteristics of these tokens? - Do you apply merging per attention head or to the entire key/value vectors?

Rating

6

Confidence

5

Soundness

2

Presentation

2

Contribution

3

Limitations

In appendix.

Summary

Authors proposed a KVCache Compression Method, Minicache, is introduced as a method to efficiently compress the Key-Value (KV) cache in large language models (LLMs) by leveraging the high similarity of KV states between adjacent layers in the middle-to-deep portion of LLMs. This compression is achieved by disentangling states into magnitude and direction components and interpolating directions while retaining distinct state pairs to minimize storage overhead. MiniCache is training-free, complements existing compression strategies, and demonstrates significant performance improvements, including up to a 5.02x compression ratio, a 5x enhancement in inference throughput, and a 41% reduction in memory footprint, all while maintaining near-lossless performance across various models and benchmarks.

Strengths

1.[Novel Approach, Interesting Observation] The authors explore KV cache compression from a novel perspective by focusing on the depth dimension, which is previously unexplored. They identify a significant finding: KV cache states exhibit high similarity between adjacent layers in the middle-to-later stages of LLMs. This discovery is backed by a thorough analysis and visualization of layer-wise similarity statistics, leading to a well-motivated methodology for compressing the KV cache. This approach has substantial implications for the development of KV cache compression techniques. 2.[Sound Methodology] The authors propose a training-free strategy for cross-layer merging, formulating a reparametrization compression technique to enhance inference efficiency. This method carefully considers outliers, retaining these specific tokens and restoring them through a well-constructed strategy. 3.[Strong Experiments, Well Compatibility] Extensive experiments across multiple datasets with four different models demonstrate that the proposed MiniCache consistently performs effective compression without compromising performance, achieving a superior compression ratio of ~5× and outperforming existing methods. Furthermore, the proposed method is compatible with existing quantization strategies, highlighting its practical values. 4.The paper is well-written, which is easy to understand.

Weaknesses

1.I am concerned about the computational overhead during the reparametrization and restoration stages. Does this compression strategy increase computational overhead? 2.The authors selected the middle layers for cross-layer merging compression. Have the authors considered any principled methods to further increase the compression ratio by compressing the shallow layers in future work?

Questions

See weaknesses.

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors address the limitations inherited from the SLERP method and propose future research directions to enhance the adaptability of the merge compression strategy.

Summary

The paper proposes a novel method called MiniCache for compressing the KV cache in large language models (LLMs) by merging cache states across layers. The authors argue that this approach significantly reduces the memory footprint and enhances inference throughput without significant performance loss. The paper includes evaluations on various models and benchmarks, demonstrating the effectiveness of MiniCache in terms of compression ratio and throughput improvement.

Strengths

1. The idea of compressing the KV cache by merging states across layers introduces a new perspective on reducing memory usage during inference. This approach could inspire further research in inter-layer redundancy exploitation. 2. Memory consumption and inference speed are critical challenges in deploying LLMs. The proposed MiniCache method addresses these issues directly, offering a potentially valuable tool for practitioners working with resource-constrained environments. 3. The method does not require additional training, making it easy to integrate into existing inference pipelines without the need for extensive retraining or fine-tuning of models.

Weaknesses

**Major Weekness 1: Lack of Baseline Comparisons** In Table 1, the authors only include quantization methods for performance comparison, neglecting other KV cache eviction methods such as those proposed in [1] and [2]. **Major Weekness 2: Lack of Evaluation for Instruction-following Benchmarks** Given that instruction-tuned models are more generalizable for downstream applications, it is essential to evaluate how MiniCache performs on instruction-following benchmarks such as MT-Bench [3]. [1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS 2023 [2] Efficient Streaming Language Models with Attention Sinks, ICLR 2024 [3] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023

Questions

1. Beyond memory compression, what is the impact of MiniCache on latency?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

As mentioned above.

Summary

This paper introduces MiniCache, a novel approach to compressing the Key-Value (KV) cache in large language models (LLMs) to enhance inference efficiency. The KV cache is crucial in storing key-value states of previously generated tokens, significantly reducing redundant computations and lowering latency during autoregressive generation. However, as the sequence length increases, the KV cache's size also grows linearly, leading to substantial memory consumption. MiniCache addresses this issue by compressing the KV cache across layers from a depth perspective, leveraging the observation that KV cache states exhibit high similarity between adjacent layers in the middle-to-deep portions of LLMs. The proposed method involves disentangling the states into magnitude and direction components, interpolating the directions while preserving the lengths, and retaining highly distinct state pairs unmerged to minimize information loss. MiniCache is a training-free, general approach that complements existing KV cache compression strategies like quantization and sparsity. The authors conducted comprehensive evaluations using various models, including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral, across multiple benchmarks. The results demonstrated that MiniCache achieves superior compression ratios and high throughput, with LLaMA-2-7B showing a compression ratio of up to 5.02×, a 5× increase in inference throughput, and a 41% reduction in memory footprint compared to the FP16 full cache baseline, all while maintaining near-lossless performance. The paper highlights the potential of MiniCache to significantly reduce memory requirements and enhance the efficiency of LLM inference, making it a promising solution for applications requiring long context inputs and extensive sequence generation.

Strengths

The paper presents a substantive contribution to the field of efficient machine learning by introducing a novel, high-quality, and clearly explained method for KV cache compression in large language models. The significance of the work is underscored by its potential to improve the practicality and scalability of LLMs, addressing a critical bottleneck in their deployment. The originality of the approach, combined with comprehensive evaluations and clear exposition, makes this paper a valuable addition to the literature on efficient ML techniques. **Originality** - **Novel Compression Approach**: The paper introduces a unique method for KV cache compression in large language models by exploring the depth dimension, which is a previously overlooked area. This novel perspective of compressing across layers rather than within layers demonstrates a creative combination of existing ideas in a new, impactful way. - **Reparameterization Strategy**: The method's use of reparameterization to disentangle state vectors into magnitude and direction components for interpolation is innovative. This approach preserves important information while effectively reducing memory usage. **Quality** - **Comprehensive Evaluation**: The authors provide a thorough evaluation of MiniCache across various models and benchmarks, including LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral. The extensive experiments validate the method's effectiveness and robustness. - **Performance Metrics**: The results show significant improvements in compression ratios, inference throughput, and memory footprint reduction, with metrics such as a 5.02× compression ratio and a 41% reduction in memory usage while maintaining near-lossless performance. These metrics highlight the quality and practicality of the proposed solution. **Clarity** - **Detailed Exposition**: The paper is well-written, with a clear and detailed exposition of the methodology. Figures and tables are effectively used to illustrate key concepts, observations, and results, aiding in the understanding of the approach and its benefits. - **Step-by-Step Explanation**: The authors provide a step-by-step explanation of the MiniCache method, from the initial observations to the final implementation, making the paper accessible even to those less familiar with the intricacies of KV cache compression. **Significance** - **Addressing a Critical Issue**: The paper addresses a significant challenge in the deployment of large language models – the growing memory consumption of KV caches with increasing sequence lengths. By reducing the memory footprint and improving inference efficiency, MiniCache has the potential to make LLMs more practical and scalable in real-world applications. - **Broad Applicability**: The approach is general and training-free, making it applicable to a wide range of models and scenarios. This broad applicability enhances the significance of the work, as it can be integrated into existing systems with minimal modification.

Weaknesses

1. **No Implementation Source Code Provided**: The paper does not include the implementation source code, which is a significant limitation. Releasing the source code would facilitate further research and enable other researchers to replicate and build upon the work. Providing the code upon publication would enhance the paper's impact and encourage broader adoption of the proposed method. 2. **Insufficient Justification for SLERP**: The introduction of Spherical Linear Interpolation (SLERP) in the paper feels abrupt and lacks sufficient justification. While SLERP is used for interpolating between vectors, the paper does not provide enough rationale for why this specific technique was chosen over other interpolation methods. More ablation studies should be conducted to demonstrate the effectiveness and necessity of using SLERP in this context. These studies could compare SLERP with alternative interpolation techniques to show its advantages and validate the authors' choice. ### Minor Writing Improvements 1. **Figure Clarity**: - **Figure 1(a)**: The resolution of Figure 1(a) could be higher, or the figure could be replaced with a vector graphic to improve clarity and readability. Enhancing the visual quality would make the figure easier to understand and more professional. 2. **Typographical Corrections**: - **Line 308**: The word "interpretation" should be corrected to "interpolation". - **Line 328**: The phrase "A larger t" should be corrected to "A larger \(\gamma\)".

Questions

1. **Distance-Based Threshold for Retention (Line 226)** Why did you choose a distance-based threshold instead of the merging ratio of overall tokens as the control for retention? Can you show the effect on accuracy and efficiency as the ratio of retention tokens varies, highlighting this trade-off? 2. **Ratio of Merged Tokens in Ablation Study** In the second ablation study, can you show the ratio of merged tokens and the efficiency trade-offs for different settings? 3. **Implementation Source Code** Can you include the implementation source code? 4. **Justification for SLERP** Can you justify your choice of SLERP over other interpolation methods?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

There should be a section discussing the limitation of this work and its social impact.

Thank you for the response

I appreciate the authors' responses, and most of my concerns have been addressed. I will keep my evaluation for acceptance