Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

Summary

The manuscript presents an up-to-date overview of approaches for optimizing the KV-Cache, which is a method for speeding up, and reducing the memory load for LLMs.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

A good and comprehensive overview to one of the most important technical aspects of modern LLMs. The introduction is reasonably detailed, and the review is well-structured. Also includes helpful diagrams. This review can also be used for introducing new researchers to this topic. Notably, this is a very technical and complicated topic, and covering it within just 9 pages is a feat of compression (pun intended), but the authors also managed to include an evaluation section and neatly describe the logic of KV-Cache evaluations. Well done.

Reasons to reject

The papers is in some places too condensed.

Questions to authors

Some acronyms are not explained: GQA (page 3), RAG (page 4), please consider explaining. Page 4: "but each time it only increases a little, which is a horrible property" - the 'horrible' epithet is not helpful. The case is not 'horrible' but surely can be made more efficient. Page 7, a typo: "obtains the ability to accumulates history" -- should be 'accumulate' Appendix A: "Mixture of Experts model" - please explain what is that. Appendix C 'Other methods' deserves to be a section in the body of the paper, and deserves more elaboration. If you get more pages for the paper, consider moving this section. The following papers in the References section have no venue/publication data: Arora et al., Beltagy et al., Han et al. Please check completeness of the references.

Ethics concerns details

N/A

Summary

This paper performs a literature review on optimizing KV-cache usage in LLMs. The paper is well written and ontology on classifying the current research and engineering effort is always welcome. While the topic is an important and active research area, I feel that the paper could have covered a rather deep review and discussion on the topic. The authors didn't discuss financial implications of different approaches while the title says 'Keep the Cost Down'.

Rating

3

Confidence

3

Ethics flag

1

Reasons to accept

The paper focuses important topic that is active and involving. Author attempt to cover approaches more broadly rather than depth.

Reasons to reject

In depth, discussion and review is missing since it is a survey paper. Work could be supplemented by experimental benchmarks.

Summary

This paper provides a review of KV-cache optimization for LLMs. The reviews are grouped according to the stage when cache optimization is performed: training, deployment or post-training. Training-stage optimization mainly changes the model's architecture for efficient caching, deployment-stage optimization mainly studies better inference system, and post-training optimization mainly perform token eviction or quantization. Moreover, related datasets and evaluation metrics are also discussed.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- Techniques on KV-cache compression are crucial for efficient deployment of LLMs. - A nice review is provided by this work, which may inspire more future research on this direction. - This work provides a good categorization of approaches on KV-cache compression, classified into training-stage, deployment-stage and post-training stage, which clearly describes how and when we can optimize the KV caches.

Reasons to reject

- This paper is mainly a literature review without empirical experiments exploring which methods are most effective in which scenarios. - It would be nice to provide a table listing the advantages and disadvantages of different methods with suggestions on their best usage. - Prompt compression techniques are quite related to this topic and related work should be discussed.

Summary

This submission is a review of the compression methods of KV Cache. It organizes existing works in chronological order, including the training phase, deployment phase, and run-time phase with post-training compression. The training stage mainly involves some novel architecture designs, such as Multi-Query Attention (MQA), and Grouped-Query Attention (GQA). For the deployment-related techniques, this paper covers some system optimization like Paged Attention and DistKD-LLM. For the post-training optimization, several training-free methods are discussed, including Window Attention and Attention Sink. Besides, some pruning, and quantization methods are mentioned. Further, this paper summarises the datasets and evaluation metrics for the KV cache.

Rating

5

Confidence

3

Ethics flag

1

Reasons to accept

1. The review's structure is clear, covering multiple stages including pre-training, deployment, and run-time optimization, facilitating a comprehensive understanding. 2. Both efficiency and model capacity, crucial aspects for KVCache, are thoroughly discussed, demonstrating a holistic approach to optimization. 3. The paper provides four key takeaways for KV Cache optimization, Principles of KV-Cache Optimization, Trade-offs in Deletion vs. Compression, Extremes in KV-Cache Management and Future Directions in Storage and Retrieval Technologies. These insights offer valuable guidance for future research, highlighting important areas for further exploration and advancement.

Reasons to reject

1. The primary reason for rejection is the lack of connections between different methods within the paper. While the taxonomy is structured around different stages, such as pre-training, deployment, and run-time optimization, there is a missed opportunity to highlight similarities and shared concepts among various methods. For instance, the Attention Sink and Repetitive Attention Pattern both emphasize the importance of focusing on specific tokens, yet this connection is not adequately explored or summarized in the review. As a review paper, it's crucial to provide a synthesis of such related ideas to enhance clarity and coherence. 2. Despite claiming to introduce metrics for assessing LLMs' performance on long texts, the evaluation section falls short by only listing general and widely used metrics like memory usage, throughput, latency, and perplexity. The paper lacks a comprehensive assessment of different methods, failing to delve into specific metrics tailored to the nuances of long text processing. A more thorough evaluation framework is needed to effectively gauge the effectiveness of the reviewed methods and provide valuable insights for future research and development in this domain.

Questions to authors

Please refer to the weaknesses. My main concern lies in the lack of in-depth analysis of similar methods.

Summary

This paper presents a review of KV-caching techniques for Decoder-only LLMs, describing methods and strategies pertaining to multiple steps along the life of a model, as well as briefly discussing evaluation metrics and proposing avenues for future work. Overall, the work is timely and necessary, and there will definitely be benefits for researchers new to this topic who choose to read it instead of (or before) diving into the growing literature.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

* The topic is relevant and in need of a review. * The paper can be of use to both practitioners wondering which strategy to pursue in the model they're training and for newcomers who want to design the next shiny KV caching technique. * The method-focused core of the review (sections 3--5) is well-structured and well-balanced among the importance and "meat" in the various sub-sub-topics. The introduction is also good.

Reasons to reject

* The non-core sections are substantially weaker. "Evaluation" (S.6) is mostly unnecessary and the space used for it could have been better spent on covering more methods, extending the descriptions for some, or integrating interesting bits from Appendices, mostly C. Not only is S6 describing at length well-known resources, it does not connect back to any of the methods described in the core sections, and the results table isn't referred to, let alone analyzed. This is followed by a "take-aways" section (S7) which does not follow comfortably from the preceding ones. Overall, there's a noticeable disconnect at the 5-6 section border. * Section 2 (Preliminaries and Notation(s)) is a good idea but not executed carefully enough (see below). * While some illustrations (Fig. 2,3) are useful and illustratory, Figure 1 is mostly confusing and inconsistent. It does not lend intuition of the topic or set expectations for the survey's contents.

Questions to authors

### Questions * In S2, does $p_i$ denote a distribution, as suggested twice, or a single probability, as suggested once and inferred by convention? * In S3, how do $n_h$ combination weights result in $n_h$ combinations? It feels like an entire dimension is absent in the former. * In S3, setting $n_g$ to $n_h$ in the $\eta$ equation results in 1.5, which cannot be true (it needs to be 1). What's the correct formula? ### Required fixes * Please refrain from personifying models - "understanding" in line 4 of the intro should be changed. Especially since this is a didactic work which may be used in introductory classes. ### Grammar and Suggestions * There are some foci of grammatical errors - four in the last three lines of the first intro paragraph, the caption for figure 2, "utilize(s)"in p.5, "quer(y/ies)" in p.6. The last sentence of the intro is not understandable. * Appendices A and B look useful, you should probably alert the reader to their existence in the main text. * The transpose notation in eq. (1) and (2) is not the standard "top" but a cross, please change. * Please don't call things "horrible" (top of S4). "undesirable" would be more suitable. * In p.5, *Static policies* paragraph, it seems like Xiao et al. and Han et al. *find* rather than *suggest* (unless they really only suggest, in which case somebody better run that analysis). * Last but not least. "(the) KV cache" is a noun phrase and shouldn't be hyphenated unless used as a modifier.