VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration

Response to Reviewer X8F2

We thank the reviewer for carefully reviewing our paper, recognizing our paper as well-written and easy-to-follow, our method is well-motivated by sparsity analysis and achieves significant decoding speedups. We also appreciate the insightful questions and constructive feedbacks from the reviewer and have been working diligently over the last week. We address them as follows. ### On Question 1 We apologize for the confusion from our notations. Specifically, $\Gamma$ is a 2D array storing the sparsity at each head and layer, while $Z$ is the sum of non-sparse ratios across layers as the normalizer when computing cache budget. We have added in-line annotations to Algorithm 1 and references from the main text to the algorithm to improve the clarity of our presentation. ### On Question 2 These are great questions. Our method does reduce the overall memory usage as we can evict 90% of KV cache tokens while maintaining great performance. Our method does not reduce peak memory usage during prefill because we propose to evict tokens after prefill is finished so that we can allocate an appropriate cache budget based on each request, whose effectiveness is demonstrated by the consistently strong performance across models and tasks. While our method can not use a higher static batch size at the moment, it can effectively increase the batch size when continuous batching, a widely used technique in LLM serving, is utilized. Intuitively, what continuous batching does is to start processing a new request whenever a request in the current batch finishes. With KV cache compression, the GPU memory usage is reduced significantly once decoding starts. Thus, while the model is decoding current requests with much fewer KV cache, new requests can be processed, which results in higher batch size and boosted throughput. Given these reasons, we think “KV Cache Compression” still well characterize our work. ### On Question 3 The reviewer brought up an excellent point. We are currently waiting to perform the benchmark again due to resource constraints, and plan to provide the exact prefill and decoding times by Monday. We appreciate the understanding of the reviewer. Meanwhile, we would like to first point the reviewer to the End-to-End Speedup column of Table 2, where we showed that significant decoding speedups lead to end-to-end (prefill + decoding) speedups across prompt lengths. ### On Question 4 We thank the reviewer for bringing this highly relevant work and have provided discussions and results in the overall reply to all reviewers.

Response to Reviewer X8F2

### On Question 5 We thank the reviewer for pointing us to these two interesting works that we previously overlooked, both of which focus on evicting visual tokens based on vision encoder’s attention patterns before they are used by the LLM. We show comparison with both methods below and included them in Appendix A.2 of the revised paper as well. While HiRED outperforms our method on CoCo-Caption for LLaVA-Mistral-7B, it underperforms on all other cases, especially on DocVQA. This large performance fluctuation implies that HiRED’s strategy of dropping visual tokens before LLM without considering the language query could lead to loss of important information. Our method consistently outperforms FastV since it only focuses on hidden states pruning and ignores the huge memory consumption of KV cache. As a result, under the same memory constraint, our method naturally retains accuracy better. Dataset | Model | Method | 1% | 5% | 10% | 20% | 40% | 60% | 80% | 100% --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- Coco-Caption (Metric: CIDEr) | LLaVA-Mistral-7B | VL-Cache | 2.64 | **82.53** | 100.36 | 102.06 | 99.93 | 101.07 | 100.08 | 100.68 Coco-Caption (Metric: CIDEr) | LLaVA-Mistral-7B | FastV | 9.98 | 41.17 | 88.82 | 97.47 | 105.40 | 110.60 | **106.90** | 100.68 Coco-Caption (Metric: CIDEr) | LLaVA-Mistral-7B | HiRED | **13.87** | 70.90 | **115.37** | **109.91** | **115.09** | **115.09** | 105.93 | 100.68 Coco-Caption (Metric: CIDEr) | LLaVA-1.6-34B | VL-Cache | 0.00 | **120.1**1 | **137.35** | **139.42** | 138.58 | **139.19** | **138.01** | 135.07 Coco-Caption (Metric: CIDEr) | LLaVA-1.6-34B | FastV | 3.10 | 15.86 | 32.55 | 80.04 | 112.39 | 116.53 | 120.70 | 135.07 Coco-Caption (Metric: CIDEr) | LLaVA-1.6-34B | HiRED | **39.04** | 114.09 | 136.31 | 134.85 | **142.37** | 135.14 | 134.79 | 135.07 DocVQA (Metric: ANLS) | LLaVA-Mistral-7B | VL-Cache | **43** | **59** | **62** | **64** | **67** | **67** | **67** | 68 DocVQA (Metric: ANLS) | LLaVA-Mistral-7B | FastV | 18 | 26 | 38 | 49 | 63 | 64 | **67** | 68 DocVQA (Metric: ANLS) | LLaVA-Mistral-7B | HiRED | 18 | 38 | 51 | 53 | 63 | 63 | 65 | 68 DocVQA (Metric: ANLS) | LLaVA-1.6-34B | VL-Cache | **41** | **82** | **84** | **85** | **85** | **85** | **85** | 85 DocVQA (Metric: ANLS)| LLaVA-1.6-34B | FastV | 2 | 1 | 3 | 9 | 28 | 42 | 47 | 85 DocVQA (Metric: ANLS)| LLaVA-1.6-34B | HiRED | 25 | 44 | 52 | 71 | 76 | 81 | 84 | 85 MathVista (Metric: ACC) | LLaVA-Mistral-7B | VL-Cache | **38** | **36** | **39** | **40** | **40** | **42** | **42** | 41 MathVista (Metric: ACC) | LLaVA-Mistral-7B | FastV | 25 | 29 | 37 | 38 | 39 | 40 | 41 | 41 MathVista (Metric: ACC) | LLaVA-Mistral-7B | HiRED | 32 | 35 | 35 | 36 | **40** | 40 | 40 | 41 MathVista (Metric: ACC) | LLaVA-1.6-34B | VL-Cache | **41** | **42** | **42** | **44** | **45** | **45** | **42** | 43 MathVista (Metric: ACC) | LLaVA-1.6-34B | FastV | 27 | 27 | 32 | 37 | 37 | 35 | 37 | 43 MathVista (Metric: ACC) | LLaVA-1.6-34B | HiRED | 32 | 41 | **42** | 42 | 39 | 39 | 42 | 43 ### On Question 6 We agree that video benchmarks would be a great addition to showcase the effectiveness of our method. However, the evaluation on Video-MME is taking extended time and we will provide an update by Monday. We appreciate the patience and understanding of the reviewer as we try our best to prepare the result.

Official Comment by Reviewer X8F2

Thank you to the authors for their response. I look forward to the additional results and the feedback from the other reviewers.

Response to Reviewer KiPr

We thank the reviewer for the thoughtful review and for the recognition that our method is well-motivated, provides good insights into the optimization of VLM cache compression, and achieves significant inference speedups and GPU memory savings. We would like to address the concerns and questions raised by the reviewer as follows: ### On Weakness 1 The way we collect the attention score matrix is actually quite straightforward. Let $Q, K \in R^{n \times d}$ be the query and key matrices respectively, where n includes both prompt and decoded tokens, the attention score matrix $A \in R^{n \times n}$ is computed as $softmax(QK^{T} / \sqrt{d})$. We note that the decoded tokens are generated by the original VLM without any KV cache compression, so the attention pattern during decoding stage faithfully reflects the true importance of prompt tokens. ### On Weakness 2 We thank the reviewer for this excellent suggestion. We have added a formal definition of post vision attention in its first occurrence to improve the readability of our paper. For reference, we formally define $\textbf{post-vision attention}$ as the sub attention score matrix that's sliced along the query dimension to only include language prompt tokens after vision tokens, as illustrated in Figure 1(b) and the updated Figure 4. ### On Weakness 3 We thank the reviewer for the great suggestion. We have conducted ablation studies on the impact of the hyperparameter $p$ for Llava-Mistral-7B on the CoCo-Caption dataset, where we vary thresholds $p$ from 0.0001 to 0.1. We found that our method is robust to $p$ in general, and more so when the token budgets are higher. We have also added this analysis to Appendix A.2 of our paper. | Token Budget (%) | $p$ = 0.0001 | $p$ = 0.001 | $p$ = 0.01 | $p$ = 0.1 | |------------------|------------|-----------|----------|----------| | 10 | 101.36 | 97.78 | 98.51 | 98.77 | | 20 | 98.60 | 97.06 | 99.87 | 101.24 | | 40 | 99.45 | 99.6008 | 99.5106 | 99.12 | | 60 | 101.04 | 100.90 | 100.71 | 100.10 | | 80 | 103.02 | 101.76 | 101.08 | 101.76 | ### On Weakness 4 The reviewer raises an insightful point that dynamically allocating cache budget based on each prompt can be computationally expensive. Fortunately, we were able to implement a highly efficient Triton-kernel that we described in detail in Appendix A.5 of our paper. We have also showed in the column “Prefill Speedup” of Table 2 in our paper that the incurred overhead is only around 1% to 4% of prefill time. We hope this information from the paper could relieve the reviewer’s concern over the computational overhead of our method. ### On weakness 5 We agree with the reviewer that we did not emphasize enough how Algorithm 1 incorporates modality properties. We would like to clarify that the `ComputeSparsity` helper method, which we have renamed to `ComputePostVisionSparsity` in our updated manuscript, leverages the post vision attention matrix, instead of the full attention matrix, to compute the attention head sparsity. Previously, we showed that post vision attention patterns are much more similar to decoding attention patterns. Thus, using post vision attention patterns to compute sparsity also better captures the necessary cache budget required to preserve performance. ### On weakness 6 We thank the reviewer for this suggestion. We would like to request more context regarding the insights the reviewer aims to extract with this I/O traffic information. In case this is helpful, attention kernels load KV context once, so reduction in the number of KV tokens translates to reduction in IO traffic between HBM and Shared Memory of GPU. Since typical VLM & LLM inference engine store and retrieve the KV cache for the entire prompt in HBM, our method significantly reduces KV cache I/O.

Response to Reviewer XVzL

We thank the reviewer for their recognition of our work with the comment “the insight that the attention patterns of generated tokens closely align with language tokens is valuable”. We understand their concerns and address them as follows. ### On Weakness 1 To better contextualize the procedures in Algorithm 1 to our core methodology, we made two improvements: 1. we relate the algorithm described in the main context to specific lines of Algorithm 1, and 2) we added comments in Algorithm 1 to explicitly note procedure `SkewedCacheBudgetAllocation` as the main method to compute layer-wise budget and procedure `ComputeSparsity`, which we renamed to `ComputePostVisionSparsity` in our edited version for clarity, as the helper method that utilizes post-vision attention to compute sparsity. ### On Weakness 2 We echo the reviewer’s point that Cache Hit Rate should be clearly defined and explained. We use $\psi(S)$ to indicate the token scores assigned to $S$ with a policy $\psi$, and $S_{\psi, k}$ (changed from $S_\psi$ for clarity) to indicate the top-k indices selected by $\psi$. We have added these explanations in the main text. Additionally, we recognized that we defined two relevant concepts, $S$, the indices of cache tokens, and $\psi$, the scoring policy that maps token indices to scalar values as their importance, much earlier in Section 2.2, which might have added burdens on readers. Thus, we briefly re-state their definitions and refer the readers to Section 2.2 for further details before Definition 3.1. We hope these adjustments improve the readability of our paper. ### On Weakness 3 We thank the reviewer for the constructive feedback. We have made major changes to Figure 4 to better illustrate our methodology and included in our updated paper. During prefill, post-vision attention matrices are utilized to compute sparsity-driven layer-wise budgets and rank the importance of cache tokens. Unimportant tokens are then evicted, allowing lower memory usage and accelerated decoding with reduced KV cache. ### On Weakness 4 We thank the reviewer for bringing this highly relevant work to our attention and have provided comparisons with ZipCache’s scoring policy in our overall reply to all reviewers. ### On Weakness 5 The reviewer has raised an insightful point that we did not compress KV cache for decoded tokens . While we plan to tackle this issue initially, an analysis on input and output lengths of current vision-language tasks reveals that the ratio prompt tokens to output tokens is quite high (median ~320x across datasets). As a result, compressing KV cache of newly generated tokens provides limited memory savings. By focusing on compressing prefill KV cache, we addressed the most pressing memory bottleneck for VLM inference tasks. Still, we anticipate the need for compressing decoding KV cache will arise when more diverse use cases emerge for VLMs. Therefore, we have listed this limitation of our current method in the Conclusion section and mentioned a periodical compression strategy for newly generated tokens to strike a balance between latency and memory saving. ### On Weakness 6 We respectfully disagree with the reviewer’s claim that the VL-Cache does not significantly outperform other methods. With 10% KV cache, VL-Cache achieves 91% - 100% of full cache performance on most datasets and models and significantly outperformed other methods. In particular, under the same cache budget, H2O only achieves 43% - 93% of full cache performance. Since all KV cache compression methods will converge to full cache performance as the more cache budget is granted, we believe strong performance at low cache budgets is a clear testament of the effectiveness of VL-Cache in retaining important KV tokens. Meanwhile, we recognize the challenges of interpreting the table as we included results for various cache budgets. Therefore, we have added more information in the table’s caption to provide guidance.

Response to Reviewer XU5m

We thank the reviewer for recognizing that our proposed method is well motivated and presents significantly better performance than existing methods. For concerns regarding the presentation of the paper, we address them as below. ### On Weakness 1 We sincerely appreciate the reviewer’s time and efforts in evaluating our paper. To help readers understand our method more easily, we have significantly revised and updated Figure 4, which provides an overview of our proposed method. We have also added more transitions between sections to improve the flow of the paper. ### On Weakness 2 These are great observations. On MathVista, multiple KV cache compression methods, including ours, perform well even with only 1% KV cache. One reason is that 54% of MathVista are multiple-choice questions. Selecting the correct answer is intuitively easier than generating it. This might explain the success of KV cache pruning methods even under extremely low cache budget. The other reason is that both H2O and our method essentially use full cache to generate the first token: H2O evicts tokens after prefill hidden states from a layer is computed, our method evicts tokens until prefill hidden states from the last layer is computed. Thus, in the extreme case of only one token is generated, both will recover full cache performance. Regarding why sometimes using partial KV cache outperforms full cache, H2O paper also observed this phenomenon, attributing it to the regularization effect of KV cache pruning. ### On Weakness 3 The reviewer raised a valid concern over the presentation of our method. With the current structure, we intended to use Section 3 to motivate our method in Section 4 so that readers can understand where we come from more easily. However, we do realize this could distract the readers, so we have trimmed Section 3 so that our method is introduced sooner. We have also made several improvements to enhance the presentation of Section 4, such as improving Figure 4 and adding detailed explanations to our proposed algorithm.