See What You Are Told: Visual Attention Sink in Large Multimodal Models

Response to Reviewer b7X3 (1/3)

We thank the reviewer for their constructive feedback and helpful observations. Our detailed responses to the comments are outlined below. If there are any remaining concerns, please inform us, and we will address them promptly. # W1. Comparison of Visual Attention Sinks and Summary Tokens in OPERA > W1. The phenomenon of visual attention sink appears to be similar to the summary tokens proposed in OPERA[1]. Summary tokens imply that higher attention weights are assigned to tokens without semantic information, such as ',' and '\n'. The visual sink token seems to suggest that within visual tokens, there are also some tokens that play a similar role to summary tokens. Thank you for the insightful and interesting comment. The visual sink tokens and the summary tokens may seem similar in terms of having high attention weights, but they are inherently different in nature. The summary tokens, or the anchor tokens in OPERA [1] was originally suggested by [2]. According to [2], the anchor tokens (1) gather the information of demonstrations to form semantic representations for deeper layers in shallow layers and (2) the information from label words is extracted to form the final prediction in deep layers. Therefore, high attention weights to the summary tokens emerge in the middle to later layers. In contrast, the visual sink tokens are observed from the early layers to the late layers. We also tried to investigate the summary tokens (e.g., ‘,’ and ‘.’) on some samples of the CHAIR dataset in LLaVA-1.5-7B. However, there is no massive activation on sink dimensions $\mathcal{D}_{\textsf{sink}} = \lbrace 1415, 2533 \rbrace$, indicating that the summary tokens and visual sink tokens are different. Furthermore, visual sink tokens are almost useless, as discussed in Sec 4.2 and Sec 4.3, while the summary tokens have a crucial role to transfer the information of the whole sentence. [1] Huang, Qidong, et al. "OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [2] Wang, Lean, et al. "Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Response to Reviewer b7X3 (2/3)

# W2 / Q1. Lack of Experimental Validation for Visual Attention Sink > 1.a. I have previously experimented with removing the <bos> token, and the attention sink phenomenon still persisted in the first token. Therefore, I am curious whether new tokens will become visual sink tokens after redistributing the attention weights. Thank you for the insightful and interesting question. Due to the autoregressive nature of LLMs, the former token does not depend on the latter token. Since the visual tokens appear before the text tokens and VAR only redistributes the attention weights while processing the text tokens, the visual sink tokens are preserved after the attention weights are redistributed. Also, new visual sink tokens do not emerge after redistributing the attention weights. On the other hand, we agree that removing the visual sink tokens and checking whether new visual sink tokens emerge is an interesting direction. Therefore, we have conducted an experiment to remove the visual sink tokens and check whether new visual sink tokens emerge, as the reviewer has experimented with <bos> token. We find that new visual sink tokens do not emerge after removing the visual sink tokens. In fact, except for the first position, the sink tokens in LLMs (e.g., ',' and '\n') also do not persist in the same position when we remove them. Therefore, our observation is coherent with the results in LLMs. > 1.b. The author has visualized tokens with high attention weights in the text, and indeed, for these cases, the visual sink tokens are unrelated to the main subject of the image. However, I believe it is still necessary to prove the trend of visual sink tokens being unrelated to the main subject of the image on a large-scale dataset based on segmentation data. We agree that the validation on the segmentation dataset can show stronger evidence for the trend of visual sink tokens being unrelated to the main subject of the image. We have conducted an additional experiment to validate whether the visual sink tokens are unrelated to the main subject of the image. We use Pascal-VOC and MS-COCO datasets, which are widely used large-scale datasets for object segmentation. We compare the portion of visual sink tokens that are not related to the main subject of the image with the portion of all visual tokens that are not related to the main subject of the image. The results are shown below. | Token Type | Pascal-VOC | MS-COCO | | :--------: | :--------: | :-----: | | Visual Sink Tokens | 90.5% | 93.7% | | All Visual Tokens | 82.9% | 90.5% | The results indicate that visual sink tokens are more likely to be located in the background regions compared to randomly selected visual tokens, supporting the conclusion that visual sink tokens are unrelated to the main subject of the image. We have added these results to the revised manuscript (Sec 4 and Appendix A.2). > 1.c. In Table 4.a, the selection of random tokens does not seem to be related to $\tau$, so I want to confirm whether two masked methods with the same $\tau$ value mean that the same number of tokens are masked. Maybe the reviewer mentioned Figure 4(a) (revised as Figure 3(b)). Yes, the number of tokens to be masked is the same for both methods for fair comparison. We have clarified this in the caption of Figure 3.

Response to Reviewer KaLi (1/2)

We appreciate the reviewer's thoughtful feedback and valuable suggestions. We provide a detailed response to the reviewer's comments below. # W1 / W2 / Q1. Insufficient Discussion on Sink Tokens > W1. Insufficient references. There is a work [1] on sink tokens in vision models that is not cited or discussed. > W2. Insufficient discussion on sink tokens. Sink tokens have been observed in different kinds of models, including language models, vision models, and multi-modal models. What's the relationship between sink tokens in different kinds of models? Are they just similar phenomenons, or are there special properties of multimodal models? Moreover, previous works claim that sink tokens exist because of excessive attention scores, so why does it work to recycle these attention scores? From the experiment perspective, the paper is also encouraged to apply previous methods to multi-modal models. > Q1. As mentioned in the weakness, the paper should discuss the relationship between sink tokens in language models and vision models, from the perspective of token properties, emerging mechanisms and proposed methods. We appreciate the reviewer for pointing out the insufficient discussion on the relationship between sink tokens in different kinds of models. Sink tokens in vision models, language models, and multimodal models are similar phenomenons, but those in multimodal models are slightly different. (**Token Properties**) High attention weights and massive activation of sink tokens are common properties in different models. We also find that visual sink tokens are more likely to be located in the background region (unrelated to the main objects). This characteristic is consistent with the findings in vision models [1] and language models [3]. Specifically, sink tokens in ViT are located in the background regions, and sink tokens in language models have limited semantic meanings. Therefore, we conclude that the tokens with less semantic meanings are more likely to be sink tokens in various models. (**Emerging Mechanisms**) In vision models and language models, sink tokens emerge as a result of massive training data [1, 2]. Since multimodal models are based on the pre-trained language models, visual attention sink phenomenon in multimodal models are more likely to be inherited from language models. The evidence is that sink dimensions in multimodal models are identical to those in base language models, as discussed in Appendix A.1. (**Proposed Methods**) As the reviewer mentioned, sink tokens exist because of excessive attention scores. However, the excessive attention scores are useless for the model's prediction, as we discussed in Sec 4.2 and 4.3. Recycling these attention scores is effective and safe because it compensates for the low attention weights on visual tokens and does not modify the original attention distribution except for the sink tokens. We tried to apply previous methods such as [3] to multimodal models, but there was no improvement in the performance. The Visual + Text setting in Table 5 can be considered as a soft version of [3] but it is not as effective as the proposed method (VAR). Overall, we agree that the paper should discuss the relationship between sink tokens in different kinds of models more thoroughly. We have added [1] to the related work and discussed the relationship between sink tokens in different models in Sec 4.2 and Appendix A.2. If there are any remaining concerns, please inform us, and we will address them promptly. [1] Darcet, Timothée, et al. "Vision Transformers Need Registers." The Twelfth International Conference on Learning Representations. 2024. [2] Gu, Xiangming, et al. "When Attention Sink Emerges in Language Models: An Empirical View." 2024. [3] Yu, Zhongzhi, et al. “Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration." Forty-first International Conference on Machine Learning. 2024.

Response to Reviewer P74f (1/3)

Thank you for your detailed and thoughtful review. First, we apologize for the lack of clarity in the experimental setup details, descriptions, and incomplete results that may have made the reading uncomfortable. The review was very helpful for improving our paper, making many parts more clarified or concrete. We provide a detailed response to the reviewer's comments below. If there are any remaining concerns, please inform us, and we will address them promptly. # Q1. Detailed Clarification of Figure 1 > Q1. In Figure 1, does the attention map show the attention scores between visual tokens and a specified text token, or between visual tokens and the output token? The attention map in Fig 1 shows the attention scores between visual tokens and a specified text token. We have clarified this in the caption of the revised manuscript. # W1 / Q2. Concerns Regarding Interpretation of Figure 4(a) Results > W1. The results in Figure 4(a) are essential for validating the effectiveness of the sink token filtering strategy, yet they are challenging to interpret. If the randomly selected tokens are drawn from all input tokens, comparing them with visual sink tokens becomes less fair, making it difficult to substantiate the filtering's effectiveness. Conversely, if the random tokens are sampled exclusively from visual tokens, the experimental results appear unreasonable. Even without visual input, the model’s accuracy on the POPE benchmark should remain above 50%, yet the figure shows a final result of 0. Furthermore, when $\tau < 10$, the tokens being pruned are predominantly standard visual tokens. Despite this, a significant difference in F1-Score persists between the two pruning strategies, contradicting the results seen when $\tau > 10$. Without a clear explanation of the experimental findings in Figure 4(a), the credibility of the sink token filtering strategy is undermined. > Q2. In the experiment shown in Figure 4(a) on the POPE benchmark, are the "random tokens" randomly selected from only the visual tokens? If not, and they are instead selected from all tokens, wouldn’t this comparison be unfair? If the tokens are indeed chosen only from visual tokens, how is it possible that the model's predictions drop to zero when masking all visual tokens? On the POPE dataset, LLaVA can maintain over 50% accuracy even without image input, so this experimental result is difficult to understand. Thank you for raising these questions. We will address them separately in three parts. Note that Figure 4(a) is changed to Figure 3(b) in the revised manuscript. ## A. Random Token Selection: Visual Tokens or All Tokens? The "random tokens" in the experiment shown in Fig 4(a) are randomly selected from only the visual tokens. As the reviewer pointed out, we agree that masking all input tokens including text tokens is unfair. We have clarified that in the figure, caption and main text of the revised manuscript. ## B. F1 Score of 0 in Figure 4(a) (revised as Fig 3(b)) We acknowledge that an F1 Score of 0 seems contradictory with common sense. The reason for this is the attention masking method we used. There are two mainstream methods for attention masking: before softmax (set the attention mask to -inf) [1, 2, 3] and after softmax (set the attention weight to 0) [4, 5, 6]. Both are widely used in literature. We chose the latter method for convenience. When masking a small number of tokens, this method does not cause significant issues because the sum of attention weights is close to 1. However, when masking a large number of tokens, the sum of attention weights is not preserved, leading to model impairment and outputting [UNK] (Unknown) tokens. Consequently, the model cannot respond, resulting in an F1 Score of 0. Since our intention was to prevent the model from seeing specific tokens, not to impair the model, we re-ran the experiment using the former method, masking before the softmax. The former method can preserve the sum of attention weights, thereby preventing model impairment. We have replaced the results in Fig 4(a) with the new results and described the method in Appendix C.3. With this change, the model maintains about 50% F1 Score.

Response to Authors (2/2)

## **Q3. Stability Characteristics of Visual Sink Tokens** I sincerely appreciate the author’s clarification regarding the scope of attention redistribution and the results presented in Figure 13, which demonstrate that sink tokens are consistently assigned higher attention scores. This has resolved most of my confusion. However, I still find it somewhat surprising if this method is applied to all text tokens. In the proposed method, the visual sink tokens at each layer are dynamic. However, for each output metric, the corresponding sink tokens remain constant. I would like to confirm one last time the conclusion the author is attempting to draw: *“For a single input sample, the sink tokens corresponding to all output tokens remain constant, while they are dynamic across layers.”* Is this correct?

Further Response to Reviewer P74f (1/3)

We sincerely appreciate the reviewer's insightful and intriguing questions. We welcome the opportunity to address these concerns and engage in further discussions about our research. Below, we provide detailed responses to each question. For clarity, we address Q3 first, followed by Q2. # Response to Q3. Stability Characteristics of Visual Sink Tokens > In the proposed method, the visual sink tokens at each layer are dynamic. However, for each output metric, the corresponding sink tokens remain constant. I would like to confirm one last time the conclusion the author is attempting to draw: “*For a single input sample, the sink tokens corresponding to all output tokens remain constant, while they are dynamic across layers.*” Is this correct? Yes, the reviewer's interpretation is indeed accurate. Sink tokens are dynamic across layers but remain constant within a single layer. We would like to clarify one additional point: as shown in Figure 8 and discussed in Appendix A.2, sink tokens emerge in the early layers and typically persist until the final layer. Therefore, the term "dynamic" does not imply frequent changes from layer to layer; rather, sink tokens rarely change once they emerge in the early layers. We hope this clarification resolves the confusion.

Further Response to Reviewer P74f (3/3)

> It’s worth mentioning that I am not familiar with the methods described in [1][2], but I really familiar with [3]. In that study, the authors modified the attention weights before the softmax operation rather than afterward. We appreciate the reviewer for pointing it out. To the best of our knowledge, in [3], the authors set the attention weights to zero *after* the softmax operation in the analysis section (Section 2.2) and propose a attention re-weighting strategy by modifying the attention weights *before* the softmax operation in the application section (Section 3.1). If we have misunderstood the paper, we sincerely apologize for the confusion. [1] Mohebbi, Hosein, et al. "Quantifying Context Mixing in Transformers." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023. [2] Jin, Zhuoran, et al. "Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models." Findings of the Association for Computational Linguistics. 2024. [3] Wang, Lean, et al. "Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. > Additionally, regardless of whether we refer to the updated or original version of the paper, the results in Figure 3 (previously Figure 4) remain puzzling. In Figure 3(a), we clearly observe many tokens with $\tau > 40$, while the range $10 < \tau < 20$ contains relatively few tokens. If, as the author claims, the performance drop in Figure 3(b) occurs because normal visual tokens are pruned when $\tau < 15$, leading to significant degradation, then pruning random visual tokens when $\tau > 40$ should presumably result in even more normal visual tokens being pruned. However, at this stage, the performance degradation is smaller than when pruning visual sink tokens in the range $10 < \tau < 20$. How is this possible? We are truly grateful for this insightful question, and we appreciate the opportunity to address it. The reason for the puzzling results in Figure 3 is that "random visual tokens" experiment is more robust to the token masking process than the "visual sink tokens" experiment. First, we would like to provide a detailed description of the experimental design to explain it. For each layer $\ell$, we select the visual tokens that satisfy the condition $\phi (\boldsymbol{x} ^{\ell-1} _ j) \geq \tau$ and mask them (depicted as "visual sink tokens" in Figure 3(b)). Similarly, we randomly select the same number of visual tokens and mask them (depicted as "random visual tokens" in Figure 3(b)). In the "visual sink tokens" experiment, although we select visual tokens for each layer, the selected tokens are almost consistent across layers because the hidden states do not change significantly from layer to layer (due to the residual connections). In contrast, in the "random visual tokens" experiment, the selected tokens differ for each layer because the selection is random. If the masked token contains important information, the model has almost no chance of acquiring that information in the "visual sink tokens" experiment. However, in the "random visual tokens" experiment, the model can recover the information from other layers. This difference in the robustness of the two experiments causes the "random visual tokens" experiment to maintain higher performance than the "visual sink tokens" experiment, even when a larger number of normal visual tokens are masked. Additionally, since random visual tokens also include visual sink tokens, there is a possibility of selecting visual sink tokens in the "random visual tokens" experiment. Therefore, directly comparing the two graphs at different $\tau$ values may be challenging. Instead, we would like to suggest considering the "random visual tokens" experiment as a reference for the "visual sink tokens" experiment at the same $\tau$ value. We sincerely hope that our responses have helped to clarify any remaining uncertainties. If we have misinterpreted any aspect of the reviewer's question, please notify us, and we will quickly resolve it. Lastly, we would like to express our gratitude to the reviewer for their valuable feedback and time spent reviewing our work.

Further response to reviewer p74f Ⅱ (2/3)

# Response to #2: About proportion of performance degradation > If I have correctly understood the author’s explanation, the claim is that during the pruning of random visual tokens, some visual sink tokens are also included. As a result, when $\phi > 40$, the number of pruned normal visual tokens is not significantly higher than the number pruned in the visual sink tokens experiment when $10 < \phi < 20$. > However, I had already considered the efficiency of pruning when raising this question. Since I lack access to specific data, I could only infer from Figure 3(a) that the ratio of normal visual tokens to visual sink tokens is approximately 4:1, while the ratio of tokens with $10 < \phi < 20$ to those with $\phi > 40$ is at least 5:1. Therefore, in the random visual tokens experiment, when $\phi > 40$, the number of pruned normal visual tokens should be at least **four times** the number pruned in the visual sink tokens experiment when $10 < \phi < 20$. > Yet, in the experiment, both scenarios show the **same proportion of performance degradation**, particularly when $\phi > 45$, where the degradation becomes inexplicably low. This inconsistency is quite puzzling to me. We are sorry for any confusion caused by our explanation. Our primary claim in "Further Response to Reviewer P74f (3/3)" was that masking the same fixed tokens across layers leads to greater performance degradation compared to masking random tokens per layer, even when a larger proportion of normal visual tokens is pruned in the latter case. (We will discuss this in more detail in the next section.) We acknowledge that our additional claim regarding the potential selection of visual sink tokens in the "random visual tokens" experiment is not the primary reason why the performance degradation does not become significantly higher when $\phi (x) > 40$. Indeed, when $\phi (x) > 40$, the number of masked normal visual tokens is approximately **three times** the number of masked visual sink tokens for $10 < \phi (x) < 20$, which aligns closely with the ratio inferred by the reviewer. We would like to explain why the performance degradation does not increase significantly when $\phi (x) > 40$ in the next section.

Further response to reviewer p74f Ⅱ (3/3)

# Response to #3: About fixed position of visual tokens pruning > Additionally, the author’s response raises a new question. The author mentioned that **in deeper layers, visual sink tokens are nearly fixed**, which means pruning visual tokens in these layers is also fixed. However, in the random visual tokens experiment, the pruned visual tokens vary dynamically across layers. Does this discrepancy make the comparison unfair, potentially leading to misleading results? **Pruning tokens from fixed positions in each layer versus dynamically pruning tokens from different positions inherently causes different levels of disruption to the information flow.** > More specifically, if visual tokens are pruned from fixed positions in each layer, wouldn’t the impact on model performance be more limited? > Does this suggest that the observed performance degradation might not stem from differences between sink tokens and normal tokens, **but rather from the distinction between pruning fixed versus dynamic positions across layers**? We appreciate the insightful question about the impact of fixed versus dynamic masking on model performance. We conducted additional experiments to investigate the impact of fixed versus dynamic masking on model performance. Specifically, we compared the performance degradation of fixed masking and dynamic masking. The number of tokens masked in these experiments is the same as the number of tokens masked in the “random visual tokens” experiment ($\phi (x) > 40$ and $\phi (x) > 45$). Thus, the dynamic masking experiment is expected to show performance degradation similar to the our “random visual tokens” experiment in Figure 3(b). Since the discussion period is limited, we evaluated the F1 score in about 10% of the POPE benchmark. The results are shown below: | $\phi (x)$ | Fixed Masking | Dynamic Masking | |--------|---------------|-----------------| | $ >40 $ | 52.0 | 61.3 | | $ >45 $ | 61.1 | 77.6 | The results of the dynamic masking experiment are consistent with the “random visual tokens” experiment in Figure 3(b). These results show that fixed masking has a greater impact on performance degradation than dynamic masking. Therefore, the comparison between the "visual sink tokens" experiment (similar to Fixed Masking) and the "random visual tokens" experiment (similar to Dynamic Masking) is biased in a way that disadvantages the "visual sink tokens" experiment. Even so, the sink token experiment consistently maintains performance compared to the “random visual tokens” experiment for $\tau > 20$. This result supports that the visual sink tokens are meaningless. These results also align with the discussion in #2. Since the number of masked normal visual tokens for $\phi (x) > 40$ is approximately three times the number of masked normal visual tokens for $10 < \phi (x) < 20$, the performance degradation in fixed masking at $\tau = 40$ is higher than that of $\tau = 10$ in the "visual sink tokens" experiment. This holds true for fixed masking at $\tau = 45$ and $\tau = 15$ in the "visual sink tokens" experiment as well. > I conducted a simple experiment myself: **I randomly selected 25% of the visual tokens and pruned these same fixed positions across all layers**. The resulting POPE test score was **82.4**, which seems comparable to the results shown in the visual sink tokens experiment in the figure-3(b). We appreciate the reviewer's effort in conducting the experiment. After performing our own "Fixed Masking" experiments, we are questioning the discrepancy between the reviewer's results and our own. Since we do not have complete details about the reviewer's experiment, we cannot provide a definitive explanation. However, we believe that the discrepancy may stem from the difference between pruning and masking. Pruning involves excluding visual tokens before they enter the layer, while masking is a post-processing step that eliminates their impact after the attention calculation has been completed within the layer. While pruning maintains performance even when a large number of tokens are removed [1], masking leads to performance degradation even when only a small number of tokens are excluded [2, 3]. If we have misunderstood the reviewer's experiment, we sincerely apologize and would appreciate it if the reviewer could provide more details about the experiment. [1] Chen, Liang, et al. "An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models." European Conference on Computer Vision. 2024. [2] Geva, Mor, et al. "Dissecting Recall of Factual Associations in Auto-Regressive Language Models." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. [3] Neo, Clement, et al. "Towards Interpreting Visual Information Processing in Vision-Language Models." 2024.

Final Response to Authors (1/2)

## **1. About the Fixed Position of Visual Tokens Pruning** In my expectation, **fixed masking** should have a smaller impact on the model's performance compared to **dynamic masking**. As a result, I hypothesized that the experimental results for the *visual sink tokens* scenario would show a relatively smaller performance degradation, meaning that the blue line in Figure 3(b) should shift upward. However, the author's experiments suggest the opposite: fixed masking has a more significant negative impact on the model's performance than dynamic masking, which implies that the blue line in Figure 3(b) would shift further downward. Unfortunately, my own experimental results do not support this claim. My experiments indicate that pruning 25% of visual tokens at fixed positions does not lead to significant performance degradation. In my implementation, I modified the `attention_mask` parameter in the `llama_model` module to mask out a portion of the visual tokens. Furthermore, in the author's experimental results, masking only approximately 15% of tokens resulted in a performance drop close to 50%, which is nearly equivalent to random guessing. Could the authors provide more details about their experimental setup?

Response to Reviewer 1YMG (1/2)

We thank the reviewer for the constructive review and insightful comments. We provide a detailed response to the reviewer's comments below. If there are any remaining concerns, please inform us, and we will address them promptly. # W1. Determination of Hyperparameter $\rho$ > W1. The reviewer's biggest concern is about the determination of hyperparameter $\rho$. According to the paper and Fig 7(b), it is benchmark-dependent. Furthermore, a poorly chosen $\rho$ can lead to performance worse than the baseline performance. This parameter greatly limited the applicability and generalizability of the proposed method. (**Benchmark-dependent $\rho$**) We appreciate the reviewer for raising this question. To clarify, we use the same $\rho$ value for all the benchmarks in the same task type (0.8 for general vision-language task, 0.5 for visual hallucination task, and 0.9 for vision-centric task). In the paper, we use the terms 'task' and 'benchmark' differently. A 'task' refers to the type of problem (e.g., general vision-language task, visual hallucination task, and vision-centric task) and a 'benchmark' refers to a specific dataset (e.g., VQAv2, GQA, LLaVA-W, MM-Vet). A single $\rho$ value can improve the performance of various LMMs on all benchmarks for the same task type. The purpose of presenting Fig 7(b) (revised as 6(b)) is to obtain the plausible $\rho$ value for all the benchmarks in the same task type using only a single benchmark per task. We will further explain the rationale and justification for the choices of $\rho$ in W2. (**Worse performance with poorly chosen $\rho$**) We acknowledge that a poorly chosen $\rho$ can lead to performance worse than the baseline performance. However, in the reasonable range of $\rho$ values, the proposed method robustly improves the performance of various LMMs on various benchmarks. For example, if $\rho \geq 0.5$, VAR can consistently improve the performance on MME (general vision-language task), POPE (visual hallucination task), and CV-Bench (vision-centric task). Therefore, we argue that, to some extent, the proposed method remains applicable across various benchmarks within plausible ranges of $\rho$ values. We have discussed the hyperparameter $\rho$ more thoroughly in Sec 6.3. and Appendix B.2. # W2. Tuning of Hyperparameter $\rho$ > W2. The tuning of $\rho$ on the benchmarks to find the best value is problematic and is against the principle of machine learning. Since there is no train/validation set for most of the benchmarks, we alternatively determine $\rho$ based on the single benchmark per task and use the determined $\rho$ for all the benchmarks in the same task type. To validate that the tuning process of $\rho$ is not benchmark-sensitive, we conducted two additional experiments. First, we used a “pseudo-validation set” by randomly sampling 10% of the samples from the benchmark and determined $\rho$ using the partial samples. Second, we applied another benchmark in the same task type to determine $\rho$. The results indicate that the same $\rho$ value can be obtained from partial samples of the benchmark or another benchmark in the same task type. The results indicate that we can find the applicable $\rho$ value with minimal tuning. We added the clarification about the tuning of $\rho$ in the manuscript (Sec 6.3. & Appendix B.2.) and include the results of the experiments with 10% of the samples in the benchmark (Appendix B.2.). The results are also provided here for ease of reference. | $\rho$ | 0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1 (baseline) | |---|---|---|---|---|---|---|---|---|---|---|---| | **General Vision-Language Task** | | | | | | | | | | | | | MME (10%) | 144.00 | 144.27 | 145.82 | 147.28 | 149.33 | 151.33 | 151.58 | 151.85 | 152.59 | 152.20 | 148.50 | | TextVQA (10%) | 57.9987 | 58.1770 | 58.2403 | 58.2501 | 58.2530 | 58.3042 | 58.3898 | 58.5275 | 58.6328 | 58.5320 | 58.1200 | | **Visual Hallucination Task** | | | | | | | | | | | | | POPE (10%) | 85.9012 | 85.9012 | 86.0124 | 86.1002 | 86.4020 | 86.5283 | 86.3606 | 86.3247 | 86.2029 | 85.9930 | 85.8700 | | 1-CHAIRs (10%) | 54.9705 | 55.0330 | 55.3408 | 56.1380 | 56.3189 | 56.5543 | 56.2280 | 55.5402 | 55.0523 | 54.7548 | 54.7021 | | **Vision-centric Task** | | | | | | | | | | | | | CV-Bench-2D (10%) | 56.0034 | 56.0146 | 56.8823 | 56.8842 | 56.9140 | 56.9430 | 57.0327 | 57.1328 | 57.5475 | 57.6000 | 56.1300 | | CV-Bench-3D (10%) | 58.2923 | 58.5538 | 58.5703 | 58.6111 | 58.6550 | 58.7530 | 58.7962 | 58.9041 | 58.9542 | 59.0000 | 58.2800 |

Response to Reviewer 1YMG (2/2)

# Q1. More Experiments with Various LMMs > Q1. Do newer LMMs with anyres and with visual encoder tuned jointly also have the characteristic of visual attention sinks? Do the observation and analyses generalize across LMMs with different language models and sizes? We have added two more LMMs, Qwen2-VL [1] and InternVL2 [2], which are more recent. Qwen2-VL is a anyres model with another language model (Qwen), and InternVL2 is a anyres model with a visual encoder tuned jointly. We observed that the visual attention sinks are also observed in these models. Vision encoder tuning may not affect the visual attention sink phenomenon, as the emergence of visual attention sinks is an intrinsic property of LLMs. We have added the investigation results about the hidden states of these models in the Appendix A.1. Also, we have conducted the same experiments with the proposed method on these models and it consistently improves the performance on these models. The results indicate that visual attention sink works well on various LMMs. We have added the results of the experiments on these models in the main paper (Table 1, 2, 3). [1] Wang, Peng, et al. "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution." 2024. [2] OpenGVLab Team. "InternVL2: Better than the Best—Expanding Performance Boundaries of Open-Source Multimodal Models with the Progressive Scaling Strategy". 2024. # Q2. Qualitative Results > Q2. In Fig 8, it looks like LLaVA + VAR has more visual sink tokens qualitatively. The background tokens in Fig 8 are not visual sink tokens, but noises. Although they have high attention weights, they do not have massive activation in specific dimensions and do not consistently appear in fixed locations. Each attention head does not attend perfectly to the relevant visual tokens, and some attention weights are allocated to the noise. Some may argue that our method potentially amplifies noise, which could adversely affect performance. However, the majority of the attention budget is allocated to important visual tokens, ensuring the effectiveness of our approach. # Q3. Citation > Q3. Please include the reference of LLaVA-1.5-HD-13B. We have added the reference of LLaVA-1.5-HD-13B in the manuscript. Thank you for pointing it out.

Final General Response (2/2)

We believe that the experiments we conducted should not confuse or hinder the reader's understanding of our work. We will provide more intuitive explanations and organize the experimental settings to make it easier for readers to understand the novel phenomenon of the visual sink token and to reproduce the experiments in the final version of the paper. Finally, we sincerely appreciate the reviewers for their constructive feedback, which has greatly improved the clarity and substance of our work. We are grateful for the opportunity to engage in this discussion and to enhance our research through this process. Sincerely, The Authors [1] Geva, Mor, et al. "Dissecting Recall of Factual Associations in Auto-Regressive Language Models." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. [2] Neo, Clement, et al. "Towards Interpreting Visual Information Processing in Vision-Language Models." 2024.