Response to Reviewer GxLA (1/2)
Thank you for your thoughtful reviews. We appreciate the opportunity to clarify key contributions in our paper: the Modality Integration Rate (**MIR**), a reliable metric for evaluating LVLM pre-training; the learnable Modality Calibration (**MoCa**), a novel lightweight module ot help LVLM narrowg the modality gap during training.
Here we would like to first address some questions to clear up any misunderstandings, which may have arisen from our writing or formulation.
**Q1: What is the necessity of proposing MIR?**
**A1**: It’s essential to clarify MIR’s role in LVLM pre-training. As we highlighted in the Introduction, there has been **no** reliable metric for assessing pre-training quality specific to LVLMs. Traditional evaluation methods borrowed from LLMs, like perplexity, are **ineffective** for evaluating LVLM pre-training (See **Figure 1** in our paper, they can not accurately reflect the post-SFT model performance on downstream benchmarks) due to the different pre-training objective with LLMs. So the common way to evaluate the pre-training quality of LVLM is actually fixing the SFT data/setting to apply SFT on the pre-trained LVLM, then report performance across benchmarks. This approach, however, is **costly** and **time-consuming**. **Therefore, MIR addresses a critical need for a reliable, direct evaluation metric for LVLM pre-training that avoids the need for subsequent SFT.**
**Q2: What is the motivation behind MIR’s design?**
**A2**: Our design in MIR stems from the motivation shown in **Figure 2** of our paper, where we demonstrate that current LVLMs exhibit noticeable modality gaps at shallow layers (including the text embedding space). During training, the shallow layers of the base LLM strives to bridge this gap. This perspective **contrasts** with the common view that the projector (e.g., LLaVA’s MLP projector) can map vision representations directly into the text embedding space. Thus, MIR evaluation is on **two main aspects**:
- Whether the pre-trained projector can precisely map the vision representations to the similar domain distribution with the language's;
- Whether the base LLM has the capability to quickly narrow the modality gap between vision and language at the shallow layers.
Our MIR design builds upon the two aspects, with core computation steps that include accumulating the modality gap across all layers, text-centric normalization, and outlier removal.
**Q3: Does MIR simply use FID in its implementation?**
**A3**: We should clearly clarify about the designs and insights about MIR. The method part of our paper may make some misunderstandings for you on the design of MIR. Indeed, MIR **does not simply use FID** but instead involves **three key steps**:
- **Text-Centric Normalization**: MIR has a text-centric normalization for both vision and text tokens at the same LLM layer. It is motivated by the sensitivity of Frechet distance on the absolute value scale (e.g., calculating Frechet distance on (100x, 100y) will obtain the much larger result than calculating Frechet distance on (x, y)), which makes the results not accurate since the absolute values of token representations are gradually larger when the LLM layer goes deeper. This mechanism enables the reliable cross-layer comparison and the reasonable cross-layer accumulation in Step 3.
- **Outlier Removal**: MIR uses the “3-$\sigma$” principle to remove outliers for both vision and text tokens, recognizing that some outlier tokens contribute little semantically but can skew the distributions [1,2,3]. This step reduces outlier influence, which is crucial as Frechet distance is sensitive to such deviations.
- **Layer-wise accumulation**: MIR calculates Frechet distances (without Inception encoding) between vision and text tokens at each LLM layer, accumulates these distances, and computes the logarithm of the result for the final MIR score. This cross-layer accumulation, although straightforward, is well-motivated by the observations discussed in **A2**.
We hope these clarifications provide better insight into the motivations and design of MIR, and we look forward to addressing any further questions. We will try to address the concerns raised in **Weakness 2\&3** at the next part of our response.
[1] Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR, 2024.
[2] Sun et al. Massive Activations in Large Language Models. COLM, 2024.
[3] Gu et al. When Attention Sink Emerges in Language Models: An Empirical View. Arxiv preprint 2410.10781, 2024.