Summary
The paper proposes a new re-weighting strategy for training VLMs to improve visual-language alignment. The motivation is to assign higher weight to visually relevant tokens and lower weight to visually irrelevant and visually contradictory tokens. The re-weighting factor relies on the logit difference with and without image inputs for each token. Using the proposed CAL, the paper demonstrates general improvement in VQA and captioning tasks.
Strengths
The motivation to dynamically assign weights to different types of tokens is intuitive and well-studied in the language domain. Several studies have attempted to apply this method to vision-language tasks in a zero-shot manner, which incurs high inference costs. This paper proposes training the model with this method instead. The approach is simple and effective, demonstrating general improvements over existing state-of-the-art models across benchmarks. The analysis further validates the effectiveness of the proposed method.
Weaknesses
1. **Lack of Clarification in Presentation**: Some content is confusing. For example, equation 2 is averaged across the entire sentence with length $l$, yet the description in lines 88-89 refers to the loss objective for the $i^{th}$ sample at token $t_j$, which seems more like the loss at a single token. Additionally, equation 4 lacks clarification; it's unclear how to apply average pooling with a window size from this equation. Some experiments also lack necessary context. For instance, it's unclear which model is evaluated in Figure 2a, and the ablation study in section 3.3 doesn't specify which model is used for evaluation. The main result in section 3.3 doesn't disucuss how to apply CAL into model training, at which stage for example. The annotation is inconsistent; in equation 4, it's $\tilde{w}^{i,t_j}$, while in equation 5, it's $\tilde{w}(i.t_k)$
2. **More Experiments Are Required**: The paper presents decent improvements over some baselines, mainly in vision-language tasks. However, since the training recipe has changed, it would be better to also evaluate other tasks, such as language-only tasks, to see how the CAL affects the model's ability in standard language understanding. Additionally, the method seems quite relevant to hallucination tasks, so it would be beneficial to report results on those as well.
3. **Lack of Originality**: The motivation and methodology are already well-studied in language models [1,2], and prior studies have also applied changes to logits based on the presence or absence of images to measure the relevance of visual and language inputs. In this context, the paper simply applies the method during the training stage, instead of the inference stage as in [3]. Therefore, the novelty is somewhat limited.
[1] Jiang S, Wolf T, Monz C, et al. TLDR: token loss dynamic reweighting for reducing repetitive utterance generation[J]. arXiv preprint arXiv:2003.11963, 2020.
[2] Lin Z, Gou Z, Gong Y, et al. Rho-1: Not all tokens are what you need[J]. arXiv preprint arXiv:2404.07965, 2024.
[3] Zhu L, Ji D, Chen T, et al. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding[J]. arXiv preprint arXiv:2402.18476, 2024.
Questions
1. The trend shown in Figure 2a is based on which model? Is this a common phenomenon across existing VLMs?
2. How do you deal with cases where a semantic word has been split into two subparts, each with a different value, such as "tr" and "uck" in Figure 2a?
3. The CAL seems to rely heavily on the model's logits with and without image input. However, can the model accurately predict changes in logits, as shown in Section 2, during the initial alignment stages?
4. The training requires an additional forward pass of the LLM. Why is the training described as "lightweight" in the paper? Doesn't this double the cost?