Official Comment by Authors
We are very grateful for your detailed review and feedback. Please find our response to your questions below.
**Q1: ... How to select the range of attention masking, and whether the alignment effect can be optimized by dynamic adjustment?**
- Since we use BDHS to generate “rejected” responses, our primary goal was to ensure that these responses are semantically different from the “chosen” responses. To achieve this, we designed an iterative approach where a rejected response is added to our dataset only if it meets a specific threshold on the similarity metric (a hyperparameter determined through our experiments), ensuring it is not too similar to the chosen response. In this process, as you may have suspected, we observed that simple question-answer pairs typically require more iterations to produce a response that is sufficiently distinct from the chosen response. We conducted extensive ablation studies on the amount of masking (Appendix G) and compared the sensitivity of our approach to that of other state-of-the-art methods designed for similar tasks (POVID dataset). Our results demonstrate that our approach is both more effective and less sensitive to the hyperparameter. That said, exploring how the amount of masking required to (more) reliably introduce hallucination correlates with visual complexity is an interesting idea, and we plan to explore its impact in future research.
**Q2: ... Could this similarity score lead to a tendency for models to oversimplify when faced with less common or more complex visual scenes?**
- Our approach relies on the ability to measure the similarity between the embeddings of chosen and rejected responses. Most recent sentence embeddings perform well in this task, making our pipeline relatively robust to the specific choice of embedding. After evaluating multiple options, we empirically found all-mpnet-base-v2 to be an effective and computationally efficient choice. This computational efficiency is particularly important in an online setting. Furthermore, by employing SFT forcing, we avoid oversimplifying the rejected responses. In most cases, the model generates only part of the “rejected” response rather than rewriting the entire sentence. This is a crucial aspect of the BDHS algorithm, which ensures that the rejected responses remain minimally different from the chosen responses in terms of complexity and style, while being semantically distinct. For example, two responses might describe the color of an object using the same style, but one specifies the correct color while the other specifies an incorrect one. This design ensures that the model must comprehend the image in order to optimize the objective effectively.
**Q3: ... Especially on small-scale models, is it possible that the method effect is not significant due to parameter limitations?**
- We have not conducted experiments on smaller models. Initially, we started our experiments on weaker models, specifically the LLaVa 1.5 family. However, we decided against continuing with them because, although they were widely used by the community at the time, we found that weaker models had such a large margin for improvement that almost any intervention proved effective. For example, higher values of 𝛽 in the DPO setting consistently resulted in better performance. This suggests that diverging from the reference policy enhances the model’s capabilities, which may indicate underlying weaknesses in the base model. Instead, we focused our study on stronger models after the SFT stage to better highlight the benefits that the alignment stage can bring to multimodal LLMs. While we conducted some limited experiments on larger models (e.g., LLaVa 1.6 13b), these followed similar patterns to the 7b model, however were not comprehensive enough to include in the paper due to compute limitations. Based on our findings in this paper and our ongoing research in this area, we believe this approach is consistently effective in settings where we can observe LLM bias in multimodal LLM behavior.
**Q4: To what extent do current hallucination evaluation benchmarks such as POPE and MMHALBench-V truly reflect model performance in real-world applications?**
* This is certainly one of the areas our research identified as having significant room for exploration, which we discussed in Appendix B2-B4. We found our proposed MMHALBench-V to be an effective benchmark for evaluating hallucination; however, we acknowledge that there is still plenty of scope for further improvement. For the experiments presented in the paper, we carefully analyzed both the wins and losses associated with any observed advancements to ensure they genuinely reflected enhanced model performance. For example, although we observed substantial gains in results on ObjhalBench, we excluded those numbers due to the issues outlined in Appendix B3.