Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Thanks for the constructive feedback! We hope to clarify the confusion and answer the questions below. Feel free to let us know if you have any further questions! **Incorrect examples in Figure 2 (neg):** The question mentioned "Merkel" is not incorrect but a negative instruction/question. That is, "Merkel" doesn’t appear in the image. Therefore, the answer should be, "No, Hillary Clinton arrived at the Los Angeles Get Out The Vote Rally in the image.” We use this kind of negative samples to finetune Large Multimodal Models (LMMs), so that these models can learn to say “no” if the objects mentioned in the questions do not appear in the image, hence reducing hallucinations. **GAVIE Can Evaluate Open-ended Questions:** POPE [8] formulates the evaluation of hallucination as a binary classification task that prompts LMM to answer with "Yes" or "No". However, if the question is a multiple choice question (e.g., “Choose the correct statement about the weather conditions in the image: (a) Cloudy and rainy, (b) Clear blue sky, (c) Foggy and misty, (d) Snowy and cold”) or an open-ended question (e.g., “What objects are on the toddler’s feet?”), POPE can not evaluate model hallucinations on these questions as the answer space is much more complicated. In contrast, our proposed GAVIE can evaluate model hallucinations even when the answers are open-ended manners. (Fig 21-Fig 26 in the appendix) **GAVIE Aligns well with Human Experts** To assess the correlation of GPT-4 rating with the human rating, we hired three experts specializing in NLP to perform the human assessment. From Tab. 4 in the paper, all experts agree that the output from our model is the best, followed by InstructBLIP in the second place, and MMGPT performs the worst. This ranking aligns with our GAVIE. More details are shown in the A.1.1 (appendix). **Why GAVIE Uses Scores 0-10 instead of 0/1.** In an open-ended setting, the model’s answer to a question may be completely wrong, partly wrong, or entirely correct. With a score of 0-10, GAVIE can evaluate the model’s answer and produce a detailed level of correctness instead of only 0/1(correct/wrong), which does not give differentiable scores to answers that are partly wrong or completely wrong (i.e., both will receive a score of 0). However, our GAVIE can assign a score of 0 for the completely wrong answer and a partial score (e.g., 5) for the partly wrong answer. Therefore, GAVIE can not only give a more accurate ranking among compared models but also help people understand the level of correctness for answers from different models. The relationship between correctness and scores in GAVIE is summarized as follows: RELEVANCY has four grade levels: (1) The response is entirely relevant (9-10), (2) The response is mostly relevant (6-8), (3) The response is partly relevant (3-5), (4) The response is seldom relevant (0-2). ACCURACY has four grade levels: (1) The response is completely accurate (9-10), (2) The response has minor errors (6-8), (3) The response is partly accurate (3-5), (4) The response is mostly or completely wrong (0-2). **Reference** [1] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." arXiv preprint arXiv:2310.03744 (2023). [2] Chen, Jun, et al. "MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning." arXiv preprint arXiv:2310.09478 (2023). [3] Liu, Fuxiao, et al. "Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models." arXiv preprint arXiv:2310.14566 (2023). [4] Wang, Junyang, et al. "An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation." arXiv preprint arXiv:2311.07397 (2023). [5] Liu, Yang, et al. "Gpteval: Nlg evaluation using gpt-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023). [6] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023) [7] Wu, Jialian, et al. "Grit: A generative region-to-text transformer for object understanding." arXiv preprint arXiv:2212.00280 (2022). [8] Li, Yifan, et al. "Evaluating object hallucination in large vision-language models." arXiv preprint arXiv:2305.10355 (2023).

Thanks for the constructive feedback! We hope to clarify the confusion and answer the questions below. Feel free to let us know if you have any further questions! **Use Pseudo Data to Scale-up** To demonstrate the scalability of our dataset, we use GRiT [7] to generate pseudo-dense captions in place of the high-quality annotations from humans in the Visual Genome dataset. From Tab. 5, we found finetuning on pseudo captions can also improve the model performance. This demonstrates that our visual instruction generation method has great potential to be further scaled up, even when without groundtruth dense captions. Please refer to Section 6.3 for more details. **Reference:** [1] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." arXiv preprint arXiv:2310.03744 (2023). [2] Chen, Jun, et al. "MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning." arXiv preprint arXiv:2310.09478 (2023). [3] Liu, Fuxiao, et al. "Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models." arXiv preprint arXiv:2310.14566 (2023). [4] Wang, Junyang, et al. "An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation." arXiv preprint arXiv:2311.07397 (2023). [5] Liu, Yang, et al. "Gpteval: Nlg evaluation using gpt-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023). [6] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023) [7] Wu, Jialian, et al. "Grit: A generative region-to-text transformer for object understanding." arXiv preprint arXiv:2212.00280 (2022). [8] Li, Yifan, et al. "Evaluating object hallucination in large vision-language models." arXiv preprint arXiv:2305.10355 (2023).

Thanks for the constructive feedback! We hope to clarify the confusion and answer the questions below. Feel free to let us know if you have any further questions! **Difference with G-Eval [5]** The difference between our evaluation method (GAVIE) and G-eval is as follows: (1) Unlike G-Eval [5], which mainly focuses on text summarization and dialogue tasks, our GAVIE method can adapt to diverse instruction formats and tasks. To demonstrate the effectiveness of GAVIE, we compare GAVIE and human expert evaluation on our evaluation set, which includes 16 vision and language (VL) tasks. From Tab 4, we found that GAVIE aligns well with human evaluation. (2)Unlike G-Eval [5], which only generates one score for each sample according to its evaluation form, our GAVIE generates different scores for different criteria: (a) Relevancy: whether the response directly follows the instruction; (b) Accuracy: whether the response is accurate concerning the image content. Additionally, GAVIE also provides explanations to support the scores. In this way, people can clearly understand how the models perform in terms of different criteria and why GAVIE gives this score. **Blackbox GPT-4:** To reproduce the results of GAVIE, we indicate the GPT4 version ( GPT4-32k-0314) we used in Section 5. We also show some examples in Fig 34 and Fig 35 by replacing GPT4 with current popular open-sourced Large Multimodal Models (LMMs), including LLaVA1.5-13B [1] and MiniGPT4-v2 [2]. From Fig 34, we can see that LLaVA1.5-13B can follow the instructions to evaluate the answers according to two criteria, but it fails to detect that the answers are incorrect, given the image content. From Fig 35, we can see that MiniGPT4-v2-13B can not follow the instructions to generate a score for each criterion. What’s more, the score MiniGPT4-v2-13B generates is not reasonable. Therefore, current open-sourced LMMs like LaVA1.5-13B and MiniGPT4-v2 may struggle when evaluating hallucination in vision-and-language tasks. We come up with two possible remedies, which are promising directions to explore in future works: (1) One can finetune LLaVA1.5 on the vision and language evaluation tasks to enhance its instruction following abilities. Additionally, to further enhance LLaVA1.5, one can also replace the vision encoders with [6] to improve its visual perception ability to make it a better evaluator. (2) One can explore LLM-free methods for VL hallucination evaluation. For example, we can design questions with answers from a closed set (e.g., binary classification) based on different types of hallucinations. For object hallucination, we can construct the question as “Is there a {hallucination object} in this image?”. For attribute hallucination, we can design questions like “​​Is the {object} {state} in this image?”, “Does the {object} {action} in this image?”, “Is the {object} {color} in this image?”. For relation hallucination, we can construct questions like “Is there a direct contact between the {object 1} and {object 2} in this image?”. For all the cases described above, the groundtruth answers to these questions will be “yes” or “no”. In this way, we can evaluate the output of LMMs without GPT4. **Reference:** [1] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." arXiv preprint arXiv:2310.03744 (2023). [2] Chen, Jun, et al. "MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning." arXiv preprint arXiv:2310.09478 (2023). [3] Liu, Fuxiao, et al. "Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models." arXiv preprint arXiv:2310.14566 (2023). [4] Wang, Junyang, et al. "An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation." arXiv preprint arXiv:2311.07397 (2023). [5] Liu, Yang, et al. "Gpteval: Nlg evaluation using gpt-4 with better human alignment." arXiv preprint arXiv:2303.16634 (2023). [6] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023) [7] Wu, Jialian, et al. "Grit: A generative region-to-text transformer for object understanding." arXiv preprint arXiv:2212.00280 (2022). [8] Li, Yifan, et al. "Evaluating object hallucination in large vision-language models." arXiv preprint arXiv:2305.10355 (2023).

Thanks for the constructive feedback! We hope to clarify the confusion and answer the questions below. Feel free to let us know if you have any further questions! **Compare Our Model with LLaVA1.5 and other LMMs:** As current open-source LMMs are built on strong LLMs, they may over-rely on language priors (parametric memory) and generate words more likely to go together with the instruction text regardless of the image content. To address the hallucination, besides the positive instructions with 16 tasks, we build negative instructions with existent object (attributes and relationship) manipulation, non-existent object manipulation, and knowledge manipulation. After finetuning LMMs on our dataset, their answers will be more consistent with the image. Following the suggestions by the reviewer, we compare LLaVA1.5 with our model across three datasets. Note that the backbone of our model is Mplug-owl. (1) We first analyze the discriminative task of AMBER [4], including object existence, object attribute, and object relation hallucination. The evaluation score is F1, and the groundtruth answer is yes or no. | Method | Existence | Attribute | Relation | | ---- | ---- |--- |--- | | mPLUG-Owl-7B | 29.9% |33.5% |26.4% | | MiniGPT4-v2-7B [2] | 80.0% |41.1% |57.9% | | LLaVA1.5-7B [1] | 83.2% |64.0% |65.7% | | Ours-7B | 81.4% |70.4% |69.2% | Although our model achieves a slightly lower F1 score regarding object existence, we outperform LLaVA1.5 in both the object attribute and object relation hallucinations. We attribute our model’s success to the existent object (attributes and relationship) manipulation in the negative instructions. (2) We secondly analyze the results of our evaluation set via GAVIE. Both LLaVA1.5 and MiniGPT-v2 use Visual Genome as their training data. | Method | GAVIE-Accuracy | GAVIE-Relevancy | | ---- | ---- |--- | | LLaVA1 -7B | 4.36 |6.11 | | LLaVA 1.5-7B [1] | 6.42 |8.20 | | MiniGPT4-v1-7B | 4.14 |5.81 | | MiniGPT4-v2-7B [2] | 6.01 |8.10 | | mPLUG-Owl-7B | 4.84 |6.35 | | Ours-7B | 6.58 |8.46 | Although LLaVA1.5 and MiniGPT4-v2 improve significantly compared with LLaVA1 and MiniGPT4-v1, our model still achieves the best accuracy and relevancy. We further show some qualitative examples in Fig. 33 of the appendix. (3) We finally analyze Hallusionbench [3], an image-context reasoning benchmark covering various topics and image types. The groundtruth answer is yes or no. | Method | Accuracy (gt=yes) | Accuracy (gt=no) | | ---- | ---- |--- | | LLaVA 1.5-7B [1] | 92% |15% | | MiniGPT4-v2-7B [2] | 85% |11% | | Ours-7B | 88% |47% | From the results, we find that both LLaVA1.5 and MiniGPT4-v2 achieve high accuracy on the positive set but perform poorly on the negative set. Our model can achieve the same level of accuracy when groundtruth answer is yes and much higher accuracy when groundtruth answer is no. We attribute our model’s success to the knowledge manipulation in the negative instructions. Generally speaking, LLaVA 1.5 performs well when meeting object hallucinations. However, attribute hallucination, relation hallucination, and knowledge hallucination are still challenging for LLaVA 1.5. Therefore, our LRV-Instruction dataset with various manipulation negative instructions and positive instructions of 16 tasks is an excellent resource to address the hallucinations. Thank you for your comments and suggestions. We will include the discussions in our final paper. **Relevancy Score:** To make a fair comparison, we compare our model with LLaVA1.5 and MiniGPT4-v2, which both use the Visual Genome as their training data to address the domain difference. From our evaluation set (GAVIE), our model outperforms both LLaVA1.5-7B and MiniGPT4-v2-7B regarding the instruction-following ability. | Method | GAVIE-Relevancy | | ---- | ---- | | LLaVA 1.5-7B [1] | 8.20 | | MiniGPT4-v2-7B [2] | 8.10 | | Ours-7B | 8.46 | **Human Evaluation Policy:** The details of human evaluation guidelines are shown below: As for each question, there are an instruction, an image, and several answers. Suppose you are a smart teacher, please score the answers according to the two criteria. (1) Accuracy: whether the response is accurate concerning the image content. (2) Relevancy: whether the response directly follows the instruction without unrelated answers. There are four options for the scores: A score of 1 ( Very Poor): The response is seldom relevant to the instruction and is mostly wrong according to the image. A score of 2 ( Poor): The response is partly relevant to the instruction and partly accurate according to the image. A score of 3 (Good): The response is mostly relevant to the instruction and has minor errors according to the image. A score of 4 (Excellent): The response is completely relevant to the instruction and completely accurate according to the image.