Multi-Object Hallucination in Vision-Language Models

Summary

This paper proposes a benchmark of multi-object hallucinations in Large Vision Language Models (LVLMs). Specifically, this paper explores thoroughly on how LVLMs behaves when multiple objects are prompted in user instructions. They introduce Recognition-based Object Probing Evaluation (ROPE), an automated pipeline used for collection their benchmark. This paper also presents some interesting findings, including that existing LVLMs may follow shortcuts to generate hallucinated responses.

Strengths

**S1: Novelty**. Studying how object tokens affect each other in LVLM inference sounds new. Most papers addressing LVLM object hallucination are focusing on single-object. Multi-object hallucination sounds a new perspective. **S2: Thorough analysis**. This paper conducts comprehensive analysis and discuss on 9 potential impacting factors that may lead to multi-object hallucination, which are quite a lot experiments. **S3: Interesting academic findings**. The instruction format that induces LVLM to hallucinate is interesting. This is validated by authors a quite significant factor that leads to multi-object hallucination, which can not be simply remedied by scaling up LLMs.

Weaknesses

**W1: missing evaluations**. Noticing that this paper mainly focuses on object hallucination, I wonder why authors avoid evaluating published methods that address single object hallucination in LVLMs, including decoding methods like OPERA [15] or RLHF methods [43, A]. It is quite important to check how existing single-object methods perform on the proposed multi-object benchmark. However, this part is missing from this paper, which is encouraged to be presented clearly and thoroughly to support a new benchmark. **W2: limited significance on real-world use case.** Despite interesting findings and comprehensive analysis on proposed multi-object hallucination, this paper experiments under a very specific type of inference setup. Specifically, their experiments are conducted on a specific type of instruction format (Figure 2) and limits its output tokens to objects only (line 140-150). For proposing a new benchmark, I believe this problem is worth to do some thinking. But this part is missing from this paper. Authors are suggested to discuss more on potential impact from this paper. [A] RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback. CVPR 2024.

Questions

## **Q1** For Figure 3 (right), ground truth object for obj5 is "orange", but both LLaVA-7B and LLaVA-34B hallucinate it as an "apple". This paper explains this phenomena in line 165-166 as follows, > *However, they tend to make more mistakes when all tasked object classes are different or **when a new object class is introduced after multiple repeated tasks***. It should be noted that authors use a very specific type of instruction here, > *Provide the class names in the format: 'obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>', with no additional words or punctuations.* I have a few questions for this example here. **(1-a)** how LLaVA-7B and LLaVA-34B performs when prompted with "*Is there an orange in this image?*" This can be a baseline to this example. **(1-b)** it should be noted that authors adopt three types of instructions in this paper (line 136-150). What type of instruction is used for this sample? **(1-c)** noticing that in this paper LVLMs are forced to follow a special format and decode object tokens only, what do authors think about significance of such prompting? ## **Q2** Given special instructions (Figure 2) used in this paper, a simple exploration is to train an LVLM (e.g., LLaVA) that follows such instruction well. One reason for why LVLMs performs badly is probably that, these models do not fit these special instructions very well. Authors are suggested to include such models and results accordingly. Specifically, authors can train a LLaVA model with instruction data below. For LLaVA [29] and take ground truth objects in Figure 2 as an example, this could look like, > *A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions*. > **USER**: *<image>*. > **USER**: *Provide the class names in the format: obj1: <class1>, obj2: <class2>, obj3: <class3>, obj4: <class4>, obj5: <class5>, with no additional words or punctuations*. > **ASSISTANT**: *obj1: fork, obj2: knife, obj3: whisk, obj4: lemon, obj5: jar*. This is also explored in a recent paper [A], which I think will improve this paper and make findings from this paper more convincing. [A] Object Recognition as Next Token Prediction. CVPR 2024.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Authors include Limitations in Appendix B.1.

Summary

Previous evaluations of large vision-language model (LVLM) hallucinations have primarily focused on single objects. This paper introduces a novel hallucination evaluation benchmark named ROPE, which simultaneously assesses multiple objects within a single scene during testing. The authors present several empirical findings about multi-object hallucinations, including the statistical correlation in object distribution and its impact on hallucinatory responses.

Strengths

1. The paper is well-written and clearly articulates the differences between ROPE and previous benchmarks. Especially, observations and motivations are quite reasonable. 2. Multi-object hallucination is an under-explored aspect in the LVLM hallucination scene. Also, its related findings are valuable and worth further investigation. 3. The experiments and analyses conducted are extensive and detailed.

Weaknesses

1. The authors can consider adding a simple baseline method for a prospective solution for mitigating multi-object hallucinations, as a starting point. 2. Latency analysis in evaluation could be included to provide a more comprehensive analysis. For example, comparing 5 times single-object evaluation vs. multi-object evaluation. Also, especially for student/teacher forcing as they require multiple consecutive inferences. Understanding the time efficiency of the benchmark is important for practical applications.

Questions

1. Do you have any initial ideas or hypotheses for mitigating multi-object hallucinations? 2. Adding a baseline methods is not a mandatory rebuttal subject, but will be a good addition to the current version.

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes, this paper generally addressed the limitations.

Summary

This paper introduce a novel multi-object hallucination evaluation task, which assess model capabilities of classifying multiple objects given visual prompts or spatial tokens. The benchmark dataset contains both commonly seen images and unseen images regarding the instruction dataset. Evaluation results show that open-source MLLMs and even proprietary GPT-4o struggles on this task.

Strengths

1. The motivation of assessing multi-object hallucination is clear. 2. The benchmark can facility the research on understanding MLLMs from a new perspective. 3. The experiments are meticulously designed and cover a broad spectrum of factors.

Weaknesses

1. The writing in the experiment section, particularly in Section 5, could benefit from significant improvement.

Questions

1.The legend for Figure 5 appears to be misaligned with its content, leading to confusion. Also the discussion about Figure 5 is confusing. The analysis about (c) (f) (h) (i) are not aligned with sub-figures, while discussions about (d) (g) are aligned with the content. (a) and (b) are not addressed in the discussion at all. 2. It is unclear which prompting strategy (bounding box or special tokens) is utilized for each model listed in Table 2. This information should be specified for clarity. 3. Typos: LLaVA-7B (unseen) student-forcing (wild) score should be in bold.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

the authors adequately addressed the limitations

Summary

1. The work proposes a **new benchmark evaluation method (through modifying existing datasets)** to measure if VLMs can accurately recognize multiple objects in an image simultaneously. This evaluation pipeline, designed by adding 5 bounding boxes to each image in the dataset and creating a fixed prompt, requires the model to identify which class each object within these boxes belongs to. 2. Using this benchmark dataset, the authors analyze the performance of VLMs in recognizing multiple objects simultaneously and the **factors that may lead to their hallucination**. 3. The contribution of the article lies in its **detailed analysis and insights into understanding VLMs' multi-object recognition capabilities**. Several important takeaways are: - performance degradation with multiple object recognition: VLMs perform worse when tasked with recognizing multiple objects at once compared to single object recognition. (not convinced by the current experiment results, need more clarification from authors -> see questions below) - using shortcut to explain hallucination behavior: When multiple objects of the same class are required to predict at the same prompt, VLMs tend to use previously recognized classes to predict subsequent object classes, leveraging shortcuts. It suggests the model simply relies on previously identified classes for subsequent predictions.

Strengths

1. The task is well-defined and the evaluation metric is clear (I like the student-forcing and teacher-forcing idea): identify the object from the 5 bounding boxes at the same time. 2. Comprehensive Analysis: It thoroughly examines various factors contributing to hallucination, offering valuable insights into potential causes. It can help the readers to understand hallucination causes in VLMs.

Weaknesses

1. **Unclear Benchmark Positioning**: The paper's explanation about how this benchmark differs from other datasets is quite confusing (lines 35-48). What specific capability does this benchmark target to measure compared to others? Why is image captioning verification benchmarks insufficient for multiple object consideration? Given a limited evaluation budget, why choose this dataset over others? Is current common benchmarks focus on 1) image captioning, checking if mentioned objects appear in the image and their count, and 2) image grounding, verifying if given descriptions can be located in the image? Could you explain it in more details? 2. **Writing** (minor): Some parts of the writing could be improved for better comprehension. For example, line 53 could explain earlier why the dataset is split into four sets. Readers only understand later through experiment analysis that these sets investigate if LLMs learn shortcuts by using previously predicted classes. 3. **Clarification of Shortcuts and Spurious Correlations**: The shortcut story is very interesting, could you elaborate this part more? What are the shortcuts and spurious correlations when claiming VLMs use them? What evidence supports this conclusion? IMO Teacher forcing results and Figure 4/5a indirectly support this, but a detailed, systematic analysis in one section would be better for understanding this.

Questions

3. Lines 182/183: **How are seen/unseen datasets divided**? Is it based on the classes from the Visual Genome instruction tuning dataset? 4. Table 2's single-object setup seems confusing. Is it 1) each image with 5 bounding boxes results in **5 independent sessions** with the VLM, or 2) each session asks about one object while retaining history for the next question about another object from another bounding box, or 3) every image only have 1 bounding box? Is the wild/hom/het dataset division of the single-object column different only on the bounding boxes? - Table 2: If it's 1), why is the single-object recognition accuracy so low, especially for LLaVA-34B? How does having 5 bounding boxes in a single-object setup impact results? For LLaVA-34B, performance for single and multiple objects is similar, which contradicts the expectation that performance would drop when asked about multiple objects simultaneously instead of focusing on single-object. Do you have other evidence supporting this claim? 5. Lines 215 and 218: Can you explain the difference between hypothesis 1) and 2)? This section is somewhat confusing. 6. Line 254: The logic here is unclear. Why does grounding tuning have little effect? Are there experimental results supporting this conclusion? 7. Figure 5: Suggest explaining **how hallucinate and non-hallucinate are calculated in the caption**, or provide a reference link to the relevant section. For example, for Figure 5f, is the process as follows: consider every object from every 5k images -> in Toal 25k objects, and classify them into hallucinate and non-hallucinate, calculate everyone's semantic salience? 8. Figures 5f/h/i: **It seems the results of hallucinate and non-hallucinate are reversed?** The descriptions seem contradictory. For instance, Figure 5h suggests that non-hallucinate objects have higher token entropy, **implying that more uncertain tokens are less likely to hallucinate.** 9. Line 298: Why consider the last generated token instead of other positions, like the answer token? 10. Figure 6: Why compare actual class and predicted class? I find it difficult to interpret due to lack information because each image should pair predicted and actual classes. Presenting them as two distributions loses this pairing information. A caption explanation of takeaways for these 3 images might help. 11. Do you think hallucinations generation are entangled with the model's ability to correctly recognize visual text information for the bounding boxes? Have you tested the model's ability to recognize bounding boxes alone? 12. (optional) Have you considered incorporating a user study where users provide bounding boxes and then answer questions? This would measure the model's capability in real-world scenarios where bounding box boundaries might be unclear.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

The author has listed their limitation in the appendix B.