Right this way: Can VLMs Guide Us to See More to Answer Questions?

In question-answering scenarios, humans can assess whether the available information is sufficient and seek additional information if necessary, rather than providing a forced answer. In contrast, Vision Language Models (VLMs) typically generate direct, one-shot responses without evaluating the sufficiency of the information. To investigate this gap, we identify a critical and challenging task in the Visual Question Answering (VQA) scenario: can VLMs indicate how to adjust an image when the visual information is insufficient to answer a question? This capability is especially valuable for assisting visually impaired individuals who often need guidance to capture images correctly. To evaluate this capability of current VLMs, we introduce a human-labeled dataset as a benchmark for this task. Additionally, we present an automated framework that generates synthetic training data by simulating ``where to know'' scenarios. Our empirical results show significant performance improvements in mainstream VLMs when fine-tuned with this synthetic data. This study demonstrates the potential to narrow the gap between information assessment and acquisition in VLMs, bringing their performance closer to humans.

Paper

Similar papers

Peer review

Reviewer SnhN6/10 · confidence 3/52024-07-03

Summary

VLMs offer an interesting opportunity to describe their uncertainty if the images given to them are of poor quality. This paper explores a framework for estimating whether a VLM can identify if its ability to correctly answer a visual question would improve were it given a better picture. The paper introduces a new labeled dataset for VQA where questions involve identifying how best to take better pictures. The paper introduces also a finetuning scheme with synthetically generated examples leveraging the labeled dataset.

Strengths

- The problem is interesting, relevant, and important - The novel data set is an important contribution after it is released - The performance improvement after fine-tuning on LLaVA is impressive.

Weaknesses

- It would be interesting to see how the model performs with different prompting techniques. Beyond vanilla CoT, can we try methods covered in (https://arxiv.org/pdf/2402.07927) - The problem specifically focuses on qualitative spatial reasoning with respect to moving the camera. Can this methodology be used for quantitative reasoning (e.g., Spatial-VLM, Spatial-RGPT), by identifying not just which direction, but by how much to move the camera? - The two perturbation ranges of 0.1-0.9 and 0.3-0.7 are confusing. What exactly are you differentiating with these categories? It would have been interesting to evaluate over disparate ranges (e.g., 0.1-0.3, 0.3-0.5, 0.5-0.7, …) so that we can see how performance drops with the perturbations

Questions

- Why is LLaVA-1.5 13b worse than 7b?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are lightly discussed

Reviewer Eiqe6/10 · confidence 4/52024-07-12

Summary

- The paper proposes a recourse for unanswerable visual questions — rather than simply abstain, can a VLM indicate how to adjust an image so that the question becomes answerable? - The authors introduce the Directional Guidance VQA task and a dataset for the task. - The authors propose a data generation pipeline to produce data that can train models for the directional guidance task.

Strengths

- The problem studied by the paper is practical and has the potential to have a real-world impact. - There have been a number of works on VQA models that can abstain, but abstention alone is not ideal for real-world deployments because it would be frustrating to be told that your question was unanswerable. - Providing a remedy for unanswerable visual questions via directional feedback is a useful and novel contribution. - The framing of the task is simple and reasonable. - The data generation pipeline is clever and uses a model-in-the-loop technique to systematically generate training instances.

Weaknesses

Since this is somewhat of a resource paper, it would be nice to have multiple vision-language models of different sizes. This is also a bit of a concern given the fact that a model-in-the-loop technique is used to create a training dataset for the task, since different models would ostensibly have different weaknesses. Another weakness is that this is a very narrowly scoped task and dataset that is maybe a better fit for a CV conference.

Questions

- Table 1 is confusing and looks aesthetically unpleasant. - Your table numbers are wrong. Ex: in L295 you refer to Table 6, but this is actually Table 1 and Section 6. Your `\label` is in the wrong place.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Not applicable.

Reviewer Eiqe2024-08-13

I have read the author response. I will maintain my rating of a 6. I do not think the paper should be rated lower because it introduces a resource annotated by humans that addresses a real problem. I do not think the paper should be rated higher because the scope is very narrow and there is little technical contribution — it's an application paper (train VLM on dataset with an handcrafted augmentation technique that works only for this application), albeit a useful one. Also, the organization and presentation of the paper could have been better: it was hard to parse conclusions from the body of the results section.

Authorsrebuttal2024-08-13

Thank you for the feedback

Thank you for your feedback and for maintaining a positive rating. We appreciate your recognition of the value our resource brings. To improve the clarity and organization of the results section, we will split out the conclusions into a separate section and enhance the overall structure to ensure our findings are more clearly presented in the next revision.

Reviewer eg4t5/10 · confidence 3/52024-07-13

Summary

This paper evaluates and expands the self-reflection capabilities of multiple MLLMs, so that MLLMs can actively evaluate the adequacy of given query information and give suggestions on how to find more reliable information. This paper proposes a hierarchical cognitive process pattern theory and creates a manually annotated dataset for evaluation.

Strengths

Methods for active reflection and evaluation of given information are meaningful for enhancing the quality of MLLM responses.

Weaknesses

1. The rationality of the proposed hierarchical cognitive process pattern, which comprises three levels, merits validation. First, are these three levels both necessary and sufficient for the cognitive process? If the issues at these three levels are addressed, can we then resolve the problems associated with the MLLM cognitive process? Second, does the evaluation benchmark and the ability to expand MLLM, as presented in this paper, align with these three levels? 2. The method proposed in this paper is fundamentally similar to the MLLM self-reflection method. Can it be directly implemented through self-reflection without requiring additional training? 3. A comparable method to the one presented in this paper is DualFocus [1], which not only actively evaluates image quality but also performs more sophisticated active grounding. 4. Lastly, the evaluation dataset used in this paper is relatively small, which makes the evaluation results less convincing. [1]Cao Y, Zhang P, Dong X, et al. DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models[J]. arXiv preprint arXiv:2402.14767, 2024.

Questions

Evaluate the difference and experimental effect between this method and the large model self-reflection method.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Area Chair bGbP2024-08-12

Please read the author rebuttal, other reviews and respond to the authors NOW!

Dear Reviewers, Thanks to those of you who already responded to the authors acknowledging the rebuttal and asking follow-up questions if any. Those who have not responded yet, please do the following ASAP: thoroughly read the rebuttal, the other reviews and respond to the authors about whether all your questions / concerns have been addressed or not. If not, please elaborate on which questions / concerns are still not addressed so that the authors have fair chance of addressing them before the author-reviewer discussion period ends in ~41 hours from now (August 13th, 11:59pm AoE). Your AC

Reviewer SnhN2024-08-12

Thanks for the rebuttal

I have read the rebuttal and appreciate the author's clarifications, especially with regard to the perturbation ranges. I encourage the authors to include this discussion as well as some discussion on the effect of better prompting, CoT, etc. on the results already found.

Authorsrebuttal2024-08-13

Thanks for the response

Thank you for your feedback. Could you kindly clarify whether you are suggesting revisions to the paper's discussion section, or if you would prefer us to engage in further discussion during the current reviewer-author discussion period? We plan to include our discussion on general perturbation ranges, diverse prompting and CoT strategies in the paper's discussion section in the next revision. We will also provide more details in the supplementary materials to demonstrate the sensitivity observed through our extensive prompt explorations.

Reviewer eg4t2024-08-13

Thanks for the authors' detailed response. Their reply effectively addressed my concerns, so I decide to increase the score from 3 to 5.

Authorsrebuttal2024-08-13

Thank you for the response

Thank you for the thoughtful reconsideration of our work. We're glad that our explanations addressed the previous concerns, and we appreciate your decision to adjust the score based on our clarifications.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC