LOVA3: Learning to Visual Question Answering, Asking and Assessment

Summary

This paper augments presents a data augmentation / multi-task learning technique to improve model quality for Visual Question Answering (VQA). The key idea of the paper, motivated by analogy to humans, is that asking questions and assessing answers are also key skills, apart from just answering questions. The paper seeks to train a model to "Answer, Assess, Ask" jointly, relying existing datasets for a variety of answering and asking tasks, and deriving a new dataset called EvalQABench for the assessment task. The introduction of the EvalQABench dataset, initially created by LLMs and later filtered by experts, is another potentially valuable and lasting contribution. Multiple tasks on augmented data are implemented on the LLava backbone which is an existing SOTA model. The paper compares their technique (called LOVA) to several SOTA models on a variety of datasets showing robust gains in a multitude of settings, providing confidence in the technique's validity.

Strengths

The paper is well motivated: assessing and asking (evaluation and question generation) are closely associated tasks to question answering, that can be performed on datasets that are easily derived from question answering datasets. The argument that training on these closely related tasks improves generalization on question answering is intuitive though reliant on analogy with humans, which has its own traps. The paper evaluates their technique against a variety of SOTA models, and across a multitude of tasks, proving that the gains are robust. The paper also provides ablation studies for various components, showing their utility. In general the experiment section is detailed, extensive and is a highlight of the paper. The paper has 100 citations, and extensive references to related work, making it easier to assess the novelty of the work.

Weaknesses

As the authors point out, due to cost considerations, the authors only evaluate the technique on smaller (relative to MLLMs) models. This is important as model size is a confounder when it comes to assessing the usefulness of data augmentation or multi-tasks. A technique useful for a 7B model is not necessarily useful for a 70B model. However, given the cost of inference of larger models, improving smaller models to be competitive with larger models has its own benefit. There is prior work that already includes question answering and question generation, for example the InstructBLIP paper. Viewed in that sense, this paper makes an incremental contribution adding the assessing task to the answering, asking combination that was already shown to be useful earlier. However, the EvalQABench dataset is potentially very useful for the whole subfield of visual question answering. One minor but interesting finding in the paper is that in a balanced dataset split rightdown with model with 50% Yes and 50% No answers, not all models predict Yes/No close to 50% of the time.

Questions

In section 1, there's a claim that the EvalQABench datasets is generated via a "new automatic pipeline". However, in section 3.2 the authors say "... acknowledging that the Fuyu-8B model is not flawless and recognizing that no multimodal model, including GPT-4V, is perfect, we have implemented both manual filtering and error correction...". Do the earlier claims about the pipeline being automatic overstate the case? Are they necessary? Does the feedback add value beyond just rephrasing the answer is a longer sentence. A lot of the feedback seems trivial and already captured in the q,a pair. For e.g. "What does the woman have on her back". "backpack" vs "No, the woman has a backpack on her back". As another e.g. "What are the people doing?", "motorcycling", vs "No, the people in the picture are motorcycling".

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Smaller models sizes are an understandable limitation as mentioned in the paper, and referenced earlier in the review. The introduction of EvalQA dataset and the additional "asses" task is the key incremental contribution of the paper. By looking at rows 4 and 7 in Table 6 that shows ablation studies, one could discern the incremental benefit of EvalQA. The deltas in the scores are somewhat underwhelming (unclear if they are significant).

Summary

This paper enhances the MLLM's visual understanding capability by training it to ask questions about an image and evaluate the correctness of given question-answer pairs about an image. To achieve this goal, new data is extracted from existing datasets and a new model is fine-tuned on the new data. The experiment shows that the newly added data can improve the MLLM's capability of understanding of images with higher scores on VQA tasks.

Strengths

- The paper is generally well-written and easy to understand. - The argument that training a MLLM to ask questions and evaluate answers can improve its visual understanding is reasonable and, verified by the well-conducted experiments in the paper. - The experiment setups are carefully designed to avoid unfair comparisons.

Weaknesses

- The three key capabilities of MLLMs covered by the paper--asking, answering, and evaluation--should be characterized in an interactive environment (like in an embodied environment where the MLLM is treated as the high-level perceiver/planner/controller of robots) instead of in the static environment. Consider, for example, an MLLM doing an embodied task that needs asking about some key questions, this is where the asking capabilities really make sense. However, the paper only trains and evaluates the MLLM in simple VQA problems as in previous literature. In the paper's current state, the value of the paper is limited and, from my perspective, does not meet the bar of acceptance if VQA tasks are considered only. The scope of the paper needs to be increased to a significant extent that touches the essence of MLLMs with higher-level capabilities that incorporate iterative/interactive thinking and planning. - The added synthesized data only gives the model a limited improvement in performance, while adding a large amount of computation overhead. In fact, if we use models like GPT-4(V) to synthesize random VQA data, the performance will increase as well [1], so I do not see the clear benefit of specifically doing the asking and evaluation data augmentation. This problem is relevant to the first problem: the capability added to MLLM should not be evaluated in VQA tasks. [1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs.

Questions

(Table 6) Why does adding GenQA-Grounding data improves ScienceQA performance?

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The paper mentions some limitations of the proposed pipeline. However, as mentioned above, the biggest limitation is the limited scope of the considered setting which only involves VQA (including grounding) problems without considering the embodiment of MLLMs.

Summary

The paper introduces LOVA3, a framework designed to enhance Multimodal Large Language Models (MLLMs) by incorporating not only visual question answering (VQA) but also the capabilities of generating questions (GenQA) and evaluating question-answer pairs (EvalQA). The primary objective is to improve the comprehensive multimodal understanding of AI models. LOVA3 includes the development of EvalQABench, a benchmark with 64,000 training samples to evaluate VQA data quality. The framework uses the LLaVA-1.5 model as a base, incorporating datasets like VQAv2 and GQA to train these new tasks. Experimental results on ten multimodal benchmarks demonstrate that LOVA3 significantly improves the models' performance, highlighting the benefits of incorporating comprehensive questioning and evaluation abilities into MLLMs. The paper emphasizes the approach and robust results, despite noting the increased computational cost and the need for further testing on larger models and domain-specific tasks.

Strengths

1. LOVA3 introduces a strategy that extends beyond traditional VQA tasks by incorporating question generation and evaluation. 2. The creation of EvalQABench provides a rigorous way to test and improve MLLMs. 3. The multiple perspectives of experimental results provide insights of the proposed framework across multiple benchmarks.

Weaknesses

1. Incorporating additional tasks like GenQA and EvalQA, but the two tasks are also the existing steps of the visual language instruction generation for visual question answering (e.g. SEED-Bench) or visual instruction tuning (e.g., LLaVa-Bench). They also used LLMs or MLLMs for the dataset generation and validation. To explained the special novelty or contribution would be better. 2. The work doesn't provide detailed explanations on how to validate the generated data quality from humans instead of using imperfect models (LLMs or VLMs). It uses Fuyu-8B for data generation but employs a stronger MLLM (LLaVA 1.5) as the base model for instruction tuning. Since LLaVA 1.5 is stronger than Fuyu-8B, the generated negative samples would be less challenging and easier to recognize by stronger models. 3. The paper lacks a more in-depth analysis of potential data biases and strategies to mitigate them. 4. The proposed benchmark is relevant to visual question answering and data generation for visual question answering. It would be necessary to survey and discuss the recent existing datasets (e.g., VQA-GEN, CrossVQA, OVQA, STAR) and generated benchmarks (e.g., LLaVA-Bench, SEED-Bench, SOK-Bench, CinePile) as fully considered. 5. The paper does not provide the generated dataset for review, which is important for the validation of the work.

Questions

1. How about the prompt stability of the QA generation and the differences when using the different variants of prompts? 2. Why does the work apply Fuyu-8B instead of LLaVA 1.5 for the data generation and is there any comparison between the different new VLMs?

Rating

3

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

NA

Author response to all reviewers

**We sincerely thank all the reviewers for their valuable time and their thoughtful comments and questions. We are encouraged that the reviewers find that**: - We introduce LOVA$^3$, which enhances MLLM training by incorporating two additional capabilities: question-asking and assessment (UkuU, 4XZr, 5aGT), drawn from human learning mechanisms. This approach is **insightful** (UkuU), **reasonable** (4XZr), and **well-motivated** (5aGT). - We have developed a novel benchmark for evaluating the accuracy of VQA triplets, featuring carefully selected question types that **bring valuable and lasting contributions** (5aGT). The creation of EvalQABench provides a **rigorous way to test and improve MLLMs** (UkuU). - We evaluate LOVA$^3$ against a variety of SOTA models across multiple tasks, demonstrating **robust improvements** (UkuU, 4XZr, 5aGT). The results offer **valuable insights** (UkuU) and are **reasonable** (4XZr). The ablation studies for various components **showcase their effectiveness** (5aGT). The experimental settings are **carefully designed and fair** (4XZr). To better review the details of the proposed EvalQABench and validate the reproducibility of this work, **we create an anonymous link at **https://anonymous.4open.science/r/LOVA3-9008/README.md** following the double-blind policy**. This link contains all the datasets we created for EvalQABench, as well as the training and evaluation codes and model weights. Please refer to it for further details. Finally, **we would like to emphasize again that our intention with this paper is to advance the multimodal instruction tuning of current MLLMs by introducing two intuitive and reasonable tasks: GenQA and EvalQA along with traditional VQA.** These additions are designed to enhance problem-solving capabilities. By focusing on these three tasks, **we demonstrate significant improvements across diverse benchmarks and offer insights** that can inspire further exploration and development in the field of multimodal instruction tuning. We trust that **our contributions will be thoroughly evaluated by the reviewers, as this training paradigm is genuinely novel** within the current multimodal instruction tuning domain. We attempted our best to address the questions as time allowed. We believe the comments have made the paper stronger and thank all the reviewers for their help. Please find individual responses to your questions below. ### References: [1] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension [2] Visual Instruction Tuning [3] Fuyu-8B: A Multimodal Architecture for AI Agents [4] Llama 2: Open Foundation and Fine-Tuned Chat Models [5] Improved Baselines with Visual Instruction Tuning [6] VQA-GEN: A Visual Question Answering Benchmark for Domain Generalization [7] CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization [8] Open-Vocabulary Video Question Answering: A New Benchmark for Evaluating the Generalizability of Video Question Answering Models [9] STAR: A Benchmark for Situated Reasoning in Real-World Videos [10] SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge [11] CinePile: A Long Video Question Answering Dataset and Benchmark [12] AssistGUI: Task-Oriented Desktop Graphical User Interface Automation [13] MEDIQ: Question-Asking LLMs for Adaptive and Reliable Clinical Reasoning [14] Reflexion: Language Agents with Verbal Reinforcement Learning [15] CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing [16] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond [17] CogVLM: Visual Expert for Pretrained Language Models [18] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [19] MMBench: Is Your Multi-modal Model an All-around Player? [20] MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities [21] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Response to Reviewer nUFy

Dear Reviewer nUFy, Thank you for taking the time to evaluate the ethical aspects of our work. We appreciate the reviewer’s concern regarding potential biases in our dataset and the ethical implications of our labeling process. We fully acknowledge the importance of addressing these issues to ensure the fairness and reliability of our results. We believe there may have been some misunderstandings, which we address in the following explanations. The first misunderstanding of our work we would like to clarify is that the EvalQABench is only for the EvalQA task which is adapted from the existing dataset VQAv2. There are no new datasets proposed for the GenQA task. In Table 1, the datasets listed are used for training the GenQA task with their original annotations. The construction of EvalQABench adheres to the VQAv2 dataset’s license (CC-BY 4.0). Bias: To mitigate biases, we randomly selected original VQA samples that exhibit no apparent biases from VQAv2. We are committed to transparency and have ensured that our methods align with NeurIPS ethical standards. Annotation Details: During the manual filtering stage, we asked two participants to summarize the error patterns in the generated negative answers and feedback. We then wrote code to automatically filter or correct these error samples. Throughout this process, we adhered to NeurIPS ethical standards, ensuring that no potential ethical issues were introduced. We will include details of annotation in our next version to provide further clarification. Thank you again for your valuable reviews of our submission. Please let us know if there are any further questions that need clarification. Warm regards, Author(s)

Response to Reviewer oUEn

Dear Reviewer oUEn, We sincerely appreciate you taking the time to review the ethical aspects of our submission. Warm regards, Author(s)

Please read the author rebuttal, other reviews and respond to the authors NOW!

Dear Reviewers, Thanks to those of you who already responded to the authors acknowledging the rebuttal and asking follow-up questions if any. Those who have not responded yet, please do the following ASAP: thoroughly read the rebuttal, the other reviews and respond to the authors about whether all your questions / concerns have been addressed or not. If not, please elaborate on which questions / concerns are still not addressed so that the authors have fair chance of addressing them before the author-reviewer discussion period ends in ~41 hours from now (August 13th, 11:59pm AoE). Your AC