Exploring Question Decomposition for Zero-Shot VQA

Summary

This paper studies the question decomposition of VQA. To begin with, the authors demonstrate that large VLMs equipped with LLMs can effectively leverage question decomposition to improve answering performance. Thereafter, some experiments are conducted to prove that instruction-tuned language models are capable of auto-writing decompositions. Since some questions do not require decomposition to reach the right answer, the authors then propose a novel selective decomposition method to selectively decide which questions demand decomposition.

Strengths

- The paper is well-written and well-organized. Most parts of this paper are easy to follow. - The studied problem is interesting and novel. It helps us to understand the in-context learning capability in large VLMs with instruction-tuned LLMs. - To validate the effectiveness of question decomposition, the authors perform comprehensive experiments.

Weaknesses

- My biggest concern lies in Sec. 4 about whether large models can produce effective decompositions. As shown in Table 2, most $E_{CR}$ values are pretty low while $E_{IC}$ are relatively high. According to the definitions of these two terms, it seems that most decompositions severely hurt the model performance. For example, 10% $E_{CR}$ means only 1 of 10 vqa instances can be corrected by the recomposer. In contrast, the average value of $E_{IC}$ is around 40. This means 4 of 10 vqa instances after decomposition failed. - For Sec. 5, have authors tried these newly introduced metrics after the selective decomposition? - Have the authors considered using the GQA dataset [1], which involves a longer chain of reasoning? - About Table 1, the oracle/no answer outperforms oracle/self-answer by a large performance margin. This result does not make much sense since using more context usually leads to more accurate answer predictions. - Moreover, the scrambling is not well-explained. How many words are perturbed? This question is important because a larger perturbation often causes a severe performance drop. In addition, why the authors have not tried scrambling the questions? This is also essential because the well-known bias problem in VQA is from question key words and answers. Typo: - Fig. 1 is not referred in the main manuscript. [1] GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In CVPR 2019.

Questions

Overall, I believe this is a good paper. However, the effectiveness of question decomposition from large VLMs is confusing. If the authors can address this concern, I will raise my initial score.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

To answer the questions that require reasoning or outside knowledge, this paper proposes a question decomposition method which uses LLM to infer subquestions

Strengths

1. The proposed method is intuitive and interesting. 2. The methods conduct comprehensive experiments on BLIP2 and FlanT5, and the results clearly demonstrate the effectiveness of the proposed methods.

Weaknesses

1. While this research paper serves as an investigative study, it only validates the proposed methods using FlanT5 and BLIP2 models. This limited scope raises questions about the generalizability of the proposed methods. It would enhance the study's breadth if further tests were conducted on a diverse range of zero-shot VQA methods, such as Flamingo [1], Frozen [2], and Img2LLM [3]. Additionally, incorporating various Language Learning Models (LLMs) for decomoposing questions like OPT and the GPT series would further strengthen the results. 2. I acknowledge that the proposed methods are primarily designed for tasks necessitating reasoning. However, given the community's growing interest in open-ended environments, it may be beneficial to conduct some analysis on straightforward VQA tasks. For instance, the experimental results on tasks like VQAv2 or GQA could provide valuable insights. 3. The proposal of self-generated question-answer pairs into VQA tasks is not a novel concept. To improve the understanding of how this method compares with prior approaches, a dedicated section comparing it with earlier methods such as VQ^2A [4], Weak VQA [5], and Img2LLM would be beneficial. [1] Alayrac, J. B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., ... & Simonyan, K. (2022). Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35, 23716-23736. [2] Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S. M., Vinyals, O., & Hill, F. (2021). Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34, 200-212. [3] Guo, J., Li, J., Li, D., Tiong, A. M. H., Li, B., Tao, D., & Hoi, S. C. (2022). From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846. [4] Changpinyo, S., Kukliansky, D., Szpektor, I., Chen, X., Ding, N., & Soricut, R. (2022). All you may need for vqa are image captions. arXiv preprint arXiv:2205.01883. [5] Banerjee, P., Gokhale, T., Yang, Y., & Baral, C. (2020). WeaQA: Weak supervision via captions for visual question answering. arXiv preprint arXiv:2012.02356.

Questions

Please refer to "Weaknesses"

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The proposed methods lacks the comprehensive analysis on different zero-shot VQA models and LLM models, which raises the concerns about the generalization ability. In addition, the paper should also conduct some analysis on straightforward VQA tasks to demonstrate the generalization ability of the proposed methods with different question types.

Summary

The paper examine whether the current instruction-tuned VLM (e.g. BLIP2 ) can (1) learn to generate simple perception questions that can help to answer a more complex reasoning visual question; (2) learn to actually benefit from the perception questions. The conclusion is that for (1): language models (with various sizes) can learn to produce effective decompositions just by viewing examples and adding image as content is not being super helpful. for (2): Yes, the performance of the reason question answering is better with golden perception question even without providing answers to the perception question; and achieve good results with selective Decomposition

Strengths

(1) The paper provide very detailed discussion on how the flan model can be used to generate perception questions based on only 2 exemplars. The results are very interesting and surprising in that (a) even small size model can provide "decomposition" (sometime gibberish) that helps the recomposer model and visual input does not help a lot (2) test the selective decomposition that actually using the generated perception question to boost the reasoning question without having to know whether the question is requiring the perception question (3) The paper is well-written and easy to follow

Weaknesses

(1) It is very surprising that the visual input does not help generate better perception questions. For example, in the supplement material, the reasoning question is "is it cold outside" and the perception question is "are any people wearing jacket". The perception question is very easy to be generated if we see a person in the picture, but without visual content, there could be lots of perception question that are valid, e.g. "is there snow", "are there green leaves on the tree", "is the lake frozen" that may be irrelevant to the visual question My hypothesis is that the image + text model does not use the image correctly in that the question generation process is not focusing on the image. I wonder if it is easy to see (a) adding prompts that explicit say the perception question should be based on the image (b) converting the image into multiple captions and use the captions as visual inputs (2) It seems the final recomposer should be able to take multiple perception questions as input, it would be good to sample multiple perception questions from the decomposer and use all of perception questions to see if it helps the performance.

Questions

Please comment on the weakness part ------------------------------------ post rebuttal: I read the author's rebuttal and still willing to support the paper as my initial rating

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors adequately addressed the limitations

Summary

In this paper, the authors focus on question difficulty in visual question answering (VQA). In particular, they propose to give subquestions that improve the percentage of correct answers to answer inferential questions more correctly. Using a large-scale vision-language model (VLM), the authors show that giving an oracle subquestion first improves the accuracy of the VQA. They also propose a selective decomposition for solving VQA so that the VLM model generates proper subquestions. This method is shown to improve VQA accuracy on multiple datasets.

Strengths

- The authors utilize a large VLM to decompose questions to improve accuracy for VQA. - Validation using oracle subquestions and their answers demonstrates the contribution of question decomposition. - The effectiveness of the proposed method is shown for multiple parameter sizes and datasets.

Weaknesses

- There is little technical novelty in the proposed method in this paper; it only provides an effective prompt for VQA. - Furthermore, as the authors acknowledge, despite the claim of question decomposition as its prompt, it also generates many subquestions that do not seem related to the question. As it stands, it remains to be seen why the accuracy of the VQA was improved. - The fact that the improvement in accuracy was reported only for BLIP-2 also makes it unclear whether such improvement by question decomposition is common to VLM models in general or whether it is a phenomenon found only in BLIP-2.

Questions

- Please demonstrate that the proposed approach improves the accuracy of VQA in common with multiple VLM models. - In doing so, please quantitatively evaluate how many subquestions are generated that are not relevant to VQA, e.g., through user studies. Also, please verify whether such subquestions with little relevance contribute to improving the accuracy of the VQA and provide a discussion. - There are a few logic gaps, confusing parts, and areas that need proofreading. - The second line says "unlike natural human question-answering strategies," but the authors need to write the particular human strategies to lead to the subsequent statements. - In Figure 1, there is no ground truth answers, so readers do not know if all the answers shown as After are correct. Alternatively, state that all After answers are correct. - Only the description of Oracle/Oracle (Scrambled) in Table 1 is not present in the caption. - While the VQA system is initially shown as f(v,q)=a, the function f(v,q,(q_i, a_i)) appears in the middle with the same function name without any declarations, causing confusion to the reader. In addition, while the initial discussion includes multiple subquestions and answers for the final answer, the middle of the discussion refers to only a single subquestion. Please clarify how many subquestions are generated and provided. - Please specify what decomposer_parameters is in Figure 4; if it is the number of decomposer parameters, it is odd that it is a decimal. - There is a mysterious indentation on line 234. - In line 267, in the "yes / no" section, there is an extra space before and after the slash. - The columns named Acc in Tables 3 and 4 seem unclear what Acc this is. - The formatting of the items in the References is not consistent. There are also references that are cited as arxiv preprints even though they have already been published.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors provide an adequate description of the scope and limitations of the proposed method in Section 2.4.

Summary

This paper presents an investigation of a 1-step question decomposition strategy for VQA. An example of a decomposition is the sub-question "What is the name of the plane company?" for the visual question "What country headquarters this plane company?" (Figure 1). This paper finds: - Including human-annotated decompositions in a vision-language model (VLM) prompt leads to improved performance on VQAv2. - LMs/VLMs can generate decompositions that also lead to improved performances on four diverse VQA datasets (A-OKVQA, ArtVQA, OK-VQA, and SLAKE). - A "selective decomposition" strategy improves performance (i.e. where a model only generates decompositions when it originally produces a low confidence answer), as it should avoid decomposing "harder to decompose" questions.

Strengths

This paper presents several experiments for motivating and investigating the presented method. I believe it is a moderately important research contribution and is a sufficient package for publication. - The concept of question decomposition (and its use in prompting to improve performance in compositional question answering) is inspired from works in NLP using language models. However, question decomposition has not yet been investigated for visual QA tasks. Prior to this work, it was an open question whether LM/VLMs could decompose *visual* questions or make use of visual question decompositions. - The paper is generally well written, answers its research questions or demonstrates clear improvements for each experiment, and benchmarks on a diverse set of VQA datasets.

Weaknesses

- This paper should have also measured the decomposer method (Table 2) on VQAv2. Then, we could have compared with the results from using human-annotated decompositions in Table 1. - I appreciate that this paper measures text-only biases in their experiments. But I think it is also important to particularly measure the textual biases of *only* generating decomposed questions or the final VQA answer. - It would have been nice to also see ablations from a more powerful (e.g. 100B-param, code-pretrained) language model to observe an higher bound for the text-only experiments in this paper. I feel that this paper needs to be a little more careful with some claims: - It is hypothesized that "zero-shot task decomposition techniques is likely a property of model scaling" (L281-282) and this directly informs research question 1 in Sec. 4. However, the cited paper [42] claims that techniques similar to this (e.g. chain of thought, least-to-most prompting) emerge at 60-200B parameters. This work compares models with 11B or fewer parameters. As the decomposition is limited to one step in this work, smaller models (e.g. 3B or 11B size) may plausibly be capable of this task, but the details are not yet clear. Finally, while this is one hypothesis, another line of work suggests that the capability comes (not from model scale but) from pre-training with code (Fu 2022). - Technically, the method in this paper is few-shot, since in-context examples are provided (L139). - I would adjust the following claim (L133-134): "It is unlikely the model has ever seen the exact task of decomposition-aided visual question answering". It is hard to make claims about the specific data that LMs are trained on. - I am not confident in the result from the "Oracle (Scrambled)" experiment (Table 2, Sec. 3). Do we know that the model is simply not able to override its priors and handle scrambled text effectively after just a few in-context examples? Is this decomposition "primitive" or can it be re-scrambled into multiple compositions? I think it's fine to exclude this experiment, as the text-only ablations are likely sufficient. - "Decomposition appears to be a primarily linguistic ability" (L207) is not necessarily generally true. This has only been shown on this particular set of datasets and perhaps this is an observation of the qualities of existing VQA datasets. All being said, I believe adjusting these claims in-text will be sufficient and, although my concerns are detailed, I do not believe these entail any serious methodological issues. References: - Yao Fu, Hao Peng and Tushar Khot. (Dec 2022). How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources. Yao Fu’s Notion. https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 - (Maybe there is a printed citation for this, but my point is just that there is an alternate hypothesis, as illustrated here.)

Questions

- How many in-context examples are provided to the model? Have the authors experimented with different numbers of examples? Are these examples possibly distractors when provided to VQA models, as they are irrelevant to the provided image? - Do the authors know what % of sub-question predictions are accurate (in "Oracle / Self-Answer")? Regarding text: - Probably better to separate or re-group the rows in Table 1 into different experiments for clarity. I also observe that the improvement gained from using the image in VQAv2 diminishes slightly when using a decomposition and this should probably be mentioned (regarding textual biases of decompositions). - Would the authors consider rewriting L113-115 more clearly? - Regarding L307-309, aren't there works that indeed do multi-step question decomposition (like [39])?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- This paper formulates decomposition with multiple sub-questions, but only tests one step decomposition. I think saving multiple steps for future work is fine, although the text is slightly misleading. Can just be more up-front that this paper is focusing on one step. But why didn't the authors at least test >1 step human-annotated decompositions from Selvaraju et al. [29] for VQAv2? Seems like low-hanging fruit. - We should be more clear that there may be more than one good decomposition for a visual question. - Is a decomposition really "good" by the proposed metric? L272-273 indicates that generated decompositions may be gibberish but still lead to improved performance. A more detailed human study is probably necessary for more clarity. - Recent works (e.g. Kamalloo 2023) indicate that existing QA (and likely VQA metrics), which are based on lexical similarity, correlate poorly with human judgements for open-ended text generations. So I believe "goodness" metric here (L100-101), which relies on exact match, may be unsuitable. For that reason, it might have been better to compare the log probabilities of generating the ground truth answer instead (L97-98). - I also observe that this paper did not specify which A-OKVQA setting (multiple choice or direct answer) it used. It is likely the latter, but this dataset's official metric is the VQA metric (i.e. comparing the model prediction to 10 human annotations). Would the authors please clarify which answer they compared against for their exact match setup? References: - Ehsan Kamalloo, Nouha Dziri, Charles Clarke and Davood Rafiei. “Evaluating Open-Domain Question Answering in the Era of Large Language Models.” ArXiv abs/2305.06984 (2023).

Unmarked column in table is increase in accuracy (absolute)

Due to a LaTeX typo, the column name of the column indicating accuracy was not rendered. Sorry!

Summary of rebuttal phase

### Reviewers found our work novel, interesting, and surprising. - "The studied problem is interesting and novel. It helps us to understand the in-context learning capability in large VLMs with instruction-tuned LLMs." (TpMW) - "The proposed method is intuitive and interesting." (37W7) - "The results are very interesting and surprising" (zebn) - "Prior to this work, it was an open question whether LM/VLMs could decompose visual questions or make use of visual question decompositions" (yYni) ### Reviewers asked us to demonstrate our results with more LLMs and VLMS — and we did. Our evaluation suite now spans 3 VLM families, 9 datasets from different domains, and 3 LLM families, with **over 300 total experimental configurations evaluated**. Please see Table 8 and 9 in the attached PDF, which are similar to Table 3/4 in the main paper. Selective decomposition continues to work and produce net gains in VQA Accuracy for all configurations tested. ### We were asked for a human evaluation, and we conducted one. Reviewers wanted to know what percentage of subquestions would be considered "good" by humans, and whether it is possible for even irrelevant questions to have corrective effects. We conducted a human study to answer both questions: - For all decompositions, the $\%$ of relevant subquestions is $0.61 \pm 0.1$. - \% of subquestions answered correctly $0.77 \pm 0.6$. - For decompositions which successfully correct a wrong answer (wrong $\rightarrow$ right), $0.72 \pm 12$ were annotated as relevant. - For decompositions which induce an error (right $\rightarrow$ wrong), $0.45 \pm 13$ were annotated as relevant. ### Reviewers, thank you for your detailed feedback!