Large Language Models are Visual Reasoning Coordinators

Summary

1. The author proposes to utilize a language model as the coordinator between different outputs from different VLMs, leveraging their strengths for visual reasoning. 2. The proposed method achieves SOTA results on multiple visual reasoning benchmarks. 3. The analysis shows how zero-shot / fine-tuned language models can coordinate VLM outputs.

Strengths

1. The proposed method with zero-shot VLMs and zero-shot / fine-tuned LLMs as coordinators are novel. 2. The performance of the model is significant with sufficient experiment results.

Weaknesses

1. Different dataset requires different visual / reasoning capabilities and knowledge. The proposed method performs differently on different datasets. For example, the ZS version performs similarly with 'Ensemble' on VQA v2 and OK-VQA, but outperforms 'Ensemble' significantly on other datasets. It's better to analyze the reason. 2. Why the proposed method performs better than other methods needs more discussion. For example, comparing with BLIP-2 on VQA.

Questions

1. It is a little unclear that in Table 3, the Cola-FT model is fine-tuned on BLIP+OFA outputs or 3 OFA variance. Why is the Cola-FT performance here much lower than the performance in Table 2 and Figure 3?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are discussed and potential solutions are proposed.

Summary

The paper proposes an ensemble based approach to solve visual reasoning problems. The paper proposes to use an instruction fine-tuned large language model to integrate answers to visual reasoning problems provided by vision language models. The paper presents two variants of the aggregation model -- using fine-tuning and using in-context learning. The model is evaluated on the VQA v2, A-OKVQA, OK-VQA, e-SNLI-VE and VSR datasets.

Strengths

* The paper reports promising results on the VQA v2, A-OKVQA, OK-VQA, e-SNLI-VE and VSR datasets. * The paper includes a variety of ablations that show the effectiveness of the proposed method -- including model size, scaling, number of video-language models as ensemble members. * The paper includes qualitative examples which highlight the effectiveness of the proposed method. * The paper is well written and easy to understand.

Weaknesses

* While the results are very promising, it would be helpful to add results on more complex datasets such as CLEVR (CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning) and GQA (GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering) which study compositional reasoning. * The paper should also consider comparing to prior work such as Visual Programming: Compositional visual reasoning without training, CVPR 2023, which also uses an external large language model to coordinate vision / language-vision models. * The paper claims in L291 "This work demonstrates the first step toward applying language models for visual reasoning", but Flamingo (Flamingo: a Visual Language Model for Few-Shot Learning, NeurIPS 2022) already shows zero-show visual reasoning on datasets such as Next-QA.

Questions

* The paper should include further details on the computational resources used. L680 in the supplementary material just states that V100 or A100 GPUs were used, but the paper should include further details about the total computational resources used. * The paper should include further motivational details on why the particular datasets VQA v2, A-OKVQA, OK-VQA, e-SNLI-VE and VSR were used? Why were more datasets which require more complex reasoning abilities such as CLEVR and GQA were not used.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper does not include any discussion about its limitations.

Summary

The paper introduces a new paradigm called Cola, which aims to coordinate multiple vision-language models (VLMs) for visual reasoning tasks. While several VLMs have demonstrated strong commonsense reasoning abilities in different domains, effectively combining their capabilities remains a challenge. Traditional methods like ensembling struggle to achieve higher-order communications between these models.

Strengths

Cola proposes a solution by employing a language model (LM) to coordinate the multiple VLMs. The LM facilitates natural language communication, leveraging the distinct and complementary capabilities of each VLM. The authors introduce two variants of Cola: Cola-FT, which involves fine-tuning the models, and Cola-Zero, which performs in-context learning without the need for fine-tuning. The authors conduct extensive experiments to evaluate the performance of Cola on various visual reasoning tasks, including visual question answering (VQA), outside knowledge VQA, visual entailment, and visual-spatial reasoning. They demonstrate that Cola-FT achieves state-of-the-art results in these tasks. Additionally, Cola-Zero exhibits competitive performance in zero and few-shot settings, without requiring fine-tuning. The paper further includes ablation studies and visualizations to validate the effectiveness of the coordinator LM. These analyses confirm that the coordinator LM comprehends the instruction prompts and understands the individual functionalities of the VLMs, allowing it to coordinate their efforts and enable visual reasoning capabilities. Using a language model as a coordinator for different VLMs is novel. Adequate experiments show a promising performance over baselines.

Weaknesses

Does the language coordinator play a role to determine the correctness of comparing multiple VLMs according to the answering language description? What if both VLMs are wrong? As is mentioned in the limitation, no rational explanation or logical steps were applied either in VLMs or LM. The paper missed the recent work in visual reasoning, like Flamingo(https://nips.cc/Conferences/2022/ScheduleMultitrack?event=54165), STAR (http://star.csail.mit.edu/), GAMR(https://openreview.net/pdf?id=iLMgk2IGNyv), SHG-VQA(https://arxiv.org/abs/2304.08682), etc. Only two VLMs are applied. It’s interesting to see the margin gains of Cola when compared with the best VLMs in the experiments

Questions

Refer to the above comments

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Refer to the above comments

Summary

The paper introduces a novel approach to ensembling multiple vision-language models (VLMs) for solving visual reasoning tasks. More specifically, the authors propose to use a language model (LM) to coordinate answers from various VLMs, which outperforms traditional ensemble approaches. Multiple experiments demonstrate the effectiveness of the proposed Cola approach.

Strengths

1. The paper is well written, with great explanations of the proposed Cola approach and various figures and tables. 2. Using the LM to coordinate VLMs for visual reasoning tasks is novel and interesting.

Weaknesses

The authors observed that long input (multiple VLMs) does not guarantee higher performance in Section 3.5. Specifically, Figure 4 shows the approach is vulnerable to the number of models. Cola models may also be affected by which VLMs are used.

Questions

In Table 3, the authors implemented "Ensemble (average)" based on Equation (1). Was each $P_i(v, q)$ normalized before averaging them?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors discussed the limitations of the work in Section 5.

Update

The results on CLEVR and GQA are promising. The final version should discuss prior work in more detail and update claims accordingly. I would keep my score and vote for acceptance.

Sincerely Looking Forward to Your Reply

Dear Reviewer, Your suggestions and comments have greatly helped polish our paper, regarding the long input and the ensemble baseline. We sincerely look forward to your reply to our response, and we are open to any discussion to improve our paper. Best wishes, The authors

Sincerely Looking Forward to Your Reply

Dear Reviewer, We sincerely appreciate you taking the time to provide thoughtful suggestions and comments, which have been immensely helpful in improving our paper. In particular, your feedback regarding the coordinator LM and Cola with the best VLMs has enabled us to strengthen and clarify these sections. We have revised the writing and added experiment results to our paper. We sincerely look forward to hearing your perspective on our response. Please know that we remain open to any discussion that could further enhance our work, and we highly value your constructive input. Sincerely, The authors

Sincerely Looking Forward to Your Reply

Dear Reviewer, We extend our heartfelt gratitude for the invaluable suggestions and comments you have provided, which have significantly contributed to refining our paper. Specifically, your insights on the performance gap between Cola-Zero and ensemble baselines helped us better present our analysis of why Cola-Zero works. We hope our response has addressed your concerns. We eagerly anticipate your response to our revised submission and are earnestly open to engaging in any discourse aimed at enhancing the quality of our paper. Warm regards, The authors

Genuinely Thankful for Reviewers Feedback

Dear Reviewer, Thank you for your insightful feedback and suggestions - they have been tremendously helpful for improving our work. We hope that our efforts on this paper will provide value to the research community. We appreciate you raising the question about the performance difference between GQA and other datasets when using Cola. OFA was trained on the GQA dataset, while BLIP was not. This likely explains the substantial gap in their capabilities on this task, and why the Cola ensemble only led to a minor improvement overall (compared to OFA), as OFA was dominant in providing useful answers. We realize we should have provided more context about the VLMs' training data and capabilities in our paper to preemptively address this discrepancy. Thank you again for catching this oversight - your expertise and guidance have been invaluable for identifying areas we can clarify and strengthen the manuscript. We sincerely appreciate you taking the time to provide thoughtful feedback throughout the review process. Best regards, Authors

Looking Forward to Further Discussions

Dear reviewers, Thank you again for your valuable time and insightful comments! We sincerely look forward to your reply to our response to let us know if we have resolved your concerns, and we are open to any discussion to improve our paper. Best regards， The authors

Thank you for your responses

Hi Thank you for your responses. Although I have a different opinion on the first question, some of the questions were resolved by the responses. But there is no revised version of the paper on the system, please check if it is uploaded successfully. Will add scores to reflect the change.