AVIS: Autonomous Visual Information Seeking with Large Language Model Agent

Summary

1. This paper proposes a novel VQA framework that uses LLMs to select external tools in multiple stages to extract the necessary information to answer the question. 2. The authors gather human decision data to develop a decision graph and construct in-context examples to guide the LLM to perform API selection and query construction. 3. The experiments validate the effectiveness of the proposed method on two knowledge-based VQA benchmarks.

Strengths

1. The experiment result is solid and shows the advantage of the framework and the improvement of each sub-module. 2. The guidance for tool selection from human demonstration is novel, and the huge improvement also gives us some insight into the weakness of LLMs.

Weaknesses

1. The LLM used is PALM, while the compared methods used GPT series models. It's better to have some results based on GPT models to demonstrate the effectiveness of the proposed framework. 2. The proposed human guidance requires a certain amount of human labor and may be hard and costly to generalize to other tasks using LLMs (manipulation, navigation, etc.).

Questions

1. It will be interesting to see how the framework performs on zero-shot setting and related discussions, especially the related work ViperGPT is zero-shot. [1] ViperGPT: Visual Inference via Python Execution for Reasoning

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation and potential solutions are discussed.

Summary

This paper proposes a VQA system that mimics the human decision-making process and leverages LLM and web searches to perform multimodal reasoning. The system consists of three main components: a transition graph, an LLM planner, and an LLM Reasoner. The transition graph is manually designed based on the human decision-making process for knowledge-intensive VQA. It guides the system to infer the answer from different states of information. The transition graph enhances the interpretability of the inference process and allows the system to focus on the current state only. The transition graph also introduces uncertainty to the decision procedure, making it more dynamic than previous methods that use static plans for tool usage. The LLM planner decides the next action (API calls and query content) based on the current state. The LLM Reasoner produces the answers from the collected information. Experiments demonstrate the effectiveness of this method on Visual-Question answering tasks OK-VQA and Infoseek.

Strengths

Leveraging the user study to build a state transition graph is a reasonable way to construct LLM applications, especially for well-understand applications such as VQA or specific expert domains. The motivation is clear and the writing is easy to follow. Experiments are well-designed to show the effectiveness of the method.

Weaknesses

Lack of error analysis.

Questions

How does the system handle the situation when the output is not informative and the pipeline gets stuck in an infinite loop? Regarding the weakness: The model improved the performance on the dataset, but it is still far from perfect. It would be helpful to analyze the sources of the errors (Prompts? LLM? transition graph?) which will benefit the community.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

This paper acknowledges the limitations of the model: 1. it is computationally intensive, and 2. it is currently only suitable for the VQA task. negative societal impact: not found

Summary

This paper introduces an autonomous framework for visual question answering framework named AVIS. AVIS utilizes a Large Language Model (LLM) as its core component to dynamically strategize the utilization of external tools. The framework comprises three key components: the planner, reasoner, and working memory. The planner determines the appropriate actions, such as selecting relevant APIs, in each step. The working memory stores the results obtained from executing the APIs, while the reasoner processes the outputs derived from the API calls. Experimental evaluations conducted on the OK-VQA and Infoseek datasets validate the effectiveness of the AVIS framework.

Strengths

1. The authors present a novel framework that leverages external tools to overcome the challenge of external knowledge dependency in Visual Question Answering (VQA) tasks, thereby enhancing the practicality of VQA in real-world scenarios. 2. The authors introduce a dynamic decision process for selecting the most suitable external APIs to address specific sub-problems at each step. 3. The authors utilize human decision-making data to construct a transition graph, which is helpful for narrowing down the action spaces in the decision-making process. 4. Extensive experiments on the OK-VQA and Infoseek datasets demonstrate the effectiveness of AVIS.

Weaknesses

1. AVIS necessitates the utilization of costly human-annotated data to guide the (LLM). To further substantiate the generalizability and practicality of AVIS, it would be advantageous to conduct experiments on the A-OKVQA[1] dataset, which incorporates the OK-VQA and Infoseek human-annotated data as in-context samples. 2. As indicated in Line 5 of Algorithm 2, the authors directly incorporate the outputs into the working memory. However, the inclusion of uninformative outputs may potentially impact the obtained results. 3. For the three baseline methods, it would be better to demonstrate the few-shot COT prompts. 4. In Line 250, some human-annotated samples may be “Could not find the answer”. How do the authors leverage this type of sample? When using this type of sample as an in-context sample, does it lead to the models rejecting the question when generating a response? 5. It would be advantageous to provide an average step count that AVIS typically requires to address a single VQA sample. 6. In Line 300, “Table 4” should be “Table 1”.

Questions

Refer to Weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/a

Summary

This work aims to more general VQA task (often needs external knowledge) via LLM-based information seeking. First, the info-seeking system is built with three components: planner, reasoner, and memory, seeking useful information with external tools/APIs. Second, they build a dataset with human decision-action samples via user study. Then, they build the transition graph from the dataset, for depicting the user state and action, which can be used as contextual instances serving the info-seeking system.

Strengths

The info-seeking system is designed in the reasonable and mainstream VQA/QA fashion.

Weaknesses

How to compare this work with the end-to-end frameworks? As LLMs growing gradually, along with the visual knowledges being grounded better and better with language knowledges, the external knowledges will become thinner and an inclusion of internal ones. In general, this is a good paper with much groundwork. I am looking forward author feedback and better version to persuade me.

Questions

How to avoid potential infinite loop such like those emerge in AutoGPT? Considering, rather than powerful LLMs, the capacity bottleneck is often caused by external tools.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None.

Any further questions about rebuttal?

Dear reviewers: Thanks again for proposing all the constructive comments and questions in your reviews. Please let us know whether you have any further questions or concerns regarding to our responses.

Response to Rebuttal

Thank you for the response. I have raised the "rating" score to 6.

Response to Reviewer

Thanks reviewers for the support to our work. We will definitely include the discussion with ViperGPT into our paper.

Any further question for discussion?

Dear Reviewer: As the author-reviewer discussion period is coming to an end, we wonder whether our response (especially the usefulness of external knowledge) has addressed your concerns? Looking forward to the further discussion regarding any more questions.