VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use

Response to Reviewer Jd2M

We sincerely appreciate your valuable feedback and the acknowledgment of the **challenges addressed in our work** and the **comprehensiveness of our experiments**. Below, we provide clarifications to address your concerns: ## W1: Motivation for Including the Input Image in the Orchestrator Agent Prompt - Thank you for recognizing the significance of our ablation study. Empirically, we observe that including the input image allows the orchestrator agent to perform **higher-level visual understanding, facilitating better tool selection and specifying more accurate input parameters**. For instance, in a multi-view reasoning task, the orchestrator uses the image input to identify reference objects, enabling it to specify an appropriate input parameter 'focus' for the specialized agent to determine whether the camera is moving left or right. - Moreover, although SOTA VLMs struggle with fine-grained or low-level perception tasks, they are capable of extracting relatively high-level visual content. This capability enhances **grounded error handling and information aggregation**, especially when specialized agents or tools encounter **execution errors or hallucinate**. Without the image input, the orchestrator cannot perform such error handling, which degrades overall performance. ## W2: Tool Set Fairness in Comparisons - We appreciate your comment regarding the fairness of tool comparisons. Existing visual programming methods often **tightly integrate tool usage with their internal logic**, making it challenging to add or modify tools without disrupting their framework. In contrast, VipAct’s modular design enables **plug-and-play** tool integration with minimal effort, requiring only the specification of a standard Python function header, as detailed in Appendix H. - To address your concern, we conducted **additional experiments** where we integrated our vision expert models into baselines while preserving their original logic. The results shown in the following table indicate minor improvements in tasks such as depth estimation and object localization; however, the overall performance gap between these methods and VipAct remains **substantial**. Empirically, we observed that even with specific tools integrated, **significant errors persisted** due to limitations in the underlying logic of these frameworks. This highlights the importance of VipAct’s cohesive design, which integrates multi-agent collaboration and a structured approach to use vision expert models. | Condition | Sim | Count | Depth | Jig | Fun.C | Sem.C | Spat | Local | Vis.C | Multi-v | Overall | | :-------- | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ------: | ------: | | MM-ReAcT (with same tools) | - | 30.00 | 8.69 | - | - | - | 63.64 | 5.41 | - | - | - | | ViperGPT (with same tools) | - | 29.17 | 3.01 | - | - | - | 48.95 | 23.57 | - | - | - | | VisProg (with same tools) | - | 3.33 | 5.31 | - | - | - | 31.47 | 18.92 | - | - | - | | VipAct | 81.48 | 70.00 | 90.80 | 68.00 | 61.50 | 60.40 | 86.70 | 63.11 | 91.28 | 62.63 | 73.59 | ## Q1: Ablation Study on Visual Versus Textual Inputs - Thank you for this insightful suggestion. We conducted an ablation study comparing three conditions: 1. Image input. 2. Detailed textual description input: - Generated using GPT-4o with the prompt: “Please generate a detailed description of the following image [IMAGE]”. 3. Both image and description as input. - The results in the following table show that using only detailed textual descriptions **underperforms** compared to using the image input. While adding both image and textual descriptions yields slight improvements in some cases, the gains are **not significant**. Empirically, we observe that generated descriptions often fail to capture fine-grained details or background elements critical to certain tasks. Furthermore, descriptions can introduce biases, emphasizing objects over contextual elements, leading to suboptimal performance. Overall, this analysis reinforces the importance of direct visual input in enabling robust and grounded reasoning by the orchestrator agent. | Condition | Sim | Count | Depth | Jig | Fun.C | Sem.C | Spat | Local | Vis.C | Multi-v | Overall | | :--------------------------------- | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ------: | ------: | | Image | 81.48 | 70 | 90.8 | 68 | 61.5 | 60.4 | 86.7 | 63.11 | 91.28 | 62.63 | 73.59 | | No image or description input | 77.78 | 59.71 | 69.35 | 61.33 | 53.85 | 51.08 | 83.22 | 60.66 | 78.49 | 48.12 | 64.36 | | Description| 79.32 | 62.72 | 73.45 | 62.37 | 54.02 | 52.46 | 83.22 | 61.34 | 81.35 | 51.42 | 66.17 | | Image + description | 81.48 | 70.48 | 90.8 | 67.52 | 62.45 | 61.32 | 84.32 | 62.67 | 91.28 | 63.34 | 73.57 |

Official Comment by Reviewer Jd2M

Thank you to the authors for the detailed rebuttal and the expanded experiments. My concerns have been almost addressed. The experimental results in Q1 are quite interesting. However, I also agree with the reviewer rhXE that **VipAct did not provide very inspiring innovations** despite its improved pipeline over existing methods like VisProg. But don't worry, I will keep my rating as the authors' experiments and analysis are self-contained, which is also a merit.

Response to Reviewer rhXE

We sincerely appreciate your constructive feedback and detailed review of our work. Below, we address your concerns and provide clarifications: ## W1: Tested on only one LLM. Testing on more, e.g., Claude / Gemini, would be more convincing and show the generalization of the framework. - **Thank you for raising this concern.** - As outlined in **our response to Reviewer cte9**, we have **justified our model selection and conducted additional experiments incorporating Claude-1.5-Snoet and Gemini-1.5-Pro**. These results demonstrate the **robustness and generalizability** of VIPACT across multiple LLMs, addressing your concern comprehensively. ## W2: Limited contribution to the community, given existing work on “LLM with visual tool use” and multi-agent frameworks. - **We acknowledge prior work** on LLMs with visual tool use. However, our contribution lies in how these components are cohesively integrated to achieve superior performance and generalization on fine-grained visual perception tasks. Specifically: - **Novelty in Framework Design:** - **Table 1** highlights VipAct's unique features, including direct image input for planning and execution, iterative reasoning, multi-image support, and specialized multi-agent collaboration. These capabilities set VipAct apart from previous visual programming methods like MM-ReAct and ViperGPT, enabling superior performance and generalization in visual perception tasks. - **Advancements in Generalization:** - While previous visual programming methods rely heavily on predefined tools with limited adaptability (as shown in Table 2), VipAct incorporates a flexible and modular design, enabling it to generalize across diverse tasks. - **Extension of multi-agent interaction to Multi-Modal Domains:** - While multi-agent frameworks have shown promise in text-based tasks, their potential in **multi-modal settings has been underexplored**. Our work bridges this gap by demonstrating the effectiveness of multi-agent collaboration in fine-grained visual perception tasks. - We hope this addresses your concerns regarding the novelty and relevance of our contributions. ## W3: The performance improvement does not seem significant enough (less than 10%). - **We respectfully disagree with this assessment.** - As stated in **Lines 974-976 of the paper**, we have conducted **significant test** between our proposal and baselines, and the average p-value for comparisons between VipAct and SOTA baselines is 0.006 (<0.01), underscoring the significance of our improvements. - We argue that in real-world applications of fine-grained visual perception, less than 10% performance gains can lead to meaningful advancements, especially on complex datasets like Blink and MMVP. - Furthermore, VipAct demonstrates enhanced generalization and flexibility, offering a robust framework for future extensions in visual perception. ## Q1: Add an average column in Table 2 to better showcase the method’s improvements. - Thank you for this suggestion. We **have included an average column in blue in Table 2 in the revised version of the paper**, providing a clearer representation of the overall improvements achieved by our proposal.

Thanks to the reply

Thanks to the reviewer for the response and the extra experiments. They have addressed some of the concerns. Therefore, I raised my rating. However, I still think the paper is not good enough for accepted for ICLR (but may be worth being accepted to some other venues) due to 1. limited novelty and contribution. The proposed method basically expands the former visual programming / agentic methods based on new capabilities of current MLLMs like GPT4o and new visual tools like visual prompting. Also, some of the capabilities mentioned in Table 1 are explored in former papers in other tasks. For instance, planning with image input has been applied to web navigation tasks [1]. Iterative reasoning has been proposed by [2] for former VQA tasks. 2. The improvement over the zero-shot baseline on the MMVP dataset is only 2.7%, even with the fine-grained design (which means extra cost on computation and longer response time). [1] WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models [2] IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models

Response to Reviewer cte9

We appreciate your thoughtful feedback! We are pleased that you recognize the **novelty of our approach**, the **flexibility and extensibility** of ours, as well as the **extensive experiments** that demonstrate its **effectiveness, generalizability**, and importance in enhancing VLMs' ability to perceive detailed visual information. We want to address your concerns with the following clarifications: ## W1/3: Reliance on GPT-4o which may not generalize effectively - Thanks for highlighting this! As we acknowledged in **lines 534–537**, our main experiments rely on GPT-4o due to its superior instruction-following and function-calling ability. We **have explored** other VLMs in the original paper, such as LLaVA (details in **Appendix C**), but found that our agent framework can not be effectively applied to this model due to its bad instruction-following ability. As noted in **lines 341–343**, other multi-modal agent frameworks **also rely heavily on GPT-4o/V** [1][2] due to the significant performance gap between it and open-source alternatives. So we argue that building our proposal primarily on GPT-4o is reasonable for advancing cutting-edge research in multi-modal agent systems. - **Additional experiments:** To further address your issues, we conducted experiments with more VLMs, including **Gemini-1.5-Pro and Claude-3.5-Sonnet**. Results: ### Claude | Method | Sim | Count | Depth | Jig | Fun.C | Sem.C | Spat | Local | Vis.C | Multi-v | Overall | | ------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ------- | ------- | | Zero-shot | 85.19 | 67.50 | 66.13 | 58.00 | 58.00 | 44.60 | 72.03 | 57.38 | 73.84 | 48.12 | 63.08 | | CoT | 86.72 | 68.33 | 71.77 | 61.33 | 52.31 | 41.73 | 77.62 | 50.00 | 81.98 | 44.36 | 63.62 | | LtM | 87.42 | 67.42 | 68.42 | 59.97 | 58.00 | 45.13 | 73.82 | 57.47 | 74.29 | 47.86 | 63.98 | | ToT | 86.90 | 67.53 | 69.48 | 57.35 | 59.46 | 43.72 | 74.92 | 58.49 | 76.14 | 46.38 | 64.04 | | SoM | 82.65 | 62.78 | 63.81 | 56.79 | 56.73 | 39.58 | 72.00 | 52.47 | 73.74 | 44.63 | 60.52 | | VipAct (Ours) | 88.89 | 67.96 | 88.59 | 65.33 | 60.42 | 50.13 | 78.82 | 61.54 | 83.72 | 49.57 | 69.50 | ### Gemini | Method | Sim | Count | Depth | Jig | Fun.C | Sem.C | Spat | Local | Vis.C | Multi-v | Overall | | ------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ------- | ------- | | Zero-shot | 78.52 | 60.83 | 70.97 | 72.67 | 44.62 | 51.08 | 74.13 | 57.38 | 81.98 | 55.64 | 64.78 | | CoT | 81.48 | 64.17 | 78.23 | 68.67 | 42.31 | 53.96 | 78.32 | 60.66 | 81.98 | 51.88 | 66.17 | | LtM | 84.44 | 62.50 | 75.81 | 73.33 | 46.92 | 50.36 | 73.43 | 63.11 | 84.30 | 54.14 | 66.83 | | ToT | 82.23 | 61.52 | 74.84 | 68.96 | 45.42 | 52.61 | 75.34 | 61.25 | 82.21 | 52.86 | 65.72 | | SoM | 60.74 | 55.00 | 65.32 | 62.00 | 47.69 | 43.88 | 74.13 | 59.02 | 74.42 | 53.38 | 59.56 | | VipAct (Ours) | 84.44 | 64.17 | 89.42 | 74.00 | 48.74 | 57.55 | 79.57 | 70.48 | 86.05 | 59.42 | 71.38 | These results demonstrate that applying our VipAct **consistently outperforms** baselines and highlights the generalization of our approach when used with VLMs possessing strong instruction-following abilities. ## W2: Results of other models on the MMVP benchmark - **Additional experiments:** We evaluated Llava-one-vision-7B, InternVL-2-Pro, and Llama-3.2-90b-vision on the MMVP dataset. Results: | Method | LLaVA-OneVision | InternVL-2-Pro | Llama-3.2-90b-vision | | --------- | --------------- | -------------- | -------------------- | | Random | 25.00 | 25.00 | 25.00 | | Zero-shot | 29.67 | 60.00 | 57.33 | | CoT | 30.33 | 57.33 | 59.33 | | LtM | 30.00 | 58.67 | 57.33 | | ToT | 31.33 | 60.00 | 59.38 | | SoM | 27.00 | 45.33 | 51.33 | - These results show that open-source models achieve slightly above random performance. InternVL-2-Pro and Llama-3.2-90b-vision perform better but still lag significantly behind GPT-4o. - **Regarding Eagle model:** We do not provide the results of this as it was published after the cutoff date (August 28) and is not required for comparison ([Guideline](https://iclr.cc/Conferences/2025/ReviewerGuide)). Moreover, Eagle’s “expert” concept differs from ours: it uses integrated vision encoders, while VipAct incorporates external vision experts without altering the underlying model architecture. We have cited this and **discussed these distinctions** in the updated paper (Lines 110–116). ## Conclusion We believe we have addressed the issues and highlighted the updates in blue for clarity in the revised paper. [1] GPT-4V: A generalist web agent [2] WebVoyager