Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

Summary

This paper proposes a multi-agent architecture for mobile device operation, Mobile-Agent-v2. Mobile-Agent-v2 includes three agents : planning agent, decision agent and reflection agent. To retain focus content, this paper designs a memory unit to record task-related focus content. The planning agent generates task progress based on the history operations, the reflection agent corrects erroneous operations and the decision agent outputs operations. Extensive experiments conducted on various operating systems, language environments, and applications shows that MobileAgent-v2 achieves significant performance improvements compared to single-agent architecture of Mobile-Agent.

Strengths

1. This paper introduces a multi-agent architecture Mobile-Agent-v2, which can alleviate various navigating difficulties inherent in the single-agent framework for mobile device operation tasks. 2. This paper designs a memory unit and reflection agent, which can avoid the loss of focus content navigating and reflection capability. 3. Experimental results demonstrate that Mobile-Agent-v2 achieves significant performance improvements.

Weaknesses

1. Insufficient baselines. More powerful baselines should be considered, such as AppAgent [1] and CogAgent [2]. 2. This paper lack description of how the decision agent retrieves the memory unit. In addition, the content or format of the memory unit storage is not included. [1] AppAgent: Multimodal Agents as Smartphone Users. 2023. [2] CogAgent: A Visual Language Model for GUI Agents. 2023.

Questions

1. Can the Mobile-Agent-v2 be generalized to IOS systems? 2. How operations knowledge is injected into the model?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors describe limitations in the paper.

Summary

This paper proposes mobile-agent-v2, a multi-agent framework for mobile device operation. The framework includes a planning agent, decision agent, reflection agent and memory unit. The multi-agent framework exhibits advantages in reducing errors in long-horizon tasks and achieves better results than single-agent framework on the evaluations designed in the aper.

Strengths

1. Multi-agent framework for mobile device operation is novel in the literature, improving the overall performance of single agent framework, with the effectiveness of each component verified.

Weaknesses

1 The paper lacks a comparison of the evaluation benchmark with previous works such as AppAgent and works mentioned in Sec. 2.2. This makes it difficult to evaluate the overall effectiveness of the framework in comparison with previous works. 2. The evaluation benchmark is relatively small and simple, and how they are evaluated is not very clear. Does the evaluation relies on human evaluator to check the success of each component in the trajectory?

Questions

1. See weakness, the evaluation details needs further clarification.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

1. This paper relies on commercial models such as GPT-4 and GPT-4V to complete the tasks. The cost of completing a task is uncaculated or discussed in the paper, but is expected to be more cost than the single agent framework. 2. The efficiency and accuracy trade-off. After introducing mutli-agent framework, the inference time should be much slower. -------- After the rebuttal, the reviewer think the heavy reliance on commercial models and the time and token cost should be discussed in the revision.

Summary

This paper introduces an agentic framework, Mobile-Agent-v2, designed to address the challenges of planning and sequential function/tool invocation for Large Language Models (LLMs) in mobile operation scenarios. Mobile-Agent-v2 comprises three agents: a planner, an actioner, and a reflector, which jointly enhance the performance of LLMs in mobile operation tasks. Experimental results demonstrate the advantage of Mobile-Agent-v2 over the single-agent framework: Model-Agent, across various real-world applications.

Strengths

- It is well-written and the structure is clear. - Its studied problem of how to design agentic workflow for LLM to improve planning and sequential tool calling is beneficial to the community. - The experimental results on real-world mobile apps look promising.

Weaknesses

- The technical contribution seems limited. Its proposed framework of using planner, actioner and reflector for agentic workflow is a common practice. - The evaluation misses some important ablation studies.

Questions

This paper proposes an agentic workflow, comprising a planner, actioner, and reflector, to enhance the planning and sequential tool-calling capabilities of Large Language Models (LLMs) in mobile operations. The motivation behind this approach is sound, and the experimental results appear promising. However, I have several concerns primarily regarding the technical novelty and the insights derived from the experiments. - The method of decomposing the responsibilities of a single agent into multiple components—planner, actioner, and reflector—is not novel. This approach has been previously proposed by [1] and is widely used in other contexts. Therefore, it is essential to clarify the key challenges and new insights associated with applying this framework to mobile operation scenarios. Without such clarification, it appears that the study merely leverages an existing method and applies it to a specific scenario without significant innovation. - The experimental results across various mobile applications are promising. However, the sequence length and type are critical factors that warrant detailed discussion. While some analysis is provided, the paper lacks comprehensive results regarding task completion accuracy when sequence length and type vary. For instance, it would be beneficial to understand, across different apps, which types of operations are more successfully completed with different tool sequences. These ablation studies are necessary to offer deeper insights.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

In this paper, the authors present Moble-Agent-v2, a multi-agent architectruure deisgned to assist mobile device operation. Mobile-Agent-v2 comprises three agents: planning agent, decision agent, and reflection agent. Frist, planning agent summarizes the task progress based on the operation hsistories. Then, the decision agent generates next operation to be executed. Finally, the reflextion agent classifies whether the excuted operation as correct, errorneous, or ineffective. For evaluation, the authors conduct experiments on real mobile devices in various mobile applications.

Strengths

- The application of multi-modal LLMs in mobile operation is intriguing - The framework that uses multiple agents demnostrates superior perfornamce compared to the single-agent framework - The paper is easy to follow and evaluation is conducted on the real mobile devices

Weaknesses

- The primary concern is the novelty of the paper. While using multiple agents for mobile operations appears novel, the complexity of tasks used in the evaluation suggests that the implementation of each component within Mobile-Agent-V2 is relatively simple. It would be benifical if the authors evalute the proposed framework in diverse user scenarios (refer to the limitation section). - It appears that the authors assume that LLMs possess the knowledge about mobile applications. For example, if the user instruction is "turn on dark mode," how does the LLM determine which setting (general setting, display setting) the agent should execute? Because, sometimes even for the human need the trial and error to execute certain task. Without grounding the available execution in the given context, the proposed method might require multiple trial and error until it successfully completes the task. - Regarding to the upper question, it would be better if the authors provide how many interactions are required to complete a task compared to the optimal execution. - It seems to be neither erroneous nor ineffective operations are recorded in the operation history. If this is the case, isn’t there a potentional risk that the decision agent might repeat the same operation? Woudn’t it be better to use these operation histories to prevent the LLM from repeating erroneous or ineffective executions? - There seems some potential risks associated with inaccuracies in every agents within the proposed framework, especially since it utilizes LLMs. For instance, the planning agent might inaccurately summarize operation histories, the decision agent could misunderstand visual inputs, or it might generate incorrect executions. Can Mobile-Agent-v2 automatically identify which component is responsible for an error? or could the authors provide more statistical information about this?

Questions

- Can the authors provide more information about knowledge injection? It would be helpful if the authors included examples to illustrate this process.

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

- Including some discussion on the scope limitations of Mobile-Agent-v2 would be beneficial. For instance, what happens if the required application is missing—can Mobile-Agent-v2 suggest an alternative method? Additionally, for ambiguous instructions, is Mobile-Agent-v2 capable of resolving these by seeking clarification from the user?

Summary

The paper titled "Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration" presents a multi-agent architecture designed to address the challenges of mobile device operation tasks, specifically focusing on task progress navigation and focus content navigation. The proposed system includes three agents: a planning agent, a decision agent, and a reflection agent. The architecture aims to improve task completion rates and operational efficiency compared to existing single-agent architectures.

Strengths

**Innovative Multi-Agent Architecture:** The introduction of a multi-agent system to handle different aspects of mobile device operation is a novel approach. It effectively distributes the workload among specialized agents, which likely contributes to improved performance. **Comprehensive Experimental Evaluation:** The paper includes a detailed evaluation across different operating systems, language environments, and applications, providing robust evidence of the system's effectiveness. **Significant Performance Improvements:** Experimental results show that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to a single-agent architecture, demonstrating the practical benefits of the proposed system.

Weaknesses

**Limited Discussion on Scalability:** There is insufficient discussion on the scalability of the proposed system. The paper does not address potential challenges when scaling the architecture to more complex tasks or larger datasets. **Potential Overhead of Multi-Agent Coordination:** While the multi-agent approach shows improved performance, the paper does not discuss the potential computational overhead and complexity introduced by coordinating multiple agents, which could be a significant drawback in resource-constrained environments.

Questions

- How does the proposed system handle the increased computational overhead associated with running multiple agents concurrently on resource-constrained mobile devices? - Can the system be extended to support more complex multi-app operations that require intricate coordination between different agents? - How does the reflection agent determine the appropriate corrective measures for erroneous operations, and what is the success rate of these corrections in practice?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

see weakness

Official Comment by Reviewer 4qiL

Thank you for taking the time and effort to address all questions. There are a few follow-up questions. a) Could you provide more details about the experiments comparing the performance of Mobile-Agent-v2 and AppAgent, such as the environment, setting, number of evaluation tasks, etc. b) How is the storage and retrieval costs of memory cells?

A few further questions

Thank the authors for the detailed response. However, the reviewer is still confused about whether the successful judgment is done by humans or GPT-4V. If it is by GPT-4V, to what degree is the assessment reliable? As mentioned in lines 215-216 of the main paper, there are 88 instructions tested in total, with only 8 instructions for multi-app operations, what is the relationship between the so-called 1500+ evaluation with the 88 instructions? Mobile Agent-v2 seems to increase the overall time to finish the task in comparison with Mobile agent-v1, not to mention it is impractical to use GPT-4V for mobile devices. Further, the time calculation does not consider the information transmission time through the web or cable in real-world applications when using GPT-4V.

Response to the Official Comment by Reviewer Xdcr

Thank you very much for your feedback and the time you dedicated to reviewing our paper. We greatly appreciate your insights and are pleased to hear that the clarifications we provided were helpful in addressing your concerns. If you have any further questions or suggestions, please do not hesitate to reach out. We would be happy to discuss them with you.