DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

Summary

This paper shows the potential of Reinforcement Learning (RL) for designing an effective digital agent for in-the-wild control through Graphical User Interfaces (GUIs). The proposed approach relies on the advantage of the pre-trained visual language models (VLMs) while tackling real-world stochasticity by training an RL agent that interacts with an environment instead of relying on static demonstrations. Accordingly, this work proposes a novel autonomous RL approach, namely DigiRL, for training device control agents, which consists of two stages: an offline RL phase where the agent is trained on static demonstrations, followed by an in-the-wild, offline-to-online RL stage for training the agent through interacting with an environment. Consequently, this work also introduces a scalable and parallelizable Android learning environment with a reward model (evaluator) based on a robust VLM-based model. To show the effectiveness of the proposed method, an evaluation of different tasks given diverse instructions is carried out from the Android in the Wild dataset on real Android device emulators. The results show a significant improvement of DigiRL compared to the existing state-of-the-art agents, including wrapped proprietary VLMs such as GPT-4V and Gemini 1.5 pro. The paper claims to be the first to succeed in developing an autonomous offline-to-online RL approach to enable state-of-the-art performance on device control problems.

Strengths

- The usage of RL in designing a successful digital agent for a device control task is fascinating. - I appreciate implementing such a scalable Android learning environment, and I hope the authors will open-source everything so that other researchers can reuse it. - The clarity of the paper is worth mentioning. - The experimental results section is rich, especially the ablation studies.

Weaknesses

- Although the POMDP definition sounds correct when defining the problem, a contextual POMDP is even more appropriate for such a problem [1]. - A pseudo-code or an illustrative diagram should be added to facilitate understanding the method. - It is not clear how the policy and the value network are conditioned given the context $c$. (implementation-wise) [1] Hallak, Assaf, Dotan Di Castro, and Shie Mannor. "Contextual markov decision processes." arXiv preprint arXiv:1502.02259 (2015).

Questions

- How are the policy and the value network conditioned given the context $c$? - Is the RL agent trained in a multi-task fashion (which I believe is true)? I mean the agent is trained with more than one task concurrently accessing fully or partially the same models. - If yes, do you think classical Multi-task learning or Multi-task reinforcement learning approaches would enhance the performance even more?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

I believe the authors discussed the method's limitations in the final section. I agree with the authors regarding the impact of such application on the economy, society, and privacy, and that needs careful review in the future to limit any harm.

Summary

This paper introduces a novel autonomous reinforcement learning (RL) approach, DigiRL, for training in-the-wild device control agents. DigiRL first employs offline RL to fine-tune a pre-trained vision-language model (VLM as the agent) using stale task-specific data, and then further refines the agent through online RL by continuously interacting with parallelized emulators. DigiRL achieves a 49.5% absolute improvement in task success rate over existing state-of-the-art agents, establishing a new benchmark for digital agents in device control.

Strengths

1. The paper is well-written and motivated, many important technical/implementation details are covered. 2. The paper considers a challenging problem, autonomous device control, where existing LLM-based methods struggle to achieve acceptable success rate. The proposed method leverages VLM and RL techniques and significantly improves compared to these baselines. 3. The experiments are comprehensive and informative, covering LLM/RL agents, prompting and learning paradigms, offline and off-to-on RL, as well as failure modes analysis. 4. The authors implement a multi-machine emulator system to support parallel and real-time training of online RL.

Weaknesses

Major Points: 1. From a ML methodological point of view, the novelty/contribution of the paper is limited. To perform offline and off-to-on RL, the paper adopts a number of existing techniques such as AWR, doubly-robust estimators with little customization (e.g., hard filtering on the advantages instead of computing $\exp(A)$, which is mainly indended for easier implementation), all well-known to the community. The only thing seems "new" is training value functions with cross-entropy losses, also directly taken from [1], and the equations in line 250-251 seems to be qustionable (see my comments in the Questions section). Moreover, no theoretical insight is provided to elucidate why these specific designs are chosen. [1] Stop regressing: Training value functions via classification for scalable deep rl, 2024. 2. Limited Scope. The entire paper focuses on a very specific domain (autonomous device control). The scope of the proposed method might be too narrow to be of general interest to the ML/RL community. Minor Points: - In section 4.2, how to properly balance the two estimators, one with higher variance and one with higher bias to achieve the optimal result? What's the hyperparameter profile of the combined estimator? Have you tried any alternative designs and can you give theoretical insight to justify this specific design choice? - Regarding the offline and off-to-on RL setting: to my knowledge, the main point of offline RL is to leverage a large body of stale data to safely and efficiently pretrain a RL agent. Therefore, for off-to-on RL, where online RL operates as the fine-tuning stage, one should use data far less than the offline pretrained dataset to ensure the setting is meaningful. The fact that authors intentionally use the same amount of data for both offline and online stage, which assumes access to a large amount of online data might make the offline pretraining unecessary. To see this, I recommend the authors to directly perform online RL on the combined dataset and it's highly possible that such "purely online" agent outperforms its off-to-on counterpart. - The authors spend quite a few words discussing the challenges of stochasticity and device control as a POMDP. However, I do not see any specific design or techincal contribution targeting such problems.

Questions

1. In line 250-251, the CE loss pairs r with log V and (1-r) with log (1-V). Intuitively, this means one would like to make the distribution of r and V as close as possible (when r-> 1, V -> 1 and vice versa). This seems to contradict the claim in line 240-241 that "Intuitively, if a rollout attains a high value of A(sh, ah, c), it means the value function V is small". 2. How do you perform the train/test task split, if not random split? It is odd to see in Table 1 that almost all testing performance clearly surpass the training performance (normally should be the opposite), which suggest that the testing tasks are in general easier than the training tasks and not i.i.d?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

N/A

Summary

This paper proposes an autonomous RL approach, RL for digital agent (DigiRL), to finetune a pretrained VLM as an in-the-wild device control agent through GUI. The authors build a parallelizable Android learning environment with VLM-based evaluator to identify the key design choices for RL. The training include two stages an offline RL phase on existing data, then followed by an online RL phase by interacting with real-world graphical user interfaces using the Android learning environment. The proposed method with only 1.5B model size outperforms other state-of-the-art models such as GPT4-V or 17B CogAgent in the Android-in-the-Wild (AitW) dataset.

Strengths

- The paper is well-structured and easy to follow. - Many design choices are well motivated. - The experiments are nice and well support the claims.

Weaknesses

- Overall there is no major weakness. There are only several questions and potentially interesting empirical studies to look at. Check more details in the question sections.

Questions

- Could the authors compare more advanced LLM reasoning or planning algorithms like Chain of Thoughts (CoT), Tree of Thoughts (ToT), Reasoning as Planning (RaP), etc.? - Following the previous question, is it possible to compare with those planning/search-based methods with the autonomous evaluator or the trained value model of DigiRL as value functions? - What is the training time and compute requirement for online training? - In Figure 7, what does the AWR reweighting mean? Is it simply AWR? - With the auto-curriculum setup, it may be interesting to look at what types of data/replay are critical throughout the online learning process; with simple categorization like failure mode in Figure 5 or whatever characterization can be interesting. As detailed in the "Challenges of stochasticity" paragraph in section 3, could the authors provide some studies on unpredictable distractor and technical glitches?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed in the paper.

Summary

This paper tackles AI agent training for controlling digital devices (e.g., web navigation). The proposed framework, named DigiRL, is a 3-stage training process consisting of model pre-training, offline fine-tuning (offline RL), and online fine-tuning (online RL). To achieve this goal, the authors first build a parallelizable Android learning environment that enables fast online interactions for policy learning; they then adopt a VLM-based evaluator to provide reward signals for the agents; finally, they perform ablation studies to examine several key design choices in typical policy-based RL methods for the third stage. Compared to larger models trained without this stage, the proposed approach enjoys significant performance enhancement due to the online fine-tuning stage.

Strengths

+ The authors did a good job introducing the background, the problem setup, the baselines, and the details of their proposed method. + Fine-tuning large VLMs in an online fashion can be challenging; the performance improvement obtained by the proposed method, which is relatively simple, is substantial and the overall approach looks promising.

Weaknesses

The main issue is the limited comparison with online RL methods for fine-tuning LLM-based agents. The only RL method compared in the experiments is Filtered BC (besides vanilla AWR, which the proposed method is based on). Filtered BC is strictly speaking not an online RL method. Admittedly, AI agent training for device control is a relatively under-explored new area, and the authors claim that theirs is the first successful offline-to-online RL approach for device control AI agents, I believe more experiments with other online RL baselines not originally designed for device control is required to justify DigiRL's advantages. For example, the classic on-policy methods such as REINFORCE and PPO, or the more recent ones that are more sample-efficient, such as [1, 2] (which might be considered more-or-less recurrent work, though). Further comparisons also help to provide more insight into the unique challenges of the device control problem for digital agents. [1] Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents [2] REBEL: Reinforcement Learning via Regressing Relative Rewards

Questions

- Will the proposed method further scale well with more online interactions? - While multi-turn interactions are a challenge of the device control problem, is there any component in the proposed framework that specifically helps to tackle it? I am willing to adjust my ratings after seeing the authors' responses.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Thanks for the rebuttal

The rebuttal addressed most of my previous concern. Thanks!