Summary
This paper empirically studies model's automatic evaluation of trajectories on digital agents, such as agents operating on the web and Android. The authors focus on two methods to construct the evaluator, one is an end-to-end approach that directly uses a pretrained VLM, another is a modular approach that first transcribes the screenshot to captions with a VLM, and then feeds all text to a LM. Both methods demonstrate high accuracies against oracle evaluator and human judge. The authors then utilize such automatic evaluators to improve the policy agent, through reflexion and filtered behavioral cloning respectively. In both settings, the evaluation is able to improve the tasks.
Reasons to accept
1. Empirically studying whether current models are able to evaluate digital agents’ trajectory success answers a previously unknown question and is useful practically as well.
2. The authors successfully utilize the model evaluators to improve the performance on two difficult tasks, with different approaches.
3. This paper studies both end-to-end and modular approaches, across both proprietary and open-source models, which can provide practical guidance for different readers.
Reasons to reject
As an empirical paper, I expect the experiments to be conducted in a more comprehensive manner with more details, and in general the empirical results are relatively sparse. Some examples:
(1) while the authors talk about both trajectory-level and step-level evaluators, step-level evaluation is only applied for the iOS agent refinement, and there is no study how accurate such step-level evaluation is. From Table 2, I guess step-level evaluation may not be very accurate given small improvement over self-training;
(2) end-to-end methods only evaluate GPT-4V, while I would like to see the performance of more open VLMs;
(3) In Table 2, filtered BC only finishes 3 more tasks than self-training in a test of 52 tasks. I am not sure how significant this result is.
Questions to authors
1. The high accuracy in Table 1 is nice, but I am feeling this is because the special properties of the two environments – for example, the final screenshot is typically enough to classify whether it satisfies the goal. I wonder whether the authors think such high accuracies are general, and whether there are environments where the final state is difficult to classify, I would like to see more discussions on this aspect as potential limitations of the paper.
2. Webarena by design is a text-only task originally, I see the authors are using GPT4-V on that, are the authors experimenting with VisualWebArena or am i missing sth?
```
After rebuttal, I bump up my score to 6 given the clarifications and added results to increase the significance of the previous results.
```