Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale

Summary

The paper proposes a method for synthetically creating a dataset of agent trajectories for navigating websites. They propose using annotated programs, specifically python code interleaved with comments, as a representation of task and motion planning trajectories, where tasks are described in natural language in the comments, and actions are python API calls to the environment. A dataset of trajectories is created by applying GPT-4 to two sources of data. The first source is wikiHow, which contains ungrounded high-level tutorials. GPT-4 is used to generate plausible trajectories consistent with the tutorial. The second source is ClueWeb, which contains HTML snapshots of real websites. GPT-4 is again used to generate plausible trajectories consistent with the website. The paper finds that the resultant dataset of trajectories, when used to finetune CodeLlama-instruct-7b, outperforms open-weight models of similar sizes on Mind2Web, MiniWoB++, and WebArena. This result demonstrates that finetuning on the dataset transfers well to tasks.

Strengths

- The paper is well-motivated, and for some sections clearly written - Creative usage of two different data sources, tutorials and snapshots, to complement each other - Makes progress towards building competent LLM-agents - Demonstrates that synthetic trajectories improve benchmark performances.

Weaknesses

- Requires the use of GPT-4 to generate the synthetic data. While the paper demonstrates that using GPT-4 to generate training data can improve smaller open weight models, it's not clear if this can help push the SOTA. - Unequal comparison with human annotation: Sec. 6.2 compares with the human annotated trajectories in Mind2Web. As acknowledged by the authors, Mind2Web tasks are very restricted and simple compared with the tasks in synatra. A fairer comparison would be to collect a smaller set of human annotations on roughly the same task distribution as synatra. - 6.3 rather ad hoc

Questions

- open-source vs open-weight. Be careful with the distinction! (e.g. L54) - How would GPT-4 perform if trained on synatra? Is synatra limited to distilling agent-capability from stronger models to weaker models? What stopped you from finetuning GPT-4 or GPT-3.5 on synatra? - In 6.3 you write that you "conducted a detailed examination" comparing synatra with GPT-4, but I can't seem to find the analysis. - Given that the comparison with human annotation in 6.2 doesn't seem very fair, I'm not sure if you can claim in the abstract that "we show that the synthetic demonstrations can be more effective than a similar number of human demonstrations." - To help answer the quality/quantity trade-off, one might look at how finetuning codellama does when varying the size of the dataset. As prior work [0] suggested, maybe only a small number of examples is needed, which pushes the trade-off towards obtaining smaller quantities of high-quality human-annotated data instead of large quantities of synthetic data. [0] Zhou, C., et al. "LIMA: Less Is More for Alignment. arXiv, Article." arXiv preprint arXiv:2305.11206 (2023).

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- Broader impacts adequately discusses potential unethical usage of LLM agents enabled by this work. - Needs discussion of robustness of LLM agents and potential security risks posed by deploying them. - Limitations of using synthetic data adequately discussed.

Summary

This paper proposes a demonstration generation approach for learning UI control by mixing two different sources. The first one uses GPT4 to generate scenarios, grounded actions, and observations using WikiHow plans which is initially filtered by GPT-3.5. Another data source is ClueWeb, form which the authors extract web pages, sample a segment for any page, and ask GPT4 to come up with scenarios and next actions. Both data sources are combined into a single dataset of 50K traces where a program format is used. A CodeLlama model is fine-tuned with the resulting dataset. Results show that it achieves comparably when compared to other baselines, including a basic GPT-based prompting method. Ablation analysis shows that program format is important, both data mixtures add value, and proposed dataset transfer to other benchmarks better than Mind2Web traces.

Strengths

The paper proposes a data synthesis approach for training UI agents. It combines WikiHow plans and actual observations with the strength of GPT-based models to fill in the gaps. the approach is agnostic to any domain and can serve as a good initialization.

Weaknesses

My main concerns are weak baselines and lack of some others. 1. There are many baselines missing from the comparison and available baselines are weak. For example, HTML-T5 [11] achieves 49.7 and MindAct [6] achieves 30.9 on Mind2Web dataset. Performance on Miniwob++ reaches up to 99.2% [A]. There are many other baselines including prompting and fine-tuning based ones with different pre-training that should be included. 2. Can you further fine-tune on demonstrations from available benchmarks? While a semi-supervised approach is helpful, how much does it improve when human demonstrations added? 3. You also mentioned that a unique "id" is assigned to target elements in your training demonstrations. Given that the actions condition on the current state, what prevents the model from overfitting to this unique id when detecting salient elements? 4. You mention that you use viewports in Mind2Web benchmark to fit into the context window. But to extract viewports, you need to know where the ground truth element resides in the DOM (or accessibility tree) or you assume you have access to an oracle scrolling agent. I think both cases are different from your baselines and suggests leakage of test data. 5. When you use NL format rather than programs, how do you ground generated actions to executable environment actions? 6. What is the performance of the model trained with Mind2Web demonstrations on Mind2Web test sets? When tested on other benchmarks, how do you ground the action on the particular benchmark? 7. It is not clear if the claim that collected demonstrations have a similar distribution as Mind2Web as they have fatter tails. 8. Text and legend in Figure-2 are not readable. A. Longtao Zheng, Rundong Wang, and Bo An. Synapse: Leveraging few-shot exemplars for human level computer control. arXiv preprint arXiv:2306.07863, 2023.

Questions

Please see my above concerns for particular questions.

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations of the work are addressed.

Summary

This paper investigates turning indirect knowledge from the internet (such tutorial) to the direct knowledge (in the form of textual demonstrations) for LLMs. The algorithm design and the experiments are based on the web browser domain, which is a general interface to various web applications. The proposed method, Synatra, handles two types of knowledge source: tutorial articles and HTML snapshots, and presents corresponding demonstration synthesis approach. Such knowledge is then transferred to the LLMs by fine-tuning the LLMs on the synthesis data. Experiments show that Synatra can improve performance of CodeLlama-instruct-7b, outperforming open-sourced/fine-tuned models.

Strengths

Overall, I think the paper is well-motivated. There are many tutorials available on the internet. It is interesting to explore how to effectively utilizing such knowledge. The method is simple and effective, while the writing is easy to follow.

Weaknesses

I have some concerns about the experiments. 1. The comparison experiment is a bit unfair to me, for several reasons: - There are some RAG based methods to be considered, as this can be regarded as a knowledge retrieval problem. - Much of the baselines are not fine-tuned. Although fine-tuned with interactive data, the data in AgentLM, CodeActAgent is not out-of-the-box like Synatra. Maybe the authors can consider baselines that fine-tune LLMs with more direct demonstrations. - In Synatra, the demonstration is processed and generated by GPT-4. These high quality data further improves the performance of Synatra. 2. There lacks some important details about the method, e.g., how is the LLM fine-tuned and the training hyper-parameters. Besides, the configurations about baselines and evaluation dataset should be detailed.

Questions

1. How is success rate calculated in these web tasks? 2. In Synatra, how is the LLM fine-tuned on the synthesis data? 3. What is the source of performance improvement? Based on the results in Figure 4 (right), Synatra clearly outperforms human annotation method. I do not think this is simply attributed to the coverage of the annotation data. Maybe it is because GPT-4 has been trained on these testing tasks?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors have discussed the limitations in the conclusion section.

Summary

This paper studies the problem of dataset generation for the digital task in the context of LLM. The motivation is that the current data collection process often requires human annotation, which is very costly and not scalable. As a result, the paper proposes using large language models (LLMs) to generate a synthetic dataset based on indirect knowledge. The indirect knowledge often refers to tutorials designed for human consumption and observation trajectories that lack associated tasks and actions. In contrast, "direction" knowledge consists of complete states and actions. To generate synthetic data based on indirect knowledge, the paper proposes several methods: (1) writing a prompt to let LLMs imagine their missing actions or states given partial trajectories, and (2) randomly sampling the context from the external large-scale dataset such as webpages. Table 1 shows the results that the proposed approach achieves the best performance among the baselines.

Strengths

1. The paper is easy to follow and read. 2. The method seems to solve the common lack of demonstration data issue in the current LLM fine-tuned paradigm. 3. The method seems to be novel.

Weaknesses

1. The method falls into the category of the perpetual data machine, in which one uses LLMs to generate its own training dataset. In addition, the proposed method is similar to self-refine paper (https://arxiv.org/pdf/2303.17651), in which LLMs iteratively fine-tune itself based on the generated dataset. 2. Table 1 does not compare the proposed method to the existing methods that use their own dataset to fine-tune such as self-refine paper. For instance, the paper could add a baseline in which the model is fine-tuned based on the other iterative fine-tuned procedure on the same dataset used in the paper. This will further verify the proposed method

Questions

1. I am curious to learn more about the author's thoughts on perpetual data machines, and the comparison to self-refine papers. 2. Please answer the weakness section

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

See the weakness section.

Response to author response

Thanks for your detailed response. It seems that other reviewers also concern about the comparison and the reliance on GPT-4. I appreciate authors' additional results, addressing some of concerns regarding the comparison. In addition, I still think the source of performance improvement over human demonstration is unclear. Specifically, I am not convinced that demonstrations generated by GPT-4 are more effective than that of human. You mentioned 'basic task formats'. However, human can also provide demonstrations that conform to the task format. The advantage of proposed method is that it does not require human cost, but requiring a high-quality LLM for annotation.