Thank you for your continuous feedback!
Here is our further clarification of your questions:
1. We sincerely apologize for any confusion and we need to sincerely say that we still have not understood why you suggested that our statement could reason that DPO should perform better on seen tasks. We need to reiterate that our method enables the model to read both correct and incorrect trajectories for the same task, using $\rho_{\rm TaskKnow}$ (detailed in Figure 11) to summarize the knowledge why the correct trajectories are better than the incorrect ones, and training the WKM to learn to generate this kind of knowledge to augment the agent model. On the other hand, DPO directly trains the agent model by minimizing the loss (increasing the distribution gap between correct and incorrect trajectories) to favor the generation of correct trajectories. Therefore, we believe there is no direct relationship or conflict between the two approaches. Our previous statement of "similar to DPO" might have caused misunderstanding; it only proves that the improvement of our agent model augmented by synthetic task knowledge is better than training the agent model with DPO.
2. **(1)** As mentioned in lines 122-123, the prompt $\rho_{\rm StateKnow}$ that we use to summarize state knowledge is detailed in Appendix I.2, which is displayed in Figure 12. In response to Q2, we have clarified to you that the key part of our prompt is the State Knowledge Example (colored in purple), which is actually the few-shot examples. Therefore, we stated that generating state knowledge directly with Equation (5) is equivalent to the experiment we supplemented in Q3, where knowledge was generated directly using few-shot examples. As shown in our previous table, while it is certainly possible to generate knowledge by few-shot examples, it is clear that its effectiveness is not as good as that of the WKM.
**(2)** Regarding the question you raised about why training Model A with data annotated by Model A itself can lead to improvements for A, **this issue has been extensively validated in the field of LLM synthetic data (also known as self-training) [1][2][3][4][5][6]**. The key to self-training lies in ensuring the quality of self-generated data. For instance, [3] achieves this by implying the correct labels of the data to the model, while [6] relies on the model's self-judgment capability to assess the quality of the data. Our approach ensures the quality of synthesized knowledge by meticulously designing few-shot examples and using entirely correct expert trajectories. The advantage of this method is that it does not depend on a large amount of manually annotated data or powerful closed-source models (e.g., GPT-4), fully leveraging the model's own potential. However, this method usually has an upper-bound because we cannot guarantee that the synthesized data is 100% correct. As the model generates more data, the diversity of the data is also likely to decrease. This is a key research direction in the field of data synthesis today, that is, how to improve the diversity and quality of synthesized data. The principle behind self-training is still under investigation; we speculate that it may be because the data annotated by the model itself is more in line with the "data distribution" understood by the model. From the model's perspective, this distribution may be smoother. Although this method may not be as effective as training on fully human-annotated data, the most important aspect is that it solves the resource consumption brought about by a large amount of manual data annotation. And if one day human-annotated data is exhausted or LLM grows into a super model that humans cannot supervise, we can only rely on the data synthesized by the model itself to improve the ability of LLM, so this is a very promising research direction in the LLM community.
[1] On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey.
[2] KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents.
[3] STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning.
[4] Best Practices and Lessons Learned on Synthetic Data for Language Models. (Google DeepMind)
[5] Self-training Language Models for Arithmetic Reasoning.
[6] Self-Rewarding Language Models.
We look forward to your further reply and would like to continue the discussion.
Thanks again!