Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy

Summary

The paper introduces an LLM agent that can solve the game of Press Diplomacy only by self-play, without fine-tuning or regularizing on human data. The method simplifies the previous architecture from the PIKL/CICERO papers by FAIR (https://openreview.net/forum?id=F61FwJTZhb) by avoiding using reinforcement learning and human demonstrators. Given that the authors use commercially available LLMs (GPT4 and LLAMA) I contribute the success of the method to the alignment of the LLMs to human preferences and communication style. Additionally, they exploit the planning capabilities of current LLMs to bypass the need for RL.

Strengths

1. The introduction of sub-goals is an interesting feature. It acts as in a chain-of-thought mechanism, which is proven to improve the LLM output and avoid hallucinations. Smaller reasoning steps are more robust than long-horizon planning. 2. The paper proposes a novel architecture with a memory buffer that acts like a pseudo-RAG to improve the context of the LLMs information. 3. The authors provide a very complete repo to reproduce the experiments.

Weaknesses

1. Well, the paper benefits from the improvement of LLMs over time, as compared with GPT2 (used by CICERO). The alignment to human data is not needed as now the LLMs have the alignment incorporated by the vendor. Same for the planning step, the LLMs intrinsically got better at planning so now we can demote RL for free. 2. When evaluating against CICERO, the win rate over CICERO is not that high with your model winning the game only (<1%) of the time. I believe the win rates are not impressive, but the fact that the victories are achieved by a more lightweight model that is cheaper than training CICERO. As it doesn't require human data. 3. The alignment with humans playing and conversational style is hard to assess as it will require having your model to play against humans in an online platform and assessing whether players realize they were playing against an AI player.

Questions

1. See weakness above.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have not provide any analysis on the model limitations.

Summary

This paper focuses on diplomatic activities using LLM-based societal agents without relying on human data. It introduces a new paradigm for AI diplomacy agents that can improve through self-play and experience collection. The new agent, Richelieu, achieves state-of-the-art performance compared to current methods and demonstrates generalizability across different LLMs.

Strengths

This paper presents a new AI agent for diplomatic scenarios that surpasses all previous methods. This agent can improve autonomously without relying on human data. And the accompanying figures effectively illustrate the concept.

Weaknesses

- There are some typos (line 61, line 69), and certain citations (line 80, line 237, line 276) are not properly functioning. - Some experimental details are still unclear. - Which base model is utilized in the experiment for Figure 4 and Table 1? Is the improvement over all other baselines attributed to the agent paradigm or to the strong capabilities of GPT-4 or LLaMA3 compared to previous base models like Cicero, which only used a 2.7B model? - In the main experiment for Table 1, when Richelieu plays as a randomly selected country, what model or agent represents the other countries? - In the ablation study, what are the effects of blocking each module, such as only blocking the sub-goals module? - Techniques such as memory and planning in building agents have been proposed previously, which reduces the novelty of the method.

Questions

- Is the social reasoning flow the same or different for the planner and negotiator parts? - Can this paradigm generalize to other fields in social simulation?

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

None

Summary

The paper presents “Richelieu,” a self-evolving large language model (LLM)-based agent designed for the game of Diplomacy. Richelieu integrates strategic planning, social reasoning, and memory reflection to handle complex multi-agent environments without relying on domain-specific human data. The model self-evolves through self-play games, demonstrating its effectiveness.

Strengths

1. This paper studies an interesting social science problem and introduces an LLM-based paradigm to build an AI Diplomacy agent. 2. This paper proposes a self-evolve strategy through self-play without human data. 3. The presentation of this paper is good, clearly describing the core idea of the paper.

Weaknesses

1. Lack of in-depth analysis of the middle process. In the proposed framework, the Social Reasoning and Planner with Reflection play an important role. For example, I wonder if the LLM can accurately model relationship and inferring intention. The author could provide more analysis or examples of these modules to prove the effectiveness of LLM on this task. 2. Some technical details are not clear. For example, in the process of experience retrieval, both object similarity and state similarity are considered, but no descriptions of their implementations. 3. While the paper demonstrates the effectiveness of Richelieu, scalability to larger and more diverse environments remains to be fully explored.

Questions

1. The generation process of LLM is uncertain, which may lead to unstable reasoning and reflection, and finally inconsistent results. Does this paper consider this problem and how to deal with it? 2. Could you further elaborate on the significance of self-play games in the self-evolution process? 3. How to consider both object similarity and state similarity in the process of experience retrieval? 4. Could you provide examples of how Richelieu adapts its social reasoning and negotiation tactics based on past interactions?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See Weaknesses and Questions

Summary

This paper propose a new framework for LLM-based agents to play diplomacy games and improve themselves. The proposed framework, Richelieu, have several components and many abilities. The authors perform good experiments and ablation study to show how the framework.

Strengths

- The framework have several core components, and I believe the authors build a good agent that can handle the game. - The game or the setting is good for evaluating the comprehensive ability of LLM-based agents.

Weaknesses

- What does "evolve" mean in the article? Does it only refer to the storage of memory modules? If so, it seems that the model itself has not been updated. - The baseline in the article is not a LLM-based agent, can a stronger baseline be provided? For example, ReAct, PlanAct, AutoGPT. - The article lacks some references to work in the field and discussion of related work, e.g., [1][2][3]. [1] The Rise and Potential of Large Language Model Based Agents: A Survey [2] ReAct: Synergizing Reasoning and Acting in Language Models [3] ProAgent: Building Proactive Cooperative AI with Large Language Models

Questions

See Weaknesses

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

- The self-evolve in the paper is not clear. - For the memory module and self-play module, a more detailed analysis should be included.

Improved rating:7

I would like to thank the authors for clarifying my questions and engaging with interest. Since my concerns were clarified I have upgraded my rating to 7. This is a good paper that merits an accept due to its contributions over a widely researched topic such as Diplomacy and negotiation.

Reply

Thank the authors for your clarifications. I will increase my score from 5 to 6.

Reviewers EuFx and raEc

The authors have responded to your reviews. Have they sufficiently answered your questions, and if not, do you have any further clarifying questions you would like to ask? The author discussion period ends tomorrow (13th).