As AI systems pervade human life, ensuring that large language models (LLMs) make safe decisions remains a significant challenge. We introduce the Governance of the Commons Simulation (GovSim), a generative simulation platform designed to study strategic interactions and cooperative decision-making in LLMs. In GovSim, a society of AI agents must collectively balance exploiting a common resource with sustaining it for future use. This environment enables the study of how ethical considerations, strategic planning, and negotiation skills impact cooperative outcomes. We develop an LLM-based agent architecture and test it with the leading open and closed LLMs. We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%. Ablations reveal that successful multi-agent communication between agents is critical for achieving cooperation in these cases. Furthermore, our analyses show that the failure to achieve sustainable cooperation in most LLMs stems from their inability to formulate and analyze hypotheses about the long-term effects of their actions on the equilibrium of the group. Finally, we show that agents that leverage"Universalization"-based reasoning, a theory of moral thinking, are able to achieve significantly better sustainability. Taken together, GovSim enables us to study the mechanisms that underlie sustainable self-government with specificity and scale. We open source the full suite of our research results, including the simulation environment, agent prompts, and a comprehensive web interface.
Paper
References (78)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The paper design a generative simulation platform for strategic interactions and cooperative decision-making in LLM agents. Three similar economics scenarios where all agents exploit a common pool resource with sustaining it for the future are tested. They design a LLM-based agent architecture and test them on open and clossed LLMs. Extensive evaluations includes the benchmarking, norm rebustness given a new greedy newcomer, the sustainability improvement from universalization reasoning, ablation of communication and analysis of agent dialogues.
Strengths
1. Well-written, easy to understand by a reader out of this field. 2. propose a new multi-agent platform to test LLM agents on sustainable behavior in three simplified economic scenarios. 3. Evaluations are comprehensive and insightful.
Weaknesses
Are these three scenarios too similar? they seems to be one task but with different name. Can you test on more settings with a shared common resources?
Questions
None
Rating
6
Confidence
1
Soundness
3
Presentation
3
Contribution
2
Limitations
Yes
Summary
The paper introduces a generative simulation platform to investigate the dynamics of resource sharing among multiple large language model (LLM) agents. Specifically the authors construct a common pool resource problem where the classic social science problem, tragedy of the commons, can be demonstrated. Authors show that most LLM agents fail to achieve a sustainable equilibrium. The paper also highlights that "Universalization"-based reasoning significantly enhances sustainability.
Strengths
* The introduction of GovSim provides a novel platform to study the cooperative behavior of LLM agents in resource-sharing scenarios. The design bridges the development of LLM-agents with classic social science theories, offering a unique perspective into developing rational LLMs. * The paper offers thorough analyses and a variety of experiments. Experiments are conducted over most state-of-the-art LLMs. I particularly like the introduction of "newcomers" which consolidates the experiments. * The evaluation metrics proposed are well-motivated and theoretically grounded.
Weaknesses
* While the GovSim environment is novel in its exploration of social science concepts, it is somewhat limited in its ability to accurately represent and visualize the complexity of actual agents' behavior. A platform where agents' actions can be grounded would further enhance the argument * The paper would be strengthened if the authors incorporated human participants in the study, allowing LLMs to coordinate with humans in real-time. This would provide valuable insights into how AI agents interact and cooperate with human decision-makers.
Questions
* Could the authors provide insights into how the platform could be further scaled up to handle more complex scenarios and larger agent populations?
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
N/A, see weaknesses
Summary
This paper proposes GOVSIM, a simulation platform for studying cooperative decision-making in Large Language Model (LLM) agents. The authors test various LLMs in three resource-sharing scenarios, finding that only a few instances (2 out of 45) achieve sustainable outcomes. They demonstrate that communication between agents is crucial for cooperation, with negotiation being the primary form of interaction. The study also conducts a robustness test of the effect of introducing a greedy newcomer into an established norm. The authors provide insights into improving agent performance through universalization reasoning and offer a comprehensive analysis of the factors contributing to sustainable cooperation in AI systems.
Strengths
- This paper is clearly written and easy to follow. - Discussions about the cooperative abilities of agents and potential safety issues are both interesting and important. Additionally, the scenarios used in this article to discuss these issues are very engaging. - Experiments on different models are adequate.
Weaknesses
1. I regard the primary flaw of this article as it does not test the performance of the GPT-4 model. If my understanding is correct, Table 3 in Appendix D indicates that the paper uses GPT-4 Turbo (as the GPT-4 in the paper). GPT-4-turbo and GPT-4 are entirely different models, and many existing studies suggest that GPT-4 performs better on certain tasks. Additional testing using the GPT-4 model could confirm whether LLMs have the ability to cooperate. Note that recent work has not only found that some LLMs possess cooperative abilities but also exhibit spontaneous cooperation. 2. The discussion on robustness testing is inadequate, failing to explain why the system can be stable in certain circumstances. In the experiment, instead of introducing a large disturbance by the "LLM newcomer" in just one round, it would be more effective to introduce disturbances in different rounds to see if other LLM agents can adapt to these disturbances and maintain environmental sustainability by reducing their share in certain rounds where disturbances take place. 3. The paper claims that the three proposed scenarios have similar mathematical structures, but the performance of the third scenario (River Pollution) is significantly worse than the other two. Although the article attributes this to the need to consider more factors, there is no data analysis or log analysis to support this claim. This weakens the generality of the research. 4. The paper does not demonstrate the generalizability of its conclusions through prompt sensitivity analysis (e.g., paraphrasing). We cannot be sure whether specific prompts influenced the simulation results or if there is any data leakage involved. 5. The research lacks clear contributions. While I believe that discussing the cooperative modes of agents is very meaningful, the article does not profoundly demonstrate how to enhance cooperation or explain why cooperation cannot be achieved. Many existing studies have already discussed how agents' communication abilities can enhance cooperation, which is not a distinct contribution point.
Questions
In addition to the issues raised above, the author may further explore the following questions: 1. Lack of explanation and discussion on the performance differences between models. Although many experiments demonstrate the differences between models, there are no tests attempting to explain how they differ. The author could address this issue by showing the differences in dialogue and actual behavior between models, even if only qualitatively. 2. Similar to question 1, the interpretation of experimental results could be richer. For example, the author's interpretation of Figures 5c and 5d only explains why some models that can understand the beliefs and numerical implications of other agents perform well, but does not explain why llama3-70b performs poorly, even though it has relatively high accuracy in Figure 5c. A more profound explanation could help us understand the sources of performance differences between models and the underlying reasons for agents to cooperate. 3. Ablation experiments could be added. For instance, instead of having the agent remember only the key information for planning, providing all information for the round could be tested to see if the agent's performance improves. There are many summarizations and simplifications in the study, which might potentially impact the decision-making performance of the models. Minor issues: - P.2 Line 72: pertubation -> perturbation? - P.4 Line 138: The sudden appearance of the word "policy" is very abrupt in reading. - P.34 There appears to be a duplication of GPT-4 in Table 20.
Rating
4
Confidence
5
Soundness
3
Presentation
4
Contribution
2
Limitations
The authors have adequately addressed the limitations.
Summary
This paper presents GOVernance of the Commons SIMulation (GOVSIM), a generative simulation platform to study strategic interactions and cooperative decision-making among large language model (LLM) agents. The authors investigate sustainable resource sharing in a society of AI agents using different LLMs to determine their ability to achieve cooperative outcomes. The study finds that most LLMs fail to maintain sustainable cooperation, largely due to communication deficits and the inability to consider long-term consequences. The paper also introduces "Universalization"-based reasoning, which significantly improves the sustainability of the agents' actions.
Strengths
**Novelty:** The introduction of GOVSIM as a platform for studying cooperative behavior in LLMs is innovative and addresses a critical gap in the literature on AI safety and multi-agent interactions. **Comprehensive Analysis:** The paper thoroughly evaluates different LLMs across multiple resource-sharing scenarios, providing a broad view of their cooperative capabilities. **Open Source Contribution:** The authors promise to provide the full suite of their research results, including the simulation environment, agent prompts, and a web interface, which can foster further research and development in this field. **Ethical Considerations:** The study integrates ethical reasoning (Universalization) into the agents' decision-making process, demonstrating a forward-thinking approach to embedding moral principles in AI behavior.
Weaknesses
**Limited Scenario Complexity:** The resource-sharing scenarios in GOVSIM are relatively simplistic and may not capture the full range of complexities found in real-world resource management. **Generalizability:** The findings might not generalize well to more complex or heterogeneous environments, especially those involving mixed human-AI interactions. **Communication Limitations:** The study highlights the importance of communication but does not provide a detailed analysis of how different communication strategies or protocols might improve cooperative outcomes. **Over-reliance on LLMs:** The study assumes that current LLMs can approximate human-like strategic reasoning and negotiation skills, which might be an overestimation of their current capabilities.
Questions
- How does the performance of LLM agents in GOVSIM compare to human performance in similar resource-sharing scenarios? - What specific communication strategies or enhancements could be implemented to improve cooperative outcomes among LLM agents? - How would the introduction of more complex, real-world variables (e.g., variable resource regeneration rates, multiple resource types) impact the agents' ability to cooperate sustainably? - What measures can be taken to improve the generalizability of the findings to more diverse and heterogeneous environments? - How do different LLMs handle the introduction of multiple adversarial agents or more sophisticated strategic manipulations?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The paper introduces a novel and valuable simulation platform for studying cooperative behavior among LLMs, contributing to the field of AI safety and multi-agent systems. However, the scenarios presented are a bit simplistic to provide meaningful insights into real-world applications.
Reviewer - Author Discussion
Thanks everyone for their hard work on the papers, reviews, and rebuttals. We now have a comprehensive rebuttal from the authors which responds both overall and to each review. I'd please ask the reviewers to please post a comment acknowledging that they have read the response and ask any followup questions (if any). This period is to be a discussion between authors and reviewers (Aug 7 - Aug 13) so please do engage now, early in the window, so there is time for a back and forth. Thanks!
Thank you for the detailed response. I will keep my current score.
> prompt sensitivity analysis To address this worry, we developed two paraphrased versions of the fishing scenario prompt, which were 61% and 81% different from the original, respectively. We tested these paraphrased prompts using GPT-4-Turbo on 5 seeds, achieving average survival times of 9 months and 12 months. Aggregating these results with our original setup yields an average survival time of 11 months across the variants. This analysis demonstrates that while there is some heterogeneity in performance across prompt variations, our core findings remain robust. Our intention in developing the three scenarios (lake, pasture, pollution) was to better characterize variability and robustness of LLM behavior. Average across 5 seeds for each run – while we observe some heterogeneity – our results are robust. > any data leakage involved. We don’t believe data leakage is possible as all scenarios are developed specifically for this project. All prompts are available in Appendix C for inspection and are not otherwise available on the internet in any form. Thank you for the insightful feedback. We hope that if we have sufficiently addressed your concerns you will consider raising your score. If any outstanding questions remain, please let us know and we will promptly respond.
Dear Reviewer, Thank you again for kindly reviewing our paper. We have read your reviews very seriously, conducted additional experiments, and clarified confusion points. Since the deadline of the discussion period is approaching in 2 days, could you let us know if our answers and additional experiments help address your original concerns? We really cherish your feedback, and would be more than happy to provide additional details per your request!
Official Comment by Reviewer KPE1 (1/2)
> On the Impact of Different Models on Results While I appreciate the additional tests conducted using the original GPT-4 model, I must counter the results provided in Table 1 of the PDF document and those presented in the appendix. The results indicate significant differences between various approaches and substantial performance disparities across different models. These variations raise concerns about the consistency of model performance and the generalizability of research findings. For instance, in the example of sheep grazing pasture, Claude-3 and GPT-4 exhibit significant differences. Additionally, within the Fishing Pond scenario, the performance of the Claude-3 Sonnet, which generally performs better, falls short compared to Claude-3 Haiku. However, the authors did not provide any explanation for these notable differences between models. Merely presenting numerical results is insufficient, as potential cherry-picking in prompt settings could raise concerns among readers. This issue aligns with the fifth point I raised under weaknesses, where I highlighted the lack of explanation regarding why cooperation occurs or fails to occur. I believe this analysis is crucial. > Comparison with Existing Work I agree with the authors that maintaining shared resources in the future is both a challenge and a critical issue. However, as previously noted (and also mentioned by Reviewer 9ZQN), the complexity of the scenarios in this study is limited, which diminishes the contribution of the work. Furthermore, while the authors argue that most existing studies are confined to simple social dilemmas involving only two agents, I am willing to provide a series of existing game-theory-based studies involving communication, as well as work that goes beyond classical game theory. I do not see the novelty in this work regarding (1) multi-agent communication and (2) complex social situations. Additionally, the definition of "prompting LLMs to generate agents" is confusing to me—what distinguishes this from other approaches? I suggest that the authors conduct a more thorough investigation of existing work to better understand the true contribution of this study. This also ties into the fifth point regarding the contribution: while I agree that the public resource issue is significant, I am highly concerned about the ability of this work to generalize to more complex real-world situations. The "complex dynamics" discussed in this work may not be as robust as the authors believe. > On Robustness Testing The new entrant scenario offers some insights into how models handle disturbances, but it only addresses a single type of disturbance at a fixed point in time. As initially suggested, introducing disturbances across different rounds would provide a more comprehensive test of the agents' adaptability and the framework's robustness. In real-world scenarios, a fixed disturbance at a single time point is not representative. Given the new results, I am more curious whether cooperation occurs consistently. Can the authors report how often good cooperation was achieved across multiple runs? I understand that space constraints prevent listing all prompts, but since the code is not open-sourced, I cannot determine whether cooperation arises from specific prompts, random chance, or is genuinely a result of LLM's in-context learning. > On Qualitative Analysis I believe the discussion of the proportion analysis in Section 3.6 is sound, but it does not provide enough insight into how agents generate cooperation. Since the authors suggest that the greater the proportion of negotiation-related discussions, the more likely cooperation is to occur, what are the underlying reasons for this? Why does cooperation remain weak even when the proportion is large? I think this work still lacks in-depth analysis to make the conclusions reliable. Of course, I appreciate the authors' commitment to adding more examples, which will help improve the reliability of the work. Regarding the issue in Section 3.7, please revisit my second question on “why llama3-70b performs poorly, even though it has relatively high accuracy in Figure 5c.” I find the authors’ claim that “requires all of the underlying sub-skills” lacks evidence. The definition of aggregated sub-skills is too vague to be convincing, and more detailed explanations are needed here.
Official Comment by Reviewer KPE1 (2/2)
> Sensitivity Analysis I appreciate the additional sensitivity analysis experiments added by the authors; I believe they are reasonable. > Data Leakage My concern about data leakage is not whether these scenarios have been learned by the LLM, but rather that this alone does not entirely rule out the possibility of data leakage. While the specific prompts may be unique, the basic concepts and structures of resource management scenarios are widely discussed in academic literature and public discourse. Similar scenarios or principles might have become part of the training data for LLMs, potentially influencing their performance in subtle ways. I am not asserting that data leakage has occurred, but it should be the authors’ responsibility to prove its absence through experiments, even if just simple control experiments. Without a detailed analysis of the relationship between model outputs and their training data, it is difficult to definitively rule out data leakage or overfitting to common patterns in resource management problems. This is why I have consistently emphasized the need for more detailed analysis of experimental results rather than merely presenting surface-level data. Otherwise, many details in the paper, such as the differences in sub-skills between llama and Claude, and the discrepancies between different models on the same task, remain unexplained. Overall, I appreciate the effort the authors have put into addressing these concerns, but I believe there are still a series of fundamental disagreements on many key issues, and I view these disagreements as systemic. I welcome further rebuttals from the authors on these points.
Thank you for your points. We want to clarify a few misunderstandings, and state again our stance of the contributions of our work. ## On the Impact of Different Models on Results > original GPT-4 model tests We thank the reviewer for appreciating the introduction of the original GPT-4 (version 13 Jun 2023) in our experiments as requested. In the original review this was described as the “the primary flaw of this article” so we focused our time and financial resources towards promptly generating these new results. We have now shown that it does not impact the contributions we have claimed. > I must counter the results provided in Table 1 of the PDF document and those presented in the appendix. Our goal in introducing three different scenarios (fishing pond, sheep grazing, river pollution) in the original manuscript and our new results described in the rebuttal that include two additional prompt variants is to find systematic patterns in performance. To see this more clearly, below is a table showing the aggregate performance of each model across the key metrics (standard errors will also be included in the published version but are currently excluded for readability). We believe these results show the strong systematicity our results depend on. This table will replace Table 1 in the PDF as it better clarifies how they agents behave in this task in addition to survival rate (i.e., lower over-usage and equality). This table also includes a new metric, Survival Rate (q) which we define as the proportion of runs which achieve maximum survival time, i.e. m = 12, q = #{m=12}/ #runs. In the camera ready version, we update all results to include this metric which we believe helps clarify models performance when average survival times have nearby values. | **Model** | **Surv. Rate** | **Surv. Time** | **Gain** | **Eff.** | **Eq.** | **Over-usage** | |---------------------|----------------|----------------|----------|----------|---------|----------------| | **Llama-3-8B** | 0.0 | 1.0 | 20.0 | 16.7 | 57.3 | 20.0 | | **Llama-3-70B** | 0.0 | 1.0 | 20.0 | 16.7 | 90.7 | 38.7 | | **Mistral-7B** | 0.0 | 1.0 | 20.0 | 16.7 | 82.6 | 37.3 | | **Mixtral-8x7B** | 0.0 | 1.1 | 20.1 | 16.7 | 75.0 | 33.3 | | **Qwen-72B** | 0.0 | 1.8 | 24.0 | 20.0 | 83.9 | 32.4 | | **Qwen-110B** | 20.0 | 4.5 | 36.3 | 30.3 | 89.6 | 47.0 | | **Claude-3 Haiku** | 0.0 | 1.0 | 20.0 | 16.7 | 91.0 | 35.7 | | **Claude-3 Sonnet** | 0.0 | 1.3 | 20.5 | 17.1 | 84.4 | 32.0 | | **Claude-3 Opus** | 46.7 | 6.9 | 58.5 | 48.8 | 91.4 | 21.0 | | **GPT-3.5** | 0.0 | 1.1 | 20.3 | 16.9 | 91.2 | 35.3 | | **GPT-4** | 6.7 | 3.9 | 31.5 | 26.2 | 91.4 | 27.1 | | **GPT-4-turbo** | 40.0 | 6.6 | 62.4 | 52.0 | 93.6 | 15.7 | | **GPT-4o** | 53.3 | 9.3 | 66.0 | 55.0 | 94.4 | 10.8 | >For instance, in the example of sheep grazing pasture, Claude-3 and GPT-4 exhibit significant differences. Our goal was not to highlight one particular model as the “best” or “winner”, but rather to upperbound performance in our novel benchmark GovSim, study general methods that in principle could apply to all LLM-based agents to improve performance (e.g., Universalization), and try and understand how LLM performance in GovSim can be explained by underlying reasoning abilities and ability to communicate. LLM agent evaluation is a moving target as new models and capabilities are announced every year and thus we designed our platform to have greater longevity than any one particular model.
>Additionally, within the Fishing Pond scenario, the performance of the Claude-3 Sonnet, which generally performs better, falls short compared to Claude-3 Haiku. However, the authors did not provide any explanation for these notable differences between models. In the default GovSim environment, neither model achieves sustainable cooperation (survival rate is 0 for both models across all environments). We assume the reviewer is looking at the results from our Universalization experiments. The universalization prompt improved performance across all model classes on each measured metric. Survival rates went up as a result of lower over-usage and greater equality (where possible). Claude-3 Haiku had a greater reduction in overuse than Claude-3 Sonnet, and thus had a higher survival rate when the agents were able to Universalize their actions. We prefer not to speculate on the cause of this particular difference as these closed models are relatively opaque, but instead refocus the text to make clear that our main take-away from this set of experiments is that Universalization greatly improves the sustainability of cooperation for all LLM based agents studied. We will add to a sentence in our results saying something like, “although some models benefited more greatly than others”. We will also add the aggregate table below to the appendix and have clarified the language in the paper to make these points more clear. | **Model** | **Surv. Rate** | **Surv. Time** | **Gain** | **Eff.** | **Eq.** | **Over-usage** | |--------------------|----------------|-----------|----------|----------|---------|----------------| | Llama-3-8B | 6.7 | 5.4 | 48.8 | 40.7 | 72.7 | 8.4 | | Llama-3-70B | 66.7 | 8.3 | 63.4 | 52.8 | 85.8 | 10.0 | | Mistral-7B | 0.0 | 2.1 | 27.6 | 23.0 | 84.5 | 36.8 | | Mixtral-8x7B | 33.3 | 4.9 | 38.0 | 31.7 | 74.9 | 14.6 | | Qwen-72B | 20.0 | 5.5 | 51.9 | 43.3 | 80.2 | 14.3 | | Qwen-110B | **100.0** | **12.0** | 92.5 | 77.1 | 96.6 | 0.9 | | Claude-3 Haiku | 53.3 | 7.2 | 69.0 | 57.5 | 89.0 | 6.2 | | Claude-3 Sonnet | 26.7 | 5.3 | 51.9 | 43.3 | 91.9 | 13.7 | | GPT-3.5 | 26.7 | 7.3 | 57.1 | 47.6 | 81.9 | 6.9 | | GPT-4-turbo | 80.0 | 11.5 | 101.8 | 84.9 | 98.3 | 5.4 | | GPT-4o | **100.0** | **12.0** | **112.2** | **93.5** | **99.7** | **0.4** |
>Merely presenting numerical results is insufficient, as potential cherry-picking in prompt settings could raise concerns among readers. This issue aligns with the fifth point I raised under weaknesses, where I highlighted the lack of explanation regarding why cooperation occurs or fails to occur. I believe this analysis is crucial. We agree with the reviewer that survival rates (or any single number) are insufficient to fully characterize behavior in GovSim. Instead they are a starting point. With the new aggregate tables shown above we can better highlight other metrics besides survival rate such as equality and over-use which have an impact on survival. We can commit to this change for the camera ready. We agree that explanation is crucial and we summarize the explanatory tools that are already included or have been added during the revision period: 1. Six quantitative metrics for each LLM based agent both in aggregate and broken down by scenario (Tables above and Table 1) 2. Subskill analysis for each LLM based agent (Figure 5) and broken down by scenario (Appendix Figures 11-13) 3. Time series graphs for the top six performing models in each scenario (Appendix Figures 8-10) 4. Examples dialogue included in the Appendix (Appendix G and a commitment to add more) 5. An open source web dashboard (included with the submission) that allows for in-depth analysis of dialogue for all seeds included in the manuscript. (Screenshot shown in Appendix Figure 7). 6. Prompt rephrasing that shows robustness against cherry picking (New in rebuttal) Together these 6 methods point to the following two part explanation. Agents fail to cooperate when agents over-use the resource (harvest too many resources). For many of the agents this happens on the first round and as such there is no cooperation. To avoid this outcome we first show that LLM based agents must have sufficient mathematical understanding to understand the simulation dynamics (as indicated by Figure 5a) is necessary. Smaller and weaker models such as Llama-3-8B, Mistral-7B among others, struggle to grasp the simulation dynamics. Second, lack of the ability to form beliefs about the actions taken by other agents is also strongly associated with survival (Figure 5d). The universalization experiment can be interpreted as an intervention on these ability. Universalization gives the agents both beliefs about others and some mathematical indication of what would be sustainable if all agents acted symmetrically. Consistent with our explanation, Universalization significantly improves survival especially in the weaker models (Section 3.4) – survival time increases by 4 months for models that are not already near ceiling (t-test; p < 0.001). We thank the reviewer for the opportunity to lay out these resources and will update the paper accordingly to make these explanations more clear. ## Comparison with Existing Work > the complexity of the scenarios in this study is limited, which diminishes the contribution of the work We value the reviewer’s concern. However, following the Goldilocks principle, we targeted the current complexity of GovSim to be a challenge for the most advanced models but not too complex such that all models fail. At the current level of complexity, the highest survival rate is below 54%. At this level, GovSim remains a challenging problem for the research community while still allowing us to study cooperative dynamics and emergent norms. With greater complexity, all models would fail and our ablations and interpretability studies would not yield insights. While we are excited about increased complexity, we have outlined what we believe to be the most promising directions and made our platform open source and easily extendible to allow for scenarios of higher complexity in future work. > Additionally, the definition of "prompting LLMs to generate agents" is confusing to me—what distinguishes this from other approaches? In our work we introduce the Act-Discuss-Reflect (ADR) Agent Architecture, a phase-based generative agent framework. The ADR agent architecture is a class of models development in line with recent work “Generative Agents: Interactive Simulacra of Human Behavior” (Park et al., 2023). The Generative Agents framework summarizes and then retrieves relevant information at inference time to condition the LLM’s output. The key component of this architecture is a “memory”, a vector-based database, which contains agents’ past experiences, actions and observation of the environment. Different modules (e.g., reflection) use and create new memories based on recent experiences. These agent based architectures are necessary to analyze social dynamics that include open-ended conversation as 12 rounds of dialogue fills up too much of the context window of even the most powerful LLMs. In contrast, prior work has largely focused on simple decision making or game theoretic tasks and asked an LLM to select an option.
>I am willing to provide a series of existing game-theory-based studies involving communication, as well as work that goes beyond classical game theory. We would value any additional references that you are willing to share with us and can commit to discussing them in the related work. However, just to clarify we do not claim that we are the first to study game theory in the context of communication (e.g., prior work human behavior in behavior economics & multi-agent reinforcement learning where communication is templated or done in an emergent language) nor are we the first to “go beyond classical game theory”. We believe the novelty of our work comes from leveraging LLM based agents that are capable of communicating in an open-ended fashion and our focus on common resource pool problems. ## On Robustness Testing > Introducing disturbances across different rounds would provide a more comprehensive test of the agents' adaptability and the framework's robustness. In real-world scenarios, a fixed disturbance at a single time point is not representative. Our goal was to study a minimal perturbation to the system as a study of robustness. This minimal perturbation is analogous to those studied in evolutionary game theory when empirically computing whether a particular strategy is an evolutionary stable strategy (ESS, see below for references). We will add the following references to our discussion to better motivate our use of a single disturbance. M. A. Nowak, A. Sasaki, C. Taylor, and D. Fudenberg. Emergence of cooperation and evolutionary stability in finite populations. Nature, 428(6983):646–650, 2004 D. Fudenberg and L. A. Imhof. Imitation processes with small mutations. Journal of Economic Theory, 131(1):251–262, 2006. Overall, we agree with the reviewer that multiple disturbances will reduce sustainable cooperation to a greater extent than a single disturbance. We will add a sentence to the result section with approximately the following meaning: “While the results here only show a single newcomer, we expect that larger perturbations (e.g., multiple newcomers or more adversarial newcomers) will have a greater impact on disrupting sustainable cooperation.” Below we show that the overall survival rate is 33% with a single disturbance (down from 54% in the baseline condition). If agents were already robust to a single disturbance we would have introduced larger perturbations. However, even a single disturbance remains a challenge for even the best model so we believe studying more sophisticated perturbations is best left to future work. > Can the authors report how often good cooperation was achieved across multiple runs? For the prior experiments these numbers are defined as the survival rate in the above tables. If you mean for the newcomer experiments the results for GPT4o are: With the newcomer the survival rate was (33.3% vs. 53.3% in the baseline condition; GPT4o, 3 scenarios with 5 seeds each). | **Scenario** | **Surv. Rate** | **Surv. Time** | **Gain** | **Eff.** | **Eq.** | **Over-usage** | |---------------------|----------------|----------------|----------|----------|---------|----------------| | GPT4o | **53.3** | **9.3** | **66.0** | **55.0** | **94.4** | **10.8** | | GPT4o + Newcomer | 33.3 | 6.6 | 34.8 | 31.3 | 71.7 | 15.7 | > I understand that space constraints prevent listing all prompts, but since the code is not open-sourced, I cannot determine whether cooperation arises from specific prompts, random chance, or is genuinely a result of LLM's in-context learning. To clarify, the code and dashboards from the manuscript are available to the reviewer. They can be accessed near the top of the page by downloading the supplemental information. As mentioned in footnote 1 in our submission, “Our code and data have been uploaded to the submission system and will be open-sourced upon acceptance.” By averaging across scenarios and rephrased prompts we show that cooperation does not depend on specific prompts. By showing that the results are systematic and are a function of the underlying agent subskills we show that they are not random.
## On Qualitative Analysis > Since the authors suggest that the greater the proportion of negotiation-related discussions, the more likely cooperation is to occur, what are the underlying reasons for this? Why does cooperation remain weak even when the proportion is large? I think this work still lacks in-depth analysis to make the conclusions reliable. We do not make the claim that the greater the proportion of negotiation-related discussions, the more likely cooperation is to occur. Our intent in analyzing agent dialogue is to give readers a sense of what occurs during the simulation process rather than relying on just aggregate metrics. >Of course, I appreciate the authors' commitment to adding more examples, which will help improve the reliability of the work. Thank you. All simulation runs will also be available in the open-source GitHub repo. > Regarding the issue in Section 3.7, please revisit my second question on “why llama3-70b performs poorly, even though it has relatively high accuracy in Figure 5c.” I find the authors’ claim that “requires all of the underlying sub-skills” lacks evidence. The definition of aggregated sub-skills is too vague to be convincing, and more detailed explanations are needed here. The aggregated sub-skill metric is the average score of the 4 subskill tests: (a) simulation dynamics, (b) sustainable action, (c) sustainability threshold (assumption), (d) sustainability threshold (belief). We have added this definition to the camera ready version of the paper. While llama3-70b scores highly on some of the subskills it performs poorly on others. Our empirical finding is that each subskill is highly correlated with survival time however the aggregated sub-skill is a better predictor than any individual skill. When these subskills are used as independent variables in an OLS regression analysis, we find a highly significant R2 = 0.95, p < 0.001. We hope this clarifies both how the aggregated sub-skill was calculated and gives additional statistical rigor to the subskill analyses. We have added both to the manuscript. ## Sensitivity Analysis >I appreciate the additional sensitivity analysis experiments added by the authors; I believe they are reasonable. We express our gratitude to the reviewer for recognizing the value of the additional sensitivity analysis experiments. ## Data Leakage > Without a detailed analysis of the relationship between model outputs and their training data, it is difficult to definitively rule out data leakage or overfitting to common patterns in resource management problems. While we agree that it would be interesting to analyze the relationship between LLM training and model outputs, this is far beyond the scope of our study. Firstly, all LLMs studied (including the open-weights models) are not open with respect to their training data so we’d be guessing at what these models are trained with. Second, to the extent we can assume that these models were trained on Common Crawl and other web data, data leakage cannot explain the variance we see across models. Third, understanding the link between training data and model capabilities deserves a manuscript of its own and goes beyond the scientific contributions we aim to make in this work. We will add a sentence to the limitations section to address the reviewer’s concern salient to the reader. Thank you for the engagement with our paper.
Dear Reviewer, Thank you again for kindly reviewing our paper. We have provided a detailed response to your question, and highlighted the variety in our experiments. Should our response address your concerns, would you considering raising your score, perhaps to a clear acceptance? Or if you have further questions, we would be more than happy to provide additional details and engage in the discussion! Thank you in advance for your kind consideration.
Decision
Accept (poster)