Learning to Cooperate with Humans using Generative Agents

Summary

This work proposes a novel method to train agents to coordinate with humans in team tasks. The main contribution is that instead of training an RL cooperator policy based on a limited number of human models, the authors use a generative model to capture the latent variable representation of human policy. The learned continuous latent space is believed to cover the strategy space of a specific team task, therefore can be used to sample diverse human policies during agent training. Experimental results confirm the benefit of introducing generative models in comparison to previous methods using behavior cloning human models or population-based RL models. The authors also conduct human evaluations to pair trained agents with novel human participants via crowd-sourcing. The team performance and perceived utilities are higher when human players are paired with agents trained with the proposed method.

Strengths

1. This paper is well written and easy to follow. The introduction and related work sections motivate the work and position it among previous literature well. The problem definitions and methods section are clear. I enjoyed reading it. 2. The problem of learning to cooperate with (unseen) humans is generally underexplored. This work is relatively novel by introducing a generative model and the adaptive sampling method to capture diverse human strategies. 3. The experiments are well designed. The proposed method is evaluated among a variety of task scenarios against traditional methods. Results show that the proposed GAMMA method improves the team performance in comparison to baseline methods such as population RL or behavior cloning. The evaluation environment, i.e. Overcooked-AI, is a rather common benchmark facilitating comparison with other work in the future. 4. The authors conducted evaluations with real human participants, instead of purely relying on simulated human models. The results confirm the actual ad-hoc teamwork performance of the proposed agent with unseen human participants, yielding high ecological validity.

Weaknesses

1. This work lacks a formal analysis about the constructed latent space of human policy given the training data set. There seems to be no theoretical guarantee or quantitative analysis about the generalizability of the proposed generative model over training data, as claimed in lines 198-203. If the collected trajectories in D are biased / do not cover the entire strategy space, the performance of the proposed method might be impacted. Provide more details about Fig. 1, or quantitative analysis of the generalization / interpolation properties would help the readers to value the proposed method. 2. Details about the human data used during training are missing. Are they independently collected from human teams with optimal / consistent team strategies? The setup is very important because the policy distribution of the human data would impact the learned generative models and human proxy models used during training. Explaining the data collection setups and comparing the nature of collected trajectories from agents / humans would enhance the paper. 3. Training a Cooperator policy via RL given a partner policy conditioned on a distribution of latent variables is not trivial. There seems to be a hidden assumption about the distribution parameters to make sure partner policies do not differ too much so the RL policy can converge. The paper does not report details about this process. 4. The hidden assumption of human-adaptive sampling is that human data are closer to the target partner policy compared to synthetic data, so we want to fine-tune the generative model and Cooperator policy on it. This might not be the case if human trajectories are noisy. 5. The authors mention type-based reasoning in the related work section but do not include them in the baselines [1]. It is worth discussing how the proposed method and type-based methods perform differently, given different human policy distributions. 6. One potential future direction is to extend the framework to be online adaptive to the target partner during interaction. For example, using a similar technique to infer the human-centered latent Gaussian distribution of a specific individual human, and change the agent action policy accordingly on the fly. [1] Zhao, M., Simmons, R., & Admoni, H. (2022, October). Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 9116-9123). IEEE.

Questions

1. How are different policies sampled and visualized on the latent strategy space in Fig 1? 2. When sampling different latent variables of the partner at each episode, could PPO guarantee to learn an optimal cooperator policy? 3. Are trajectories in the human data set all from the same human-human team with a consistent strategy or a population of individual human players with different strategies randomly paired into teams? Are those human strategies optimal or not? 4. How is the held-out performed on human data and self-play agents respectively? Line 475 only mentions a 30%-70% split without mentioning if agent policy type is stratified. 5. What does this sentence mean in lines 286-288? If the held-out is properly performed on human data, e.g. removing trajectories independently collected from different human participants, PPO-BC has no chance of exploiting information about the unseen evaluation data set. This question might be related to the previous concerns about human data distribution and collection details. 6. How novel are the strategies in human evaluation compared to the original training human data? 7. How accurate is the BC model? What is the action prediction score?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

In the supplementary materials, the authors acknowledge the limitation of using limited human data which prevents the generative model from learning from scratch. I actually see this as a strength of the paper since human data is usually expensive to collect in practice. While the limited amount of human data is acceptable, the limited characteristics of human data might hurt the further application of the proposed method. As discussed earlier, if the authors could consider examining the quality and features of the used data set, it would shed light on learning to collaborate with humans in ad-hoc teamwork. There are a few inconsistent patterns within the experimental results (e.g. Figure 5 (a) and Figure 6 (b)) where the proposed method does not achieve better performance than the baselines. The authors attribute those failed cases to limitations of the training data collection process. This actually illustrates the limitations of the proposed method in solving the 2 main challenges claimed in the paper: 1) lack of real (high-quality) human data and 2) the difficulty of synthetically covering the large strategy space of human partners.

Summary

This paper proposes GAMMA, a generative model that learns a latent representing partner agents' strategy. Through interpolation, the latent can be used generate diverse partners to train ego agents for human-AI cooperation. Authors studied training the generative model with both simulated data and real-human data and conducted evaluation with both human proxies and real humans. Experiment results show GAMMA improves both objective performance and human subjective ratings.

Strengths

1. This paper is well-written and easy to comprehend. 2. The targeted issue of diversity of partner agents is important in zero-shot coordination and human-AI coordination. 3. Human study is conducted at relatively large scale to show the efficacy of GAMMA in human-AI cooperation.

Weaknesses

1. The 'state-of-the-art' baselines in this paper are outdated (especially FCP). I would be happy to see comparison against works like [1],[2],[3], which are also targeted at zero-shot coordination. [1] Mahlau, Yannik, Frederik Schubert, and Bodo Rosenhahn. "Mastering Zero-Shot Interactions in Cooperative and Competitive Simultaneous Games." arXiv preprint arXiv:2402.03136 (2024). [2] Yan, Xue, et al. "An Efficient End-to-End Training Approach for Zero-Shot Human-AI Coordination." Advances in Neural Information Processing Systems 36 (2024). [3] Zhao, Rui, et al. "Maximum entropy population-based training for zero-shot human-ai coordination." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023. 2. In Fig. 4,5, GAMMA's improvements over baselines seem marginal. 3. The interpolation visulization needs better explanation. What do the points in Figure 1 stand for? Is it policy $\pi$ or hidden vector $z$? I can't find the reletive information to comprehend this figure. 4. Adding to the last question, utilizing interpolation to enhance partner diversity is not new in zero-shot coordination field. PECAN [4] directly interpolates parnter policies through policy ensemble instead of hidden vector $z$ (see figure 4 in [4]). What is the advantage of GAMMA by interpolating at the level of hidden vectors? Does it offer better partner diversity? This should be at least discussed in the paper. [4] Lou, Xingzhou, et al. "Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination." arXiv preprint arXiv:2301.06387 (2023). 5. Typos: "data-hungry seuential decision making algorithms...", on page 1 "We propose a novel, ecnonomical way" on page 2 e.t.c.

Questions

See weaknesses. Please at least add MEP [3] to the baselines in your experiments (since it has very similar pipeline as FCP in your paper and easy to run). I would happily increase my score upon seeing such comparisons. [3] Zhao, Rui, et al. "Maximum entropy population-based training for zero-shot human-ai coordination." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.

Rating

4

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

Limiations are discussed in the appendix.

Summary

The paper addresses the challenge of training AI agents that can effectively coordinate with diverse human partners in cooperative tasks without prior interaction. This work demonstrates that using generative models to create diverse training partners can significantly improve an AI agent's ability to coordinate with novel human partners in cooperative tasks. The proposed approach offers a promising direction for developing more adaptable and effective AI assistants for human-AI collaboration scenarios.​​​​​​​​​​​​​​​​ The key contributions of the work are listed as follows: 1. The work proposes a novel approach, Generative Agent Modelling for Multi-agent Adaptation (GAMMA), that uses generative models to produce diverse partner agents for training a robust Cooperator agent. Through conducted experiments using the Overcooked environment, the authors demonstrate that GAMMA consistently outperforms baselines in zero-shot coordination with novel human partners, both in terms of task completion and subjective human ratings. 2. The authors show the flexible data integration of GAMMA, i.e. it can be trained on both simulated agent data and real human interaction data. This addresses the dual challenges of limited human data and insufficient coverage of human behaviours in synthetic data. 3. The authors also propose a method, Human-Adaptive Sampling, to efficiently incorporate a small amount of human data to bias the generative model towards producing more human-like partners during training. 4. The study brings together and directly compares two previously separate approaches to zero-shot coordination: training on simulated population data, and training on human data.

Strengths

1. *The research problem is of great significance*: This work studies zero-shot human-AI coordination, which is a key topic for applying AI in the real-world. The key bottleneck the work aims to address is the coverage of behaviours by the simulated humans obtained by behaviour cloning over datasets of human cooperation, or by using MARL. This work offers a promising direction for developing more adaptable and effective AI assistants for human-AI collaboration scenarios.​​​​​​​​​​​​​​​​ 2. *The paper is well-written and well-organised*: Overall, the paper is well-written, and very-well-organised. Despite the issues mentioned in the weaknesses section below, the motivation, the proposed method, and all experiment results are clearly delivered. Overall, I quite enjoy reading the paper. 3. *The conducted experiments are thorough and the hypotheses are clearly formulated*: It's pleasant to see that the hypotheses to verify are clearly listed at the beginning of Section 5, and the following results are also demonstrated following the same order of hypotheses. 4. *The proposed method is very intuitive*: In this context, the term "intuitive" is used to convey a positive quality. It makes a lot of sense to me that human partners can be represented by latent variables, like the user embeddings used in recommendation systems. It also makes sense that depending on the user embeddings leads to higher coverage of the behaviours of simulated human partners.

Weaknesses

1. *Results lack of statistical significance, thus persuasiveness*: Despite all the good aspects of the work, including the motivation and the intuition, my biggest concern is that the improvement of the performance doesn't look significant. First, in the Cramped Room in Figure 4, there seems to be overfitting after $5e7$ steps. In the meantime, GAMMA doesn't lead to improvement on FCP at that time step. Similarly, there seems no (statistically sifnificant) difference between FCP and CoMeDi with and without GAMMA at the $5e7$-th step in Forced Coordination. Considering that the performance of both FCP and CoMeDi decrease after $5e7$ steps in Forced Coordination as well, a more rigorous conclusion could be that GAMMA can prevent the overfitting, instead of improve the performance. Secondly, the performance shown in Figure 5 also suggests that GAMMA doesn't improve the performance, since the orange curves (PPO + BC + GAMMA) twist with the blue curves (PPO + BC). Even though the methodology is convincing, the results make me doubt whether the proposed GAMMA can indeed lead to significant improvement. 2. *Need to illustrate the necessity of sampling from $z_i$*: In Section 4.2, the authors illustrate the human behaviour representation $z_i$ is used to generate batches of MDPs. A straightforward and intuitive alternative practice is simply to make $z_i$ a part of conditionals for $\pi$. The authors need to provide either a rational in more details or more empirical evidence on the chosen implementation, i.e. using $z_i$ to generate $\\{M_{z_i} \\}_{i=1}^{N}$. 3. *Length of games during evaluation*: As an experienced player of Overcooked, the 60-second setup used in human evaluation looks too short to me. According to my own experience, for human-human cooperation, it usually takes a whole trial of the game, i.e. >120 seconds. This makes me doubt whether improvement by GAMMA is only on the adaptation efficiency. Moreover, this also makes me doubt whether the improvement observed in the evaluation is only due to some randomness. I would be keen in seeing more trials of each method with human partners in longer games.

Questions

1. Why is "training on human data is highly effective in our human evaluation" counterintuitive? The authors mention in line 95 that "contrary to past published results, training on human data is highly effective in our human evaluation". I got confused by this claim as it sounds intuitive to observe better performance by using human data in human evaluation. 2. Can increasing sampling temperature also lead to greater coverage over the strategy space? In line 200, the authors claim that "the generative model has significantly greater coverage over the strategy space". A natural alternative could be simply increasing the sampling temperature. So, I wonder if this simple trick could also boost the performance/efficiency of the same baseline methods? 3. Is H3 a hypothesis? From my perspective, the H3 stated from line 261 to line 263 is not a hypothesis. 4. Why is it useful to evaluate the method by a population of held-out self-play agents? As stated in line 289, the human evaluation is completed on "a population of held-out self-play agents". After reading the paragraph from line 282 to line 289, I'm still unclear about the rationale of doing so. More importantly, line 95 state that human data is not so effective in human evaluation. Then, why is fine-tuning on human data a good evaluation? 5. How are the number of policies (8) and the numbers of random seeds (5) decided in human evaluation?

Rating

4

Confidence

5

Soundness

2

Presentation

4

Contribution

3

Limitations

Please refer to the weaknesses section.

Summary

The authors propose a novel method for training agents that effectively cooperate with humans by leveraging generative models that can sample the space of agent behaviors

Strengths

- Well-written and easy to follow - Experimental setup is clear, well-documented, and expansive - Code and even a live demo are available (very cute demo by the way!) - Novel method in an important and growing area

Weaknesses

- Experimental results are okay, but not clear that GAMMA consistently outperforms baselines. This is true in both the simpler and more complex settings. Even in cases where GAMMA-based methods have higher performance, the confidence intervals seem to overlap making it difficult to say whether results are simply due to chance. See Questions below for a few more things I'm wondering about regarding results and significance. Happy to raise my score if authors can show that the proposed method fairly consistently outperforms baselines in a statistically significant way (either by correcting me if I'm wrong in my interpretation of current results, or by increasing sample size to decrease confidence intervals). Minor suggestions (did not affect score): - increase text size in fig 2 - report results in a table (can go in the appendix) as it is hard to tell from the graphs alone whether certain methods outperform others in statistically significant ways

Questions

- how well would this approach generalize to more realistic settings? It seems like moving to the more complex setting led to reduced gains compared to baselines. - relatedly, how complex are the strategies learned by the Cooperator? Can they deal with humans who learn, adapt, and change their own strategies? - can the Cooperator instead be meta-learned and adapt online? - not entirely clear what is meant by "Error bars [4] use the Standard Error of the Mean (SE) for statistical significance (p < 0.05)". Are you saying that error bars correspond to 95% CI? or that they correspond to standard error? - following up on that, which of the results are statistically significant? hard to tell from the figures.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- Authors note human diversity is critical, but it's somewhat unclear whether human data comes only from WEIRD populations (almost certainly yes since participants were recruited from Prolific). This is worth noting in the human eval/limitations/impact sections since results may not be representative of broader populations (especially since authors mention assistance for elderly or disabled people as potential future application). A potential social impact of this is that a) agents may best serve only the demographics they were trained on, and b) agents may lead to reduction in creativity and diversity of strategies by forcing humans to adapt to the limited range of strategies that the agents are familiar with. - Authors do mention wanting to train on larger datasets in limitations, but also worth mentioning that current experiments are in very constrained environments and only in 2-player settings. In addition to dataset size, scaling along both of these dimensions would be important future work. - If I understand correctly, the human data used for fine-tuning or for creating the generators is collected either in solo-play or interaction with a simple existing policy. It seems unlikely then, that the GAMMA-based methods could deal with humans learning, adapting, and changing their strategies. Perhaps worth mentioning this as well.

Concern about statistical significance

Copying this over from my individual response below: With regard to standard errors, a gap of one standard error **between the edges of the standard error bars** corresponds to p=0.05. A gap of one standard error **between the means** does **not** correspond to p=0.05. See Fig 5 of the Cumming et al. (2007) paper for an example of this. It is still unclear which results are statistically significant. Consider applying a traditional hypothesis test to get the exact p-values rather than estimating them based on the gap between SEs.