Diverse Conventions for Human-AI Collaboration

Summary

This work suggests employing cross-play rewards to improve the measure of diversity in human-agent collaboration tasks, and presents an innovative training paradigm, namely, mixed-play, to address the challenge during optimizing cross-play, referred to as the "handshake" by the authors. As a result, the developed agent CoMeDi demonstrated superior performance in comparison to other baseline agents when collaborating with humans in the OverCooked game.

Strengths

1. This paper is well-written and readable, and provides detailed appendices and codes to supplement the paper's algorithms, experiments, and implementation details. 2. This work tried to tackle an important problem domain - ad-hoc team play in human-agent collaboration research. 3. The experimental section fully validated the problems and necessity of self-play, cross-play, and mixed-play. The results of user studies on overcooked show the effectiveness of the proposed CoMeDi approach.

Weaknesses

1. The behavior of human players tends to be non-stationary and noisy-rationally[1]. I am particularly interested in how the CoMiDi agent reacts to a player exhibiting random actions (such as a novice), or how the proposed CoMiDi method resolves this issue at the algorithmic level. This is because, upon experiencing the CoMiDi agent on the webpage (random1, random3, and scenario1_s Layout), I found that when I either refrain from any operation or deliberately obstruct one side of the path, the agent appears to be unintelligent and would not go around from the other side. Such performance contradicts the authors' claim that the CoMiDi agent surpasses human-level performance. Does this imply a lack of diversity in training or is it due to insufficient training? 2. The experimental section seems to resemble an ablation study process, with an absence of comparisons to state-of-the-art (SOTA) approaches, such as FCP[2], HSP[3]. While the authors assert that these SOTA methods are not drop-in replacements for self-play, I believe that the FCP and HSP approaches are not fundamentally distinct from SP, and they can be considered as improved methods of SP. I recommend that the authors conduct comparisons with these methods to make the proposed methods more solid. 3. The experimental section only discusses the results of partnering with expert humans. I am curious whether the authors have evaluated the CoMiDi agent's performance in collaboration with humans of varying skill levels (such as novices). Moreover, do all human partners prefer to collaborate with the CoMiDi agent over other agents? I believe user studies are crucial in the human-agent collaboration domain. Consequently, I suggest that the authors allocate more space in the main text to discuss user studies. 4. It is commendable that the author discusses the scenario where the number of teams exceeds two in the appendix. I am particularly intrigued by how the CoMiDi approach or the convention-based method can be extended to more intricate tasks, such as Google Football. It would be appreciated if the authors could offer insight in this respect. [1] Anirudha Majumdar, Sumeet Singh, et al. Risk-sensitive inverse reinforcement learning via coherent risk models. 2017. [2] DJ Strouse, Kevin R. McKee, et al. Collaborating with Humans without Human Data. 2021. [3] Chao Yu, Jiaoxuan Gao, et al. Learning zero-shot cooperation with humans, assuming humans are biased. 2023.

Questions

1. Why the CoMiDi agent is not as intelligent as expected for some relatively simple behaviors? 2. How the proposed method performs compared to the SOTA methods on OverCooked? 3. How does the CoMiDi agent perform in teams with different levels of humans? 4. What is the training manner of the proposed method, is it sequence training or joint training? 5. Did the author encounter any challenges other than Handshakes during training?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

As discussed above, this work lacks the comparison of SOTA methods, and the user studies are not sufficient, which would limit the reliability of the proposed method. Besides, the authors should have further discussion on the scalability of the proposed method, such as in more complex environments.

Summary

The paper introduces a new approach for generating diverse conventions by using a training objective that balances increasing self-play performance and minimizing cross-play performance with previously discovered conventions. To avoid handshake mechanisms that can game the proposed objective, the technique also involves a mixed-play bonus/penalty, that leads the agents to be uncertain about whether they're currently playing in self or cross play. The advantages of the proposed method in recovering meaningfully distinct conventions (relative to prior work) are shown experimentally and with a user study.

Strengths

### Originality and significance Although this is not my main area of research, this approach seems novel and highly relevant to the community. There seems to be a lot of interest in conventions, especially in the context of human-AI collaboration. Given the empirical performance of the proposed method relative to prior approaches, this seems like a significant contribution to the subfield. The theoretical motivation to the method is an extra nice feature of the approach. ### Quality and Clarity I found the approach as quite a clever way of choosing the space of possible conventions to consider (by restricting to ones that maximize self-play reward but minimize cross-play reward), avoiding the problems of assuming adversarial conventions. The concept of handshakes was well explained, and the idea of mixed-play seems like a good (although a little "hacky") solution. The related work section was done well and the overall motivation for the paper was very well executed. Overall, the paper was well-written and easy to read.

Weaknesses

- The authors do not compare to behavior cloning play (i.e. "human-aware RL" from [4]) or similar approaches because it assumes access to expensive human data. I think it would still be very interesting to make this kind of comparison, because one long-standing challenge is to be able to effectively collaborate with humans without human data [26]. It would be interesting to see how far diversity approaches such as the one presented can get us. In other words, this could even enable to answer interesting questions such as "how much actual human data are these diversity approaches worth?". This seems especially interesting given the qualitative user reports. - Rules of balance beam are pretty unclear – it might be worth trying to clarify them further - A relevant recent work that the authors might not be aware of is: Who Needs to Know? Minimal Knowledge for Optimal Coordination [Lauffer, 2023]

Questions

- There's one aspect of the paper that I don't understand (and might be worth clarifying more somewhere – unless I missed it). It seems like the whole paper is about generating diverse conventions, and so is the proposed method (e.g. in line 274 one tries to generate $\pi_n$). However, at evaluation time (e.g. in Overcooked) one must only choose one policy for CoMeDi. How is that done? In the conclusion, you say "we use a simple BC-based algorithm for training a convention-aware agent", but I don't see this referenced elsewhere? Minor typos: - line 129 spacing between commas. Footnotes have a space before preceding periods or even come before period (e.g. 140, 96)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

With real humans, there are cognitive biases that might have them deviate from optimality arbitrarily, leading them to conventions that wouldn't be discovered by CoMeDi. Will this method will only work when humans are close enough to optimal? It might be worth acknowledging this as a limitation.

Summary

The authors develop CoMeDi which optimizes a cross-play objective but avoids handshakes that can prevent cooperation using mixed play. The algorithm is evaluated in three environments of increasing complexity and on an Overcooked level with real human users.

Strengths

The paper is written well and the ideas are well motivated both by filling gaps in the literature and by tackling key challenges that the authors introduce using conceptual examples as well as toy experiments. The quantitative and qualitative evaluation on Overcooked with real human participants is the gold-standard for studying Coordination. Furthermore, the CoMeDi algorithm is an elegant solution to the problem of generating diverse conventions and cleanly avoids many of the pitfalls that can arise from adversarial training. The authors have released a playable version of their agent to the web.

Weaknesses

The Overcooked analysis was only done with a single grid out of many. This leaves open the possibility that the algorithm doesn’t work well on the more varied set of environments that are typically studied. I would raise my score if the algorithm could be evaluated on the full Overcooked suite (either with people or with baseline agents). This is critical because otherwise, the results might have arisen from a cherry-picked experiment. No formal analysis of the algorithm — how does this approach scale when the number of possible conventions or agents grows? Little evaluation done on the Overcooked environment. Besides overall reward, how well do these agents fare? How far off are the cross-play and self-play rewards from the results with people? How many conventions were generated for the Overcooked environment?

Questions

We need more information about the overcooked results. How well does the final policy perform in simulation against other policies. Not just in terms of human performance but self-play & cross-play with the other computational agents? This is a fast moving space, could the authors devote some words to how this approach differs from Adversarial Diversity in Hanabi or Generating diverse cooperative agents by learning incompatible policies. The latter seems very similar in approach. Were the experiments with people done under time pressure? I noticed when playing that game online that time ticked independently from my own moves. How does that differ from prior work on human evaluation? What does it mean to over-correcting conventions during mixed-play? What are the implications? How carefully do these parameters need to be balanced? Can the same hyper-parameters be used across all Overcooked scenarios?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This paper introduces a new approach to dealing with poor generalisation performance in human-AI settings when actors may have developed different coordination conventions.

Strengths

In general, I think the work is original and very clear in its presentation. In addition, the paper attempts to challenge a very obvious and significant issue in the domain of human-AI collaboration and takes a well-formulated approach to the challenge. More specifically: - I think the problem is well formulated, and in particular the related work in relation to diversity approaches in RL is strong. - I found the method generally easy to follow, and enjoyed its presentation in terms of creating all of the building blocks towards the final algorithm. - On the presented empirical evidence the results are strong.

Weaknesses

Whilst the empirical evidence presented is strong, my concerns with this paper are with the extent of the results and how they actually go about answering the question of this paper. Therefore, my main concern is as follows: - The paper argues that their approach is able to handle an arbitrarily diverse set of other agent conventions. Whilst this may be true, I am unconvinced by the empirical results that may try to back this up. In my opinion, the problem with the experiments that were picked is that there just really do not exist that diverse a pool of conventions in the first place. In particular, in the Overcooked environment I would be amazed if any human player undertook a strategy that was not immediately obvious. Therefore, I would have been very interested in seeing some form of analysis that was able to show that the algorithm was being put up against a diverse set of conventions in its test set. Note, I have played with the demo of the paper and I do think that it appears to be a strong approach even under strange conventions. However, my own testing is probably not a strong enough scientific argument to back this up. I'm happy to bump my score up for the paper if the authors can convince me that the above is not as much of an issue that I believe it might be.

Questions

- The fact that there are two tunable parameters here $\alpha$ and $\beta$ could potentially become a difficult balancing act in other environments. Could the authors comment a little more on how they would suggest selecting these parameters outside of full on hyper parameter tuning? Or is this the only viable option?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I would have been interested to see where the algorithm may feel and if the authors could have stress tested it a little more than in the paper.

Response to rebuttal

Thank you so much for your response, and additional experiments. I found the comparisons with BC-Play particularly interesting! I do think that having more information about the Beam environment in the main text might be useful for readers, as I feel like it is essential to understand that environment's results. Also, thanks for the pointer to equation 2, I had missed or forgotten about it after seeing it – that might be worthwhile to emphasize with a heading or bolded paragraph title. Even after reading other reviews, I still stand by my score.