Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems

Summary

The paper proposes a white-box approach to human-machine teaming, in which human teammates can see their virtual counterparts’ policies and adjust them accordingly. The framework is built on top of differentiable decision trees, and the authors propose contextual pruning as a means of simplification for training purposes and readability by the human team members. They use this framework to quantify the benefits of interpretability in HMT and the tradeoff between high interpretability and high accuracy, the latter typically better achieved by traditional black box solutions. The authors use their framework to conduct a thorough statistical analysis comparing different white and black box conditions and use their findings to list a series of guidelines for HMT.

Strengths

- I enjoyed reading this paper, and I think the idea is novel, and the contribution of this paper to the field is significant. Interpretability is crucial in HMT if virtual agents are to be seen as real teammates instead of mere tools. Being able to understand these agents' policies and change them to agree with the human's beliefs is one step in that direction. The key point here is that the changes made by humans may not be the best ones (typically achieved by blackbox solutions), but they may be the ones that foster more collaboration between humans and machines. - The paper is very well-written, and the ideas are very clear. The steps for reproducibility are detailed, and the statistical analysis is comprehensive.

Weaknesses

The authors propose a pruning strategy to simplify training and interpretability but provide no evidence supporting either. It would be helpful to validate such claims with time and memory usage assessment and IV1-C1 analysis with and without pruning. The authors briefly discuss the former in C.3.2, but it could be improved by displaying the numbers.

Questions

- In line 230, the authors state that contextual pruning significantly improves the ease of training. Are the two pruning strategies enough? I am unfamiliar with DDT, so I wonder if this is novel. How is this different than traditional pruning methods in DDTs? - As noted by the authors in line 357, some participants outperformed the maximum performance of IVI-C5 in teaming iterations 3 and 4. I wonder if those participants show an increasing trend in performance from iteration 1 to 4? I cannot tell that from the scatter plot. I am asking this because I am curious to know if the improvements are due to randomness or if humans are applying a conscious strategy when modifying the agents’ policies, leading to incremental improvements at every iteration.

Rating

8

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors discuss the limitations of their work in the Appendix. I would argue there's one more limitation: It seems performance in the mixed-initiative scenario is highly dependent on presentation, as validated by better performance improvements in participants more familiar with Trees. This work shows that interpretability is also relative. What is interpretable to me may be convoluted to another, which may also affect the team's performance. It would be interesting to check for differences in IVI-C1 with different visualization methods in future work.

Summary

In this paper the authors present an approach to human-AI teaming in the common Overcooked domain via Interpretable Discrete Control Trees (IDCTs), which are differentiable decision trees which the authors visualize and make controllable. The authors present two examples of where existing blackbox models may demonstrate a gap in teaming performance. They then present results of a human subject study that demonstrates that authors' approach is outperformed by a fictitious co-play baseline.

Strengths

Human subject studies studying human-AI teaming are still relatively rare due to their difficulty. As such, a new study is always beneficial and of interest to the community. The authors also include many variations of their approach, which is beneficial in terms of more deeply understanding the mechanisms of human-AI teaming.

Weaknesses

I like this work, but at present this paper has a number of key weaknesses holding it back. First, the current text of the paper contains a large number of unsupported claims. Almost all of the text in italics represents claims that are not substantiated by an argument, a citation, or the results of the paper. There are similarly design decisions that are not motivated or explained, such as why the authors chose to use IDCTs. Second, Section 3 introduces the possibility of a gap in teaming performance with current models. However, the evidence for this is presented as two examples. This is not sufficient to demonstrate that this specific approach has failure cases but not to demonstrate a need for the approach the authors' propose. I'd recommend working on a more full survey of existing Overcooked approaches to quantitatively evaluate the rate at which collaboration gaps may be occurring. This section represents the primary motivation for the authors' work, and so it being a weak point makes the whole paper weaker. Third, the authors do not give full technical details for any of the approaches, not the approach of Carroll et al., nor their own ICDT approach, nor their implementation of fictitious co-play. This is an issue as it makes it very difficult as a reader to understand what has been done at a technical level. For example, it's unclear to me what size the ICDTs used in the study were, I only know that the users were limited to expanding the tree to a depth of 4. But this doesn't tell me if the initial tree had a depth of 3. Similarly, the pruning approach only seems to remove redundant nodes so it's unclear how much this would actually prune (the authors state 8-16x smaller but it's unclear to which trees they refer). Given that a major problem with the ICDT's performance in the user study was their performance, these details are crucial. While there's an anonymized GitHub in the appendices, it should not be necessary to go through it to understand the work. Fourth, while the human subject study seems well-designed, certain details are unclear. It's unclear what population is being drawn on for the participants, though I would guess university undergraduates given the demographics. It's unclear how knowledgeable these participants were about AI except for one mentioned figure around familiarity with trees. It's unclear why the authors stopped recruiting at 50, since 10 per condition is considered an absolute minimum. Finally, the methodology is somewhat unclear, its unclear what order the participants experienced the domains. Was it consistent, if so, why? If not, there were 10 conditions, not 5. Finally, the results do not seem to support the authors' claims. From the result, my takeaway is that participants preferred performance over interpretability. This goes against several of the authors' claims in the paper, most notably "As seen through these findings, the ability to interact with an interpretable model is perceived significantly better across several measures". The authors also make a claim about trending towards significance, which is not statistically sound.

Questions

1. What was the size of the ICDTs used in the study? 2. Did participants experience the two domains in a fixed or random order?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The major limitations come from the poor performance of the ICDTs and the decision to limit the size of the human subject study. The authors do not sufficiently address these limitations.

Summary

In this manuscript, it focuses on the collaboration in human-machine teaming (HMT) based on interactive and explainable systems. In order to address the existing issues, such as decoupling, the author(s) explored an interesting paradigm in HMT and proposed some guidelines based on the study. Also, the author(s) pointed out some potential research directions, although some of the conclusions are intuitive, but interesting.

Strengths

In my opinion, the strengths of this manuscript are as follows: 1. Demonstrating that current HMT approaches struggle to adapt to human-preferred strategies, often resulting in suboptimal team performance. 2. The proposed architecture enables end-users to modify AI behavior, facilitating a feedback loop for team development.

Weaknesses

In my opinion, the weaknesses of this manuscript are as follows: 1. As the author(s) mentioned, due to the population setting, the findings have a population bias and may not generalize to wider situations. 2. The cited work is a little out of date. This makes this manuscript seem not ready for publishing at conferences such as NeurIPS. 3. Some descriptions in the main text are not so clear. For more details, please see the Section "Questions" below.

Questions

I read the manuscript, and I have the following questions/comments. Thanks. 1. We need to balance sparsity and stability in designing a learning algorithm. In this manuscript, the author(s) adopted L1 regularization; I am not so sure if it will cause some issues due to the instability of l1. 2. Regarding how users interact with AI across repeated play under different factors, the author(s) performed some human-subject-involved studies. About the population used in the study, can you discuss the implications of the population bias in more detail and how it might affect the generalizability of your findings? Also, maybe sex/age-based difference analysis is also worth exploring. 3. If possible, could you explain more about the potential challenges in scaling the human-led policy modification approach to more complex and dynamic environments beyond the Overcooked-AI domain? 4. Regarding the citations, I do not think the current references are sufficient, at least, for a manuscript preparing to submit to computer conferences such as NeurIPS, since the newest citation was two years ago, and there are only 4 such 2022 citations. 5. In Line 58, maybe "reinforcement learning" should be with its abbreviation, not until Line 70. 6. It would be great to be with the full name for PPO, such as Proximal ... (PPO) Some other format issues in references: (1) The author's name, sometimes using the abbreviation, sometimes not, such as Ref.[1], Ref.[33]. (2) Sometimes, the conference name has both the full name and its abbreviation, and sometimes not, such as, Ref.[4] vs Ref.[5]. (3) In Ref.[17], "ai"=>"AI". Please check carefully; it would be great if the author(s) could correct these issues. I would like to consider adjusting my current score based on the responses from the author(s). Thanks.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper focuses on developing strategies to enhance transparency and interpretability in human-AI teaming settings. Based on my understanding, two collaboration contexts have been considered: human-preferred collaboration and AI-preferred suboptimal teaming strategy. The authors have implemented specific strategies to tackle these contexts and address interpretability. The implemented strategies have been evaluated on the Overcooked game with real human users.

Strengths

The paper is well-motivated, particularly in its approach to learning human preferences and using reinforcement learning (RL) to train the agent. The paper provides valuable insights into designing human-AI collaboration interfaces to enhance user trust and acceptance, addressing a timely and relatively unexplored domain.

Weaknesses

The main weakness of the paper is that it is very hard to follow, and the methodology lacks transparency. For instance, the paper does not present a single method to investigate explainability. Overall, the chosen modifications and the designed collaboration setups are not clearly presented. It is not straightforward to link Section 4 (Methodology) with Section 5 (Studies). Similarly, the experimental results are very hard to follow. The numbers are presented but are not straightforward to interpret or use for cross-checking the claims. In particular, the evaluation metrics are not clear. Additionally, only one collaboration environment has been considered. The findings might be biased toward this specific collaboration task.

Questions

Three main questions are: - How do you envision improving the transparency of the proposed methodologies? How can Section 4 be better linked with Section 5? - What are the evaluation metrics for the chosen collaboration settings? How is interpretability evaluated? - Are there any other collaboration environments that can be considered for this line of research? How do you envision these findings being generalized to other collaboration tasks?

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors discussed the limitations and the broader impact of their work adequately in the appendix.

I maintain my decision

Thank you for the rebuttal and additional results. I think this is a good paper. I maintain my decision.

Re: Rebuttal by Authors

Thanks to the authors for their detailed response. My concerns around the claims and the results not supporting them, remains unchanged and so I am choosing to maintain my score.

Updating

Thanks for the responses from the author(s). The responses from the author(s) clarified some of my concerns to some extent. I increased my score from 4 to 5.