Effective Human-AI Teams via Learned Natural Language Rules and Onboarding

Summary

The authors propose an HAI system called IntegrAI that should help people make better decisions about when to defer to AI predictions, make predictions themselves, or combine their decisions. As part of their system, the authors perform semantic clustering using LLMs to find areas of disagreement between AI and humans and distill those clusters into simple rules that can be surfaced as insights to users. The IntegrAI-describe algorithm can be summarized as: 1. Create clusters in joint embedding space 2. Get candidate descriptions of each cluster and embed them into same space 3. Search for counterexamples 4. Update clusters and repeat The key claims are: 1. IntegrAI learns optimal integration decisions 2. Rules generated by the oracle (GPT3.5 in this case) are easily understandable 3. Onboarding calibrates human expectations about AI performance 4. Dashboard insights help humans choose which action they should take thereby improving teaming performance.

Strengths

- the IntegrAI framework is novel and very interesting - the IntegrAI-describe algorithm is a very clever way to leverage LLMs for semantic clustering

Weaknesses

1. Several of the main claims (see list in summary above) are unsubstantiated by empirical results. Specifically: - claim 2 (rules are easily understandable) is not extensively tested and the results comparing it against SEAL seem inconclusive. - experimental results provide only weak evidence for claim 3 and claim 4, and it remains unclear whether a combination of Onboarding + Rec is actually useful (e.g. if looking only at effect size means, then Onboarding alone has both higher performance and lower time than Onboarding+Rec) 2. Additionally, I think there may be some issues in the significance testing for the results in Table 3. Given the large reported standard errors, it seems unlikely that the p-values would be as small as reported. Minor suggestions: - small typo in figure 1: no panel "d)" - small type on line 57: should be "compared" instead of "compare"

Questions

See sections above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

4 excellent

Limitations

The proposed framework is a 2nd-order framework in the sense that: - 0th order is if human or AI individually make decision - 1st order is if human sees AI decision and can change mind - 2nd order is another AI sits on top of this process and tells human whether they should change their mind. For this type of framework to be useful, there needs to be a complex setting where humans struggle to model their own uncertainty/limitations as well as those of the AI. I think the traffic light setting may not be adequately complex to really let this framework shine and show off its capabilities. I think it would really strengthen the paper and provide evidence for the key claims if the authors could evaluate on a more complex setting (e.g. multi-class, greater ambiguity, etc.) Also, the proposed approach requires collecting paired data. This is a potential limitation if a lot of expensive human data is required. It would be great to see a discussion on how much paired data is required and whether this is a major limitation. An additional limitation of the paired data collection procedure the authors use in their particular case study is that integration can't be detected (i.e. currently authors only detect whether the human used the AI answer or didn't).

Summary

This paper discusses some research problems a very interesting scenario where human and AI need to collaborate to achieve a certain goal. The authors propose to learn rules grounded in data regions and described in natural language, which illustrates how the human should collaborate with the AI agent. The paper also proposes a novel algorithm, region discovery algorithm, which uses an iterative procedure where a large LM describes the region while distinguishing it from the rest of the date. Experiments show positive results when following this protocol.

Strengths

1. It tries to solve research problems in a very interesting domain. 2. The user study is extensive and its setting is convincing.

Weaknesses

1. The presentation needs huge improvement. e.g., some figures are not clear and text is vague. 2. The study was conducted with 25 human participants, which might not be representative human users.

Questions

Have you considered about the "cost" during human AI collaborations? e.g., time. If fully automated pipeline can work well with less cost, that might be our preferred one.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The relatively small scale human evaluation.

Summary

This paper presents an approach to allow effective human-AI teaming. The approach describes a process of semantic discovery of regions in an embedding space following by generating natural language descriptions of the regions and then learning refinement of the regions using counterexamples. Once regions and region descriptions are generation, human onboarding is performed alongwith proposals for whether to rely on AI for decision making or not with an explanation. Analysis shows that human-AI team performs better than just AI or humans. User study shows very interesting results (role of onboarding, role of providing recommendations, impact of time per task with/wo AI teaming and recommendations).

Strengths

- This paper presents an interesting and practical approach for developing a human AI collaboration system with a focus on both machine learning aspects (learning AI integration functions) and the HCI aspects (onboarding + natural language descriptions) of how human-AI teams can really work in balance (staying away from over-reliance on AI and under-reliance on AI). - The use of chatGPT as an oracle is also a great workaround for lack of ground truth data - Evaluation shows that the learnt AI integration function leads to lowest loss at test time and also minimum number of region proposals (which may not overwhelm the user) - Groundtruth cluster analysis of regions on the BDD dataset also shows high correlation compared to other region discovery approaches. - the contrastive approach for region description is also very intuitive and gives good results.

Weaknesses

- There's a lot covered in the paper in the limited space and the focus seems to be on region discovery and region description algorithms as well as the higher level human-AI teaming and overall system level evaluation study. It seems like one could do more extensive analysis with additional experiments with more datasets on each of the sections.

Questions

- There are other approaches for region discovery such as the deep aligned clustering algorithm, adaptive decision boundary algorithms etc (from the fields of open class learning, semi-supervised representation learning, novel intent discovery, etc) What do the authors think about these other approaches for region discovery.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- This paper covers a lot of ground and the authors have also captured the current limitation well in the Limitations section of the paper. However, given all the limitations, the paper presents some very promising ideas and directions of further research on human AI collaboration.

Summary

The authors attempt to improve human-AI collaboration through a system that (1,2) identifies similar regions in a dataset (both in terms of the task similarity and in terms of human behavior—usage of AI advice—on the task) and optimizes for how humans should ideally use the AI in the given region, (3) generates contrastive descriptions for the region, and (4) provides recommendations to users based on 2. They synthetically evaluate each component of their system, and empirically test their system with human users across two tasks.

Strengths

1. Problem setup and optimization solution are straightforward and make sense. 2. Empirical analysis using humans to assess their system (with multiple datasets). 3. Generally well-written. See “Weaknesses” for a few suggestions I have to clarify some parts (section 4).

Weaknesses

1. Only testing on one task 2. The experiment design has an obvious confounder — the raters are only offered a binary choice, while the AI 3. The setup did not allow distinguishing between ignoring the AI and collaborating with the AI. This should be possible with an adjusted setup — for example, have a 2 stage process for data collection where the person makes an initial decision and subsequently is given the chance to use AI (in section 7, it sounds like you did this, but this was not clear earlier in the paper). Additionally, rather than having people submit a discrete label, you can ask them to submit a continuous value (e.g., something like a confidence). Note that this could be done just for the dataset collection process to approximate $R^*$, and you could still use your original setup for the empirical evaluation. (Side note: if you had recorded a “confidence” for the dataset, optimizing for the regions may be easier as this is a differentiable problem now since the $r$ would be continuous-valued.) 4. It’s actually unclear how well this setup can distinguish between 0 and 1 — using the AI vs not using the AI. Since everyone sees the AI’s output, they may be biased based on that while still making their own decision. There are a number of studies that have established the biasing effect this can have. 5. The overall method in 4 (and 4.1, 4.2) is relatively straightforward, but it is a bit difficult to parse initially (some key points are buried in the details). I would recommend a concise description at the start of each section to describe what will be done in that section. (e.g., at the start of 4.1, saying something like “We generate an embedding for each datapoint by concatenating a task embedding with the human’s decision to use AI. We then define an objective function to jointly discover data regions and optimal usage of AI; it seeks to maximize task instance similarity while enforcing similar (optimal) human behavior (usage of the AI) on that task”). In my opinion, this high level overview will make the sections easier to read. 6. In the appendix, it seems the empirical analysis on the second dataset results in not significant benefits to using the proposed system. This is OK, but I would recommend adding discussion about this in the main paper. It is important to understand when your system may and may not work, and where improvements may be needed. For example, it is possible the disparity in human and AI accuracy is large enough that simply trusting the AI is a better strategy than trying to assess when to trust the AI more.

Questions

1. How important is the embedding choice for the region discovery algorithm? 2. There seem to be a lot of hyperparameters here. How important are the choices here? For example, the upper and lower bound on fraction of points in a region seem to be an important consideration — were there any experiments to assess what is reasonable for (a) human users and (b) for the generated region description? How many regions is reasonable for onboarding? 3. For the region parametrization, are the CLIP embeddings normalized? 4. For region description, (1) the initialization and (2) the embedding could matter a lot. Did you do any analysis of the variation in the final description with respect to these? 5. What do you use to generate the initial text descriptions for each image? In 4.2 you reference a few possible methods, but don’t indicate which you use. 6. In Table 1, I am unclear on what the numbers represent. How is the error calculated? 7. For the empirical studies, is the task order randomized? Also, for the people who did “predicts alone” and “predicts with help of AI”, were the tasks where they predicted alone randomized person to person?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

As the authors recognize, it would always be nice to have more empirical studies (e.g., with more users, more datasets, etc.). The authors also recognize the potential negative outcomes of this work (incorrectly identified regions and/or recommendations could incorrectly bias users to under/over-trust the AI).

Clarification on rebuttal

Thank you for the detailed response! ## Claims I'd like to make sure I understand the key claims you are making. For example, on lines 34-36 you write: "We further propose to surface the AI-integration decisions found by IntegrAI as recommendations to the human within an AI dashboard used after onboarding. This AI dashboard helps the human evaluate which action they should take, thereby leading to effective AI adoption for enhanced decision-making." and on lines 230-231 (in a section titled "Onboarding and Recommendations to Promote Rules") you write: "We accomplish this through an onboarding process followed by test-time recommendations, as described next." and at the end of section 5 in line 262 you also write "These recommendations are shown as an aid to the human to reduce their decision-making time." all of which I summarized as claim 4. In that case, your full proposed method would be Onboarding (ours) + rec. My understanding based on your response is that you are changing all of these claims about dashboard/rec now to instead be hypotheses, which are then refuted in the last few sentences of your analysis. The updated claims are then that the dashboard is actually not helpful (i.e. rec increases time but not performance) and that your proposed method is just Onboarding (ours)? Would you say your contributions are then 1) a novel method for learning rules (i.e. the content); and 2) a novel onboarding method (i.e. the delivery method)? Is rec just a baseline then that acts as an ablation of the delivery method since it involves showing rules to people but not onboarding them? Totally fine for rec to be a negative result by the way, I strongly believe we shouldn't punish authors for including negative results since these can also be beneficial for the community. If that's the case, then: - comparing "Onboarding (ours)" vs "Onboarding (baseline)" would test whether the rules you come up with are actually helpful. - comparing "rec" vs "Human-AI" would also test whether the rules you come up with are actually helpful. - comparing "Onboarding (ours)" vs "rec" would test whether the onboarding procedure you came up with is actually helpful. - comparing "Onboarding (baselines)" vs "rec" would also test whether the onboarding procedure you came up with is actually helpful. - etc. I think these comparisons (though many of them won't be statistically significant) reveal the following insights that disentangle what each component contributes: - The rules you come up with lead to increased performance - The onboarding procedure leads to decreased time (worth noting that this is at the cost of increased training time, so this only becomes worthwhile for tasks that need to be repeated many times) ## Statistical testing Thank you for the correction, the test results make a lot more sense if those were standard deviations in the table though I think there may still be typos or some other problems. For example, shouldn't the p-value for Onboard (ours) vs Human-AI be 0.004 rather than 0.0004. Also weren't 6 tests conducted in total rather than 5 (affecting the Bonferonni correction)? Finally, if you are relying on statistical significance testing to substantiate your claim, shouldn't you also test other claims you make like that "Onboarding (ours)" outperforms "Onboarding (baseline)" and "AI only"?

Author Rebuttal

Dear Reviewer, I hope you get the chance to read our rebuttal that tries to address your concerns, if it does, please consider adjusting your review in response.