Human-Aligned Calibration for AI-Assisted Decision Making

Summary

Communicating model uncertainty to human decision makers is critical for trustworthy deployment of assistive AI systems. Prior work, e.g., Vodrahalli et al 2022, have found that people may respond better to miscalibrated AI confidence scores. What underlies this odd behavior? In this submission, the authors explore this phenomenon, specifically developing a new theoretical framework built on structured causal models. In this framework, the authors find that there indeed are some data distributions wherein calibrated AI confidence scores prohibit humans from uncovering optimal policies by which to make their final decision. The authors then propose to resolve this discrepancy through “human-calibration;” multicalibration is demonstrated as one such method. The authors aim to validate their theoretical insights through empirical studies on the same human data from Vodrahalli et al 2022.

Strengths

A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results. You can incorporate Markdown and Latex into your review. See /faq. This paper has many strengths and was an enjoyable read. The paper was very well-written and incredibly well-motivated. As AI systems increasingly move to real-world applications, it is paramount that they are designed with the human user in mind. The theoretical framework from this paper offers a nice contribution to advance our understanding of the nuances of the way that AI output is communicated to humans to best support decision making. Some further strengths: - The specific problem of looking at monotonicity vs. discrepancy from monotonicty in confidence values is interesting and nicely expands on prior work. - The authors make an excellent point on lines 110-114 that human may take diff decisions with same level of confidence; it is good to see that their model incorporates sources of noise ontop of confidence. - Nice that their method is claimed to be general to other calibration methods. - The mathematics seems sound and steps naturally follow from previous; though while I’ve attempted to check through it, there is a possibility I have missed something (hence my own confidence score of 3).

Weaknesses

My biggest misgivings about the paper fall into two categories: 1) the discretization of human confidence, and 2) the empirical validation in Section 6. I will discuss each in turn. I am also concerned that the authors do not adequately discuss limitations. But note this further in the Limitations section of the review. (1) Discretization of human confidence: - The authors cite two papers from the neuroscience literature suggesting that humans have “discrete confidence”; however, the studies in these papers looks at quite different decision making tasks than are considered here. The tasks are much lower level and rote, e.g., simple perception and motor tasks, at least from my reading. These are quite different from the kinds of rich, high-level cognitive tasks that involve say, coming up with a treatment plan for a patient. As such, I feel the methods here ought to be able to handle finer-grained representations of human uncertainty/confidence rather than too coarse binning. - I am very open to discussing this — can the authors shed light on how applicable their theoretical results are to different choices of discretization? - Specifically on Section 6, I have concerns on the choice of 3 bins. 3 bins are incredibly coarse. Why this number? For instance, the “mid” confidence level spans a confidence range of 0.5 (out of the normalized 0-1 scale). I recognize that the binning was done here to ensure approximately equal counts across all bins. However, my sense is that this could yield obfuscate some of the nuance in the results? Have the authors considered a finer-grained decomposition, say to 5 or 7 bins? If nothing else, I think this should be discussed in a limitations sections as I imagine this could be an impactful design choice wrt sensitivity in the empirical results. (2) Empirical validation - I am struggling to understand the design of the validation in Section 6. What are the baselines? What are the authors trying to show here to validate their theoretical insights (and which aspects of the theoretical insights?) It is not clear to me what is being shown here. When the authors say “participants benefit from AI advice” in Table 1 [line 333], how is this different from the original paper from which the data is sourced? - The authors note that the results (which per the above, I’d be keen to hear explicated more – what is the expected behavior of these graphs, relative to what baselines?) are weaker for the Census task. Do the authors have a sense for why this may be? - The authors grouped the results of cases where participants were told that an AI system had provided the confidence with those told a human had (footnote 9 on pg 8). Is this valid? Could have influenced participants’ uptake of the knowledge? Have the authors explored how results stand up if they decompose / keep those groups separate? Do trends still hold?

Questions

- Many of my questions are listed in the Weaknesses section. In particular, it would be great if the authors could clearly spell out which parts of their experiments (Section 6) validate which parts of their theory. I am open to reconsidering some of my critiques from the Weakness section if this is clarified a bit better. - Why 8 or 10 bins? [line 298] - Could you expand please on the confounding factors in the data from other counties? I am not entirely sure why you threw away that data? [lines 287-288] My understanding is this was because those participants were told the AI had a different accuracy? This may be worth exploring (perhaps in future work) – how do your theoretical results hold up across wider swaths of data?) - How does this relate to RLHF? Please spell out the connection a bit more, or o.w. I actually think it’s unnecessary to mention here. [lines 73-74]

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors do not adequately discuss limitations, nor any potential negative societal impact. I would be keen for the authors to provide their thoughts on these two points during the rebuttal/discussion period.

Summary

This paper proposes a causal model of AI-assisted decision-making that considers both the classifier’s and decision maker’s confidence scores. The authors demonstrate theoretically why it may be challenging for the human to identify the optimal decision-making policy even if the classifier is perfectly calibrated. The authors propose a remedy by providing theoretical results using multicalibration. The authors then provide experimental results to measure misalignment and miscalibration in real-world datasets.

Strengths

- The paper was generally well written and lays out the proposed causal model of AI-assisted decision-making in a step-by-step manner, making it easy to follow - The paper covers an important, growing topic on AI-assisted decision-making and highlights the need to consider notions of both human and AI calibration

Weaknesses

My main concern with this paper is the lack of empirical results that demonstrate an *improvement* in $\mathbb{E}_\pi [u(T,Y)]$ (i.e., show results that the AUC would improve in the H+AI condition as compared to $\pi_B$ or $\pi_H$, which is a commonly used definition of complementarity) as stated as a contribution in L62. While they provide a theoretical result (Thrm 5) to guarantee near-optimality under $\alpha$-alignment, I do not believe the authors implemented multicalibration (the proposed solution in Section 5) in their empirical results. The ideal result would be to show that without multicalibration, human decision makers more often do not achieve complementary performance, as shown in Table 1 last column, and then this could be corrected by introducing some version of multicalibration. Such an implementation would suggest how the theoretical result could be translated to practice and could be very impactful in the human-AI decision-making literature.

Questions

Major question: Could the authors please address the concerns with the empirical results above? Minor questions: - In L121, you state that "allow the classifier’s confidence to depend on the decision maker’s confidence". What is a realistic example of a classifier that takes in decision maker confidence? - Why is Eq. 4 important to measure? - Are there any potential issues that can be introduced if you represent human confidence scores in different ways (e.g., binning into low-med-high or using real numbers like 0.7/0.8)? - Can you provide error bars for Table 1? - The term "misalignment" is introduced in the empirical results and seemed to be an important quantity of evaluation, could the authors clarify why the term was not more formally incorporated in the earlier sections?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This paper theoretically analyzes human decision making in the presence of AI-provided predictions and confidence estimates. They do this by defining a causal model to characterize AI-assisted decision making along with a sensible assumptions about a rational decision-maker's policy: namely that their probability of making a positive classification is monotone in the confidence values. Under this assumption, the paper shows that the traditionally accepted idea of "calibrated" confidence is not enough for a rational decision maker to make optimal decisions. However, a new notion of "human-alignment" does provide this guarantee, and in fact is achievable via the well-known idea of multicalibration. The paper also presents some experiments to validate their assumptions and to argue that human-aligned confidences might lead to better decision making.

Strengths

This paper approaches the question of AI-assisted decision making from first principles, allowing them to ground their assumptions explicitly and provide a compelling theoretical analysis under their model. The flavour of results in this paper is quite interesting and their techniques build upon and connect a some rich areas of theoretical research. Lastly, the exposition of the paper is quite easy to read, with a good breakdown of high-level ideas and detailed maths in the appendix. I believe this work would be of notable interest to several communities at NeurIPS.

Weaknesses

To me, the weakest part of this paper is the experimental section, which in my opinion doesn't add a huge amount of insight into the results. I like the idea of empirically demonstrating the monotonicity assumption, however the interplay between misalignment and the human decision-makers' ability to perform the task well is mentioned like an afterthought. It would be amazing to see a scientifically valid empirical claim that human alignment leads to better performance with AI-assisted decision making. This is likely a significant task to carry out, and it is fine to claim as beyond the scope of a theoretical paper. However, it would make this an extremely strong submission. I would also like to see a discussion of the limitations of this work.

Questions

In several of the experiments, the human performance is worse than that of the model itself. This could be due to several factors, but one reason could be that these humans were not experts at the specified task. In terms of your SCM, they aren't using a lot of latent information V that is hidden to the model. Is there a way to include a "competence/expertise" parameter in this SCM to model, and even possibly correct, a non-expert decision maker so that their performance exceeds that of the AI system?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Limitations of the work not discussed. What are the limits of this papers' ideas in terms of both theoretical follow-up and real-world usage?

Summary

The paper studies a binary decision-making process augmented by AI assistance. The decision maker optimizes a utility function satisfying certain natural conditions. The decision rule depends on the decision maker's (DM) confidence and a classifier's confidence. It is shown that there is a lack of alignment between DM's and classifier's confidences; the optimal rule might not have a natural property of placing more trust in predictions with higher confidence values. The paper then proves that a (near-)optimal decision rule with such a property exists if both confidences do not appear contradictory, with high probability, to the rational decision maker, a property dubbed "human-alignment". The paper connects this property with multicalibration algorithms for the classifier's confidence function. The paper illustrates the approach on real human data.

Strengths

The paper introduces a theoretical framework to study optimal decision-making in binary problems, where the decision rule depends on the DM's and classifier's confidence. The results like this are needed to understand the decision-making process when interacting with AI, particularly now, when strong AI systems are making their way into everyday lives. The theory was illustrated with a real-life data experiment, where the paper asks questions about misalignment, the influence of AI confidence on human confidence values, and a comparison between decision policies.

Weaknesses

The setup considered in the paper is very simple. It deals with binary decision problems and the experiment provided in the paper actually does not use any modern ML classifier (see footnote 8). Section 6 would significantly improved in terms of clarity. In particular, * In the data description part, it is not clear how the labels are defined. For instance, for the "Cities," does y is random (line 283), or does it encode the correct answer (in which case, e.g., h='low' would correspond to a human having low confidence in the correct answer). In a similar vein, it would help to explicitly define how are empirical values of $\mathbb P(Y=1|(X,Y)\in S_{h,\lambda(b)})$ estimated (line 319; what exact data are used). * Figures 2 and 3 use an "alignment violations" term, which is not defined. * Figures 2 and 3 should be better explained: each figure contains a set of four plots, each with from 8 to 10 sets of triplets of bars. In particular, the monotonicity in lines 318, 327 should be discussed in a more reader-friendly way. * The data used in the experiment were aggregated in various ways. The claims like "support for our hypothesis that (rational) decision-makers implement monotone AI-assisted decisions policies" should probably be treated with caution.

Questions

It would be interesting to see how the Authors envision future work in the context of LLM-based systems, multi-categorical or sequential decision-making.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper does not have a limitations section. Natural limitations include the setup's simplicity or the experiments' limited scope.

Response to Authors - Adjusted Score

Dear Authors, Thank you for your thorough rebuttal. I appreciate the effort you went through to produce clarifying results around discretization, and the care in which you responded to my (many) points. Your clarification on what aspects of the empirical results validated theoretical results, as well as clarifying the baselines and set-up, lead me to adjust me score. I have edited accordingly. I would encourage the authors to update the next iteration of the manuscript to more clearly highly the correspondence of empirical to theoretical results as they have here. I appreciate that the authors will update their Limitations section as well. I think the authors did an excellent job delineating the distinction between generalizing results across vs. for a fixed task. The latter of which would be great to add to the Limitations (or generally the Discussion); the authors' notes on why a human study is warranted, but til sufficient for future work, is well justified here. Having this in the updated manusript text would be useful. Thank you as well for clarifying the choice of 3 bins. The lines referenced in the text do provide further detail on this point; however, I think the text (and reader) would benefit from including a higher-level note about the relationship between the amount of discretization and data needed for the estimation. "In this context, we would also like to clarify that, while we focus on AI-assisted human decision making, our theoretical results do not depend on who (AI model or another human) gives the advice. As long as the advice comes in the form of confidence values, our results are valid." <-- This is a powerful point as well. I look forward to the authors' promised expanded discussion in their revision.