Auditing for Human Expertise

Summary

While humans and machines oftentimes make differing decisions, it's unclear whether humans make these decisions based on extra factors or information unavailable to machines. To understand this situation, the paper proposes a statistical test to determine if expert predictions are independent of the labels, when accounting for the input features. This idea indicates whether humans rely on different information, and in a sense add additional value unseen by a model. To evaluate this, the paper analyzes doctor predictions in a hospital admitance system, and find that doctors tend to use additional information.

Strengths

1. Statistical approach is well motivated and clearly demonstrates how to test for humans relying on extra information 2. Test allows for flexibility due to choice of L, allowing for different statistical properties 3. Method is evaluated on a real-world hospital dataset, and the connection between the evaluation and the methodology is clear

Weaknesses

1. Method relies on dataset containing pairs that are close in input space, yet distinct in feature space; such a situation might not be generalizable 2. Evaluation is only done on one real-world dataset; a controlled evaluation of the test would give better insights into how the test performs and the impact of different parameters

Questions

1. What are other choices for the function F, and would they be equally valid? 2. For the experiment with hospital admission data, would the results change if instead of using the GBS score, the individual factors that were used as inputs to the score were used? 3. In table 2, what does mismatched pairs mean?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Authors discuss limitations fairly thoroughly.

Summary

This paper proposes a statistical framework for measuring whether, in the context of algorithmic predictions, human experts incorporate valuable information in their decision making that is unknowable to the algorithm. The authors formalize this question as a simple hypothesis test: “are human expert predictions independent from the outcome variable, when conditioned on the feature vector”. The proposed statistical test for “presence of human expertise” is straightforward, drawing on prior literature on testing conditional independence, and provides interpretable p-values. The paper then uses this framework to analyze real-world medical data from an emergency department of a large hospital, showing that physicians do in fact incorporate information above and beyond that captured by a standard algorithmic screening tool.

Strengths

The main strengths of this paper are in it’s simplicity, clear exposition, and scoping of a well-defined problem that it tries to solve. The authors’ give an elegant framework for formally defining the problem of measuring valuable human expertise and their proposed test is quite intuitive. The paper presents all its ideas in a concise manner and also discusses its limitations quite candidly. I also like the thoroughness of the experiment conducted by the authors—it seems well executed and the results are compelling evidence for the validity of their statistical test.

Weaknesses

Some weaknesses of this work: - From an algorithmic/technical standpoint, this paper uses a straightforward notion of conditional independence to define the problem, and a simple binned conditional independence test to solve it. This simplicity is not a bad thing, but it is worth noting that the main contribution of this work doesn’t present a novel technique or technical insight. - The results and experiments of this paper would likely not extend to a high-dimensional setting. Their current experiment uses discrete, scalar features. The authors discuss this in their limitations. A concrete example: how would a test like this work for, say, radiology images, where the human predictive distribution is likely not smooth w.r.t the $\ell_2$ metric. - I understand it’s not an easy task, but this paper would be much stronger if there were additional experiments where this technique was employed to understand the interplay between human and algorithmic decision making.

Questions

- In Section 2, first paragraph: why can you assume X is independent of U, without a loss of generality? - I would be interested in seeing this used in settings where algorithmic predictions are close to replacing human predictions. Radiology images are the immediate example that comes to mind. - Could your test (or a small modification) detect if human experts are using additional information in a negative way? For example, could you use it to detect biasedness in college admissions or medical treatment choices because the expert is relying on information outside of the set of features they should be looking at?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, good discussion of limitations and prior work.

Summary

This paper proposes a method for determining whether a human expert is usefully using outside information that a model does not incorporate in order to make decisions. The goal is to test whether complementarity, humans working with models, is possible for a given task. The paper sets forth an algorithm, ExpertTest, provides some theoretical guarantees, and uses emergency room admissions as a case study for the technique. In the case study, the method indicates that doctors are making use of external information that is not captured in a commonly used risk score.

Strengths

The main strength of this paper is the novel problem that it seeks to solve. Understanding whether and how humans can add their expertise on top of automated decision systems is an important goal, and it is often understudied. It is a creative approach to a crucial problem.

Weaknesses

The primary weakness of the paper, in my opinion, is that the problem setup focuses on variables, or features, rather than also considering the functional form for the prediction itself. Fundamentally, if we are comparing the performance of something like the Glasgow-Blandford score (GBS) to humans, how do we know that the human is not using the same exact input features $X$ as the GBS, but just has a better way of mapping those $X$ to the prediction we are interested in? Why does $U$ have to exist at all for the human to be lending their “expertise” to the problem? That is, let’s say the GBS output is $\tilde{Y} = \tilde{f}(X)$, and the human output is $\hat{Y} = \hat{f}(X)$. If $\hat{f}$ is closer to the actual generating function in the ground truth than $\tilde{f}$, then the human could be “adding expertise” without actually using additional information as defined in this paper. Perhaps the human’s training would help them map these variables better than the GBS algorithm does. If I am missing something here, please correct me, but it’s very unclear to me why the method proposed leads to the conclusion that the expert is using additional features to make a decision.

Questions

1) Early in the paper, the authors equate $X$ with features, such as input features for a model. However, in the case study, the authors define $X$ as a GBS score, which is the output of a risk model. I think the notation needs to more carefully distinguish between what is an input to a model and what is an output of a model, and this may be part of the confusion that I express above in the weaknesses section. Could you clarify this point? 2) In the case study, $\hat{Y}$ is defined as whether the patient is admitted, and the ground truth $Y$ is defined as whether they suffered one of three adverse outcomes. Are these variables correlated? If a patient is admitted, are they less likely to suffer an adverse outcome? It seemed like this might actually be a setup where the “prediction” and “ground truth” are not independent of each other. 3) The third adverse outcome mentioned is initial discharge and readmission within 30 days. Does this mean that anyone who suffered that outcome had a $\hat{Y}$ of 0? I am just wondering if “initial discharge” means that the ER physician made the choice to not admit them. 4) How sensitive is ExpertTest to the choice of loss function?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I think the paper should more carefully address the assumptions in the initial problem setup, specifically what I lay out in the weaknesses section. What model of human expertise is being considered when the assumption is that any additional information is encapsulated in the $U$ variables?

Summary

This paper proposes a hypothesis-testing approach to identify whether a set of predictions made by a human expert uses additional information that is conditionally independent from the input covariates. The paper provides theoretical guarantees for the test in a general setting and asymptotically. The proposed test is then applied to real-world data with emergency room physicians.

Strengths

- The paper itself is written in a way that is easy to follow along, building the motivation and proposed work in a step-by-step manner. - The paper proposes a test that is a novel application of conditional independence work in the statistics literature for a novel use case. - The paper evaluates the theoretical test on a real-world use case.

Weaknesses

- Can the test help improve decision outcomes? Typically, the primary goal in human-AI decision-making is to achieve complementarity (e.g., as discussed in [1]), particularly by leveraging the complementary skills of human and AI. Because the test does not account for human and AI prediction accuracies, it is difficult to say whether performing such a test has any implications on complementarity. Another relevant cite is [2]. [1] Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. Bansal et al. CHI 2021. [2] A Unifying Framework for Combining Complementary Strengths of Humans and ML toward Better Predictive Decision-Making. Rastogi et al. EAAMO 2022. - Experimental validation: While it is great that the authors perform experiments on real-world data, it would be ideal to also verify the behavior of the test using synthetic where the differences between human and AI can be more carefully controlled to establish how sensitive the test is to these similarities and potentially the effect on the choice of L.

Questions

Major: - Could you address the questions raised in the weaknesses mentioned above? Minor: - Notation confusion: U is used to both denote private information in Section 2 and random variables and observations in Section 3 - What would the test return if the human is using information U that is *correlated* with X? Here, this may be a case where U is easier for a human to notice compared to X. - The table captions are not descriptive, which hinders readability. - Tone of contribution: the authors state that this test is a “necessary condition for achieving human-AI complementarity” without demonstrating results in terms of improving decision outcomes.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Clarification

Thank you for your response. I have re-read the Appendix, and the synthetic experiments there are indeed what I was looking for, so thank you. I wanted a little bit more clarification on weakness #1 (whether data points that are close in feature space but far in label space exist). You answered this a bit in the general rebuttal, but I was wondering if you clarify why this generalizes?