Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets

Summary

The paper introduces a notion of counterfactual harm for a classification setting in which a human is choosing among a subset of possible labels determined by an AI system, i.e., the automated system is used to "narrow down" the set of most likely labels for a sample. Under assumptions of counterfactual and interventional monotonicity, which imply that , the notion of harm can be identified pointwise or up to an interval. A procedure for controlling harm is introduced, based on conformal risk control.

Strengths

(S1) The described problem is important. Furthermore, not many works in this space exist, and the definition of harm is not universally agreed upon, as noted by the authors. Thus, this paper explores a valuable direction of research. (S2) Many of the concepts are introduced in a clean way, and the writing is good overall. (S3) The paper seems to combine many state-of-the-art ideas, including notions of counterfactual harm, identification under monotonicity assumptions from previous works, and conformal risk control.

Weaknesses

Major: (W1) The causal inference notation is not up to the expected standard. In Definition 1, which is key to the paper, the notation does not work. The paper is trying to describe the notion of counterfactual harm, while using only the $do()$ operator and conditioning. The do() operator and conditioning can only give quantities in the first two layers of the causal hierarchy. Therefore, the current definition, as written, is incorrect. The appearance of $\hat Y$ in multiple places with different meanings is concerning. Instead of this, the paper probably wants to use the subscript notation $\hat Y _{\lambda}$. My understanding is that the paper is trying to say $$ h_\lambda (x, y, \hat y) = E [ max (0, \mathbb{1} ( \hat Y = Y ) - \mathbb{1} (\hat Y _{\lambda} = Y) ) \mid X = x, \hat Y = \hat y, Y = y]. $$ (W2) The connection to Richens et. al [1] is mentioned in the introduction. It is clarified that the difference here is that "their definition requires one to deploy the system", while this is not the case for the definition proposed in this work. It is not clear from the text whether the current definition does not require deployment because of the definition itself, or the monotonicity assumptions that are invoked. The two definitions seem closely related, and given the comment in (W1), it would be very valuable to have these two definitions compared side-by-side, with a unified choice of corrected notation. Opening a small appendix for this would be very desirable. --- Minor comments / corrections: (W3) Relating to point (W1): in Pearl's notation, causal mechanisms are usually denoted by the assignment operator $\gets$, i.e., $X \gets f_X (U_X)$. This is useful to indicate that the mechanisms are not "equations that can be manipulated". (C1) Line 159: "if they has failed"

Questions

Please see (W1), (W2). (Q1) Lines 147-148 claim that the $\Lambda (\alpha)$ set cannot be found because counterfactual harm is a counterfactual quantity. This statement seems imprecise. For instance, the effect of treatment on the treated (ETT), is a counterfactual quantity that can be estimated with lack-of-confounding assumptions. Furthermore, any counterfactual quantity could be, in principle, bounded without additional assumptions. It seems more likely that the joint distribution of $\hat Y, \hat Y _{\lambda}$ is required to evaluate the expression (and this seems to be a consequence of the max() operator within the expectation), which is known hard problem, often solved with monotonicity assumptions. (Q2) When taking a step back, the question that comes to mind is: what is the end goal of the proposal? When minimizing counterfactual harm, it seems that "counterfactual benefit" is almost entirely ignored. Is this a reasonable notion to ignore? Furthermore, if the definition were to remove the max(), would this be equivalent to optimizing overall accuracy or something similar (note that removing the max solves the identifiability issue)? Finally, should there be an extension in which $\lambda = \lambda(x)$ is a function of $X = x$, so that harm is minimized in areas where it is largest? (Q3) Line 142: "Otherwise, the expert's prediction could not have become worse". The claim of the paper is about decision-making support systems, whereas this comment seems quite closely focused on accuracy. For instance, in the very example given in Figure 1: if someone has a common cold, mistaking this for a flu seems rather different than mistaking it for lung cancer (the consequences of these two classifications may be rather different)? Please comment on this. (Q4) Lines 127-130 are saying that an SCM can match the observational distribution. Can you comment on the significance of this observtion? Is this remark made to claim that the usage of SCMs does not limit the generality of the approach, due to their expressiveness? If so, perhaps it is worth adding such a sentence.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

Please check (W1), (W2), and (Q1), (Q3).

Summary

This paper addresses the issue with prediction methods that generate a set of potential labels, from which a human makes the final decision. The authors identify a harmful scenario where the model's intervention leads to incorrect predictions that a human would have otherwise made correctly. To model this, they use the Structural Causal Model (SCM) framework. The prediction set size is controlled by a parameter $\lambda$. For a given $\lambda$, they define counterfactual harm as the average number of instances where a human's unaided prediction would be correct, but becomes incorrect with the model's assistance. The main objective of the paper is to identify the range of $\lambda$ values that keep the counterfactual harm within a user-specified constant $\alpha$. To achieve this, the authors use conformal prediction and propose two training algorithms under two conditions: - Counterfactual monotonicity: At the instance level $(x, y, \hat{y})$, smaller and correct prediction sets lead to better human predictions. - Interventional monotonicity: At the sub-population level characterized by $(x, y)$, smaller and correct prediction sets lead to better human predictions *on average*. They conduct experiments on two image datasets where human predictions from prediction sets are available, demonstrating that their methods effectively control counterfactual harm.

Strengths

The problem formulation is interesting. The authors establish an objective that essentially tunes the threshold parameters used in Conformal Prediction, aiming to control the number of instances where the prediction sets impede the human from making the correct prediction. Solving the optimization problem requires access to counterfactual labels for human predictions across all possible prediction sets, which are naturally unavailable in observational datasets. To address this, the authors identify two weaker conditions, which are highly likely to hold in practice, to solve the problem. Finally, they apply the conformal prediction algorithm on a hold-out calibration dataset to identify the set of $\lambda$ values that bring the CF harm under control.

Weaknesses

Please refer the questions below.

Questions

**Equation 4:** In the definition of CF harm, why is the true label $Y$ a function of the predicted label $\hat{Y}$? Could you please elaborate on the notation? **Line 164:** Should $v$ not be abducted from the joint distribution of $(X, Y, \hat{Y})$ rather than just $X$? **Lines 127 to 130:** I do not understand the text in these lines. Are you stating that one can always find some distribution over the exogenous variables that aligns with the observational data, or are you claiming that the original SCM is identifiable? **Typo at Line 159:** has --> had **Proof for Proposition 1:** In the proof for Proposition 1, where have you utilized the CF monotonicity condition? The proof seems to hold for any arbitrary SCM $\mathcal{M}$. Am I missing something here? **On CF monotonicity:** I believe that under no intervention, a human predicts the label with $C_\lambda(x) = \mathcal{Y}$, i.e., the entire label set. If CF monotonicity holds, why wouldn't any conformal prediction algorithm that guarantees $1-\alpha$ coverage satisfy the CF harm requirement? **Examples for CF monotonicity violations** Can you provide examples from your considered datasets where CF monotonicity is violated. In practice, why would such violations happen? **Tradeoff between CF harm and conformal accuracy:** I believe accounting for CF harm may unnecessarily increase the prediction set size, beyond what vanilla conformal prediction would offer, for the same amount of coverage guarantees provided by both methods. Can the authors comment on this, at least empirically? If such a tradeoff exists, this approach would be interesting if one could find the subset of instances where harm occurs and explicitly find the prediction set for such instances. However, I am disappointed that the paper does not propose such approaches. **Conformal baselines** The authors should compare their algorithms with the state-of-the-art conformal prediction algorithms and evaluate how their methods perform in terms of (a) conformal accuracy, (b) prediction set size, and (c) CF harm. The current experimental section appears too weak. **Conformalizing $\hat{Y}$:** Why shouldn't this approach work? Suppose we apply conformal prediction to both the ground truth $Y$ and the expert prediction $\hat{Y}$, and then emit the prediction set as a union of the two prediction sets obtained from these conformalized models. How does your approach compare to such a proposal in terms of the accuracy $A(\lambda)$ and the prediction set size?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes, limitations are addressed.

Summary

This paper focuses on decision support designed to help humans in multi-class classification tasks via prediction sets. The paper defines the notion of counterfactual harm, namely when usage the system would lead the agent to a wrong prediction which would have been correct without the system’s ‘help’. The authors show that under some monotonicity assumptions such counterfactual harm is identifiable or partially identifiable. Against this theoretical backdrop, the paper proposes a method to find sets of interventions (namely thresholds for the prediction sets) which are guaranteed to have bounded harm. This approach is validated using experiments on human subject studies. These studies reveal a trade off between post-intervention accuracy and counterfactual harm.

Strengths

I am not an expert in prediction sets so I cannot fully evaluate how significant this contribution is, and was not able to fully check the proofs. I saw the following strengths: 1. The trade off between accuracy and counterfactual harm is very interesting, and the possibility to control the desired amount of counterfactual harm seems very important. 2. The theoretical results seem solid and are well rounded out by the corroborating experiments. 3. Despite the technicality of the paper, I could follow the argument and appreciate the message. The text is well written.

Weaknesses

1. The notation little y-hat and capital Y-hat in section 3 is somewhat confusing. Line 123 implicitly suggests the latter is the random variable generated by the user predicting with the decision support. Line 142 however states that little y-hat is the prediction the experts would make on their own. I would have expected this to be just Y-hat but with lambda = 1, as stated elsewhere in the text? 2. In the first experiment, when w is different from 110, the data contains predictions made by humans on their own, and the accuracy of the humans using prediction sets is only estimated. How good are these estimates? We only get to see a comparison with the real value in Figure 3 for w=110, and it is clear that these estimates decisively overshoot. 3. The results show that the sets of lambdas do not contain all the harm-controlling values - as the authors remark - and especially seem to contain less of the ‘good’ values, meaning those that (at the cost of some counterfactual harm within the accepted bound) achieve higher accuracy. This seems to be a serious drawback. 4. Lambda is used in line 105 but only introduced in line 109. 5. Theorem 1 and 2 could use a bit more intuition and/or explanation. Typos: line 137: use -> used

Questions

Beside remarks on the weaknesses mentioned above, I would be interested in answers to the following questions: 1. In Figure 1 you show the average accuracy of humans estimated by MNLs when w=110, but in the text you report that the second dataset contains the predictions of humans with prediction sets at all thresholds for that value of w, which is then shown in Figure 2. Why not start with the actual accuracy then, instead of the estimates? It seems to me that Figure 2 has the core findings, while Figure 1 is more of a sensitivity analysis on the fact that this phenomenon is consistent across model architectures (under the assumptions that the MNL estimates are qualitatively reliable). 2. Three of the models tested have worse average performance than the human experts. What is the point of testing decision support with such models? I suppose we would never implement a decision support that is worse than humans alone (unless perhaps we have strong evidence it is better than humans on a subgroup of data points, but this is not the case here), since it can stir the humans with poor predictions. I can see this could be interesting since you are studying counterfactual harm, but still it is an unrealistic scenario so I would not present results for three such models.

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

The paper seems to mention and discuss all the relevant assumptions and limitations.

Summary

This paper relates to decision support systems based on prediction sets. It introduces a framework using conformal risk control to design prediction sets that balance accuracy with minimizing potential harm (understood as a human who has succeeded at predicting the ground-truth label of an instance on their own failing had they used the DS system), validated with human prediction data from empirical lab studies.

Strengths

- Originality. The paper establishes formal definitions/characterizations and a theoretical foundation. The paper formally characterizes the notion of "counterfactual harm" and introduces the counterfactual monotonicity assumptions and weaker interventional monotonicity assumptions to make counterfactual harm identifiable and partially identifiable, respectively. This conceptual contribution improves the understanding of the risks associated with decision support systems based on prediction sets. - Significance. The authors demonstrated that the idea of "counterfactual harm" can be made practically useful through conformal risk control (section 5), and they performed lab experiments to validate their theoretical bounds. The result is practical: Cor 1 and 2 ensure that these systems cause less counterfactual harm on average than a user-specified threshold.

Weaknesses

1. The main weakness is the usability and motivation of this particular formulation. The paper proposes decision support systems based on prediction sets, which might not always be feasible or realistic, especially in complex domains like healthcare. Most current systems are based on risk prediction e.g. disease risk, drop out risk, for binary outcomes. I don't see any good examples of problems where prediction sets are the right approach, and this paper also does not do this work of illustrating or justifying this formulation using real world case studies. 2. Discussing the Limitations of Predictive Accuracy. While the paper discusses improving accuracy through prediction sets, there may be fundamental limitations in predicting outcomes accurately in the first place. Have the authors considered recent research or critiques regarding the actionability of predictive models? For instance, how does the work align with concerns raised by Wang et al (2024) "Against Predictive Optimization" and Liu et al. (2024) "On the actionability of..."?

Questions

1. Are there practical examples of decision support systems based on prediction sets being successfully implemented in healthcare or other social domains? How do these systems handle the inherent complexity and variability of real-world decision-making scenarios, such as patient care pathways or policy decisions? 2. Have the authors considered recent research or critiques regarding the actionability of predictive models? For instance, how does the work align with concerns raised by Wang et al (2024) "Against Predictive Optimization" and Liu et al. (2024) "On the actionability of..."?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes.

Reaction to rebuttal

Thanks to the authors for their rebuttal. I believe my points have been addressed sufficiently, and I appreciate the additional work in showing the role of the validation set. I suggest to work their responses of their points 3 and 7 into the main text. Concerning the point of another reviewer on the discussion of the relationship between prediction performance and harm in decision support systems, alongside Wang 2024 and Liu 2024 it would be appropriate to discuss van Amsterdam 2023 at al "When accurate prediction models yield harmful self-fulfilling prophecies", which contains formal results on the topic.

Follow up questions.

Thanks for the rebuttal. I have carefully read your responses. Please see below for the follow up questions. 1. I am particularly interested in understanding the notation $Y=y(\hat{Y})$. Why is the true label expressed as a function of the predicted label? 2. While I understand that $\hat{Y}$ is not fixed. but why is $Y$ not fixed? 3. If conformal predictors with $1-\alpha$ coverage guarantee gives us an $\alpha$ guarantee on CF harm already, wouldn't conformal predictors directly serve as baselines for your approach? In that case, comparing existing SOTA conformal predictors with your method in terms of prediction set size becomes crucial. I’m disappointed that such comparisons were not included in the rebuttal, despite being explicitly asked for. 3. I understand that in observational datasets, it’s not possible to detect CF monotonicity violations. However, in your benchmark datasets, where predictions were modeled based on prediction sets using MNL, did you encounter instances of such violations? Identifying such examples would strongly support the practical motivation for your problem setting. Currently, I am struggling to convince myself that CF harm occurs in practice. 4. I am unclear about the results provided in the PDF. The experiment I requested was as follows: Consider the SOTA CP algorithms in the literature and evaluate the following metrics—CF harm, conformal accuracy, and prediction set size. How does your conformal approach, which explicitly addresses CF harm through the formula $1 - m_y (x)$, compare to other SOTA methods that do not? I’m particularly interested in understanding the gap your approach fills empirically in the existing SOTA methods.