Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Model

Summary

This paper studies the partial identification of counterfactual probabilities from observational data. More specifically, the authors consider the standard bandit model containing a treatment A and a reward Y; exogenous variables U exist affecting both A and Y. The treatment A is a binary variable; while the reward Y could take any arbitrary real value bounded in an interval [l, u]. The goal is to evaluate the counterfactual probability P(y_a | a’, y’) from the observational data drawn from P(Y, A). The author first shows that this learning setting is ill-posed without any additional assumptions. That is, the learner could only obtain non-informative bound [l, u] from the observational data. This observation suggests that one should explore additional parametric assumptions to obtain informative bounds. More specifically, the authors propose a sensitivity model called Curvature Sensitivity Model. They further propose an implementation of our Curvature Sensitivity Model in the form of a novel deep generative model, called Augmented PseudoInvertible Decoder.

Strengths

- This paper is well written and well organized. The authors provide extensive examples and graphical illustrations. This is much appreciated. - Bounding counterfactual probabilities over a continuous outcome from the observational data is a challenging problem in causal inference. The authors study partial identification in multi-armed bandit models, which are widely applied in reinforcement learning literature. The target query P(y_a | a’, y’) is called the expected counterfactual outcome of un]treated ECOU; its applications include fairness analysis in machine learning (Kushner et al., 2018). Since ECOU is generally non-identifiable and existing bounding algorithms focus on discrete domains, this paper could have an impact on the causal inference community. - The non-informative bounding result in Theorem 1 is interesting. As far as I am aware, this result is novel. The authors also provide a novel sensitivity model of Curvature Sensitivity Model (CSM), which permits one to obtain informative bounds by bounding the curvature of level sets of the functions. They further show that CSMs generalize existing parametric assumptions for point counterfactual identification methods when the bound of the curvature is set to zero.

Weaknesses

- This paper studies the bounding of a specific counterfactual probability, called ECOU in canonical bandit models. However, some statements in the abstract sound more general than it is. For instance, the paper states "We prove that, in general, the ignorance interval of the counterfactual queries has non-informative bounds, already when functions of structural causal models are continuously differentiable." Based on my reading, the non-informative bounds in Theorem 1 only apply to bounding ECOU in bandit models. However, if we consider bounding a causal effect P(y_a) in the bandit model, Manski's bound (1990) has shown to be informative and applicable to continuous reward signals. - The simulation plots could be improved. For instance, it is unclear what the ground truth ECOUs are in Figure 7. Without that, it is hard to evaluate the validity of the proposed bounding strategy.

Questions

- What is the actual ECOU in Figure 7?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Assumptions behind the proposed algorithms are explicitly stated. This paper is theoretical and its long-term societal impact is not immediate to see.

Summary

The authors study the problem of counterfactual identification of continuous outcome variable with a specific causal graph. They show that the expected counterfactual outcome of (un)treated has non-informative bound when the function class is arbitrary smooth. Then, they introduce a new model called CSM that give informative bounds. Finally, they propose a deep generative based method for partial counterfactual inference in CSM.

Strengths

They study an important problem. The paper is well-written and the Appendix is educational. The analysis is quite interesting as it combines differntial geometry and causal inference. This opens a new door to improve the current results in causal inference and likely to develop a general framework for counterfactual analysis and more. The presented examples in both the main text and the appendix are informative. Theoretical results are quite interesting. In particular, Theorem 1 and 2.

Weaknesses

The main question is about the generalizability of this method to more complex causal grpahs. It is not hard to see that as the causal diagram becomes a bit more complex, then applying similar analysis might be quite challenging. It is a bit confusing that CSMs with arbitrary $\kappa$ lead to informative bounds, i.e., $l(\kappa,d)>l_a$ while, for instance, $\mathcal{B}(C^2,d)$ is not informative. Because, as it is also mentioned in the text, as $\kappa\rightarrow\infty$, CSM$(\kappa)\rightarrow \mathcal{B}(C^2,d)$. What is the interpretaion of this? The proof of Theorem 2 is of existance type that is the form of the informative bounds and how they depend on $\kappa$ and $d$ are not given. But is it still possible to say something about their dependencies, e.g., $l(\kappa,d)$ is decreasing with respect to $\kappa$? Although, the augmented Pseudo-Invertible decoder method is a fine application of deep generative methods but it is not clear how sensitive it is with respect to the model hyperparametersa and sample size.

Questions

See above

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors addressed the limitations.

Summary

This paper studies the partial identifiability of counterfactual queries and proposes an interesting proof of general non-identifiability resulting in non-informative bounds. The authors build upon the intuition used for this proof to propose a new class of structural causal model using principal curvatures. Under this assumption, the authors can achieve informative partial counterfactual identification. The authors then propose an architecture that leverages this assumption.

Strengths

This paper proposes a systematic and rigorous study of the partial counterfactual estimation problem. This is exemplified by the breadth and depth of literature search and by the organization of the related works. While general non-identifiabilty of counterfactuals is known, theorem 1 shows that constraints on the smoothness of repsonse function does not improve the identifiability, which I think is novel. The proposed curvature sensitivity model, motivated by Theorem 1, is shown to benefit from the identifiability properties which are theoretically demonstrated. This causal model structure seems to be fairly general and appears to generalize concepts such as non-intersection of response surfaces (which is pointed at in the experiments). The authors further show that the theoretical properties of the CSM can be leveraged empirically be proposing a new architecture.

Weaknesses

While there is significant effort in bringing intuition regarding how to construct counterfactuals (bending of the counterfactual level sets), I believe more intuition could be given regarding the general applicability of the CSM in practice. It is unclear when it seems reasonable to assume this model would be valid, and hence the partial counterfactual estimation correct. I appreciate that this is a general problem in counterfactual estimation but more insight into the applicability would be welcome. While there is a strong theoretical work behind this paper, it is more frugal in terms of experiments. I would encourage the authors to provide more evaluations of their approach, with different types of repsonse surfaces and noise levels. The design of more experiments could potentially lead to more intuition in the assumptions of the method.

Questions

Could the author motivate the CSM assumption from a more practical perspective ? Extending the experiments section would help strenghten the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Limitations and assumptions are clearly stated.

Summary

This paper presents a method for obtaining bounds on continuous counterfactual outcomes. In general, level-3 counterfactual outcomes (what would have happened under action $a'$, given knowledge of what did happen under action $a$) are not identifiable from either interventional or observational data. Prior work makes structural assumptions on the underlying structural causal model (e.g., monotonicity with respect to a single unobserved noise variable) to obtain identification, while this work instead makes an assumption on the principal curvature of level sets with respect to two unobserved noise variables. A computational approach is provided to obtain bounds based on this principal curvature assumption, and is demonstrated on two simple synthetic cases.

Strengths

In terms of originality and clarity, this paper presents a novel approach to obtaining bounds on continuous counterfactual outcomes, and the contribution is clearly placed in the context of related work. Having seen a few papers on the general topic of partial identification, I found Section A (Extended Related Work) in the Appendix to be a well-written, well-researched, and generally helpful overview of how the current work differs from prior approaches, primarily in the problem considered. Moreover, in terms of clarity and quality, I found Example 1 (Figure 3) to be a very helpful example of counterfactual non-identifiability, and a good explanation of the intuition for how "bending" the counterfactual level sets around the factual level set leads to a different distribution of counterfactual outcomes.

Weaknesses

## Summary I see two major weaknesses in this paper, and look forward to discussion with the authors during the response period if I've misunderstood anything below. Both weaknesses relate to the significance of the approach. 1. How should domain experts choose the parameter $\kappa$? Choosing the right bound on the principal curvature of level sets doesn't seem like something a domain expert would plausibly be able to do. 2. Even if we knew the right $\kappa$, can we guarantee that the computational approach will give bounds that are at least conservative (e.g., outer bounds on the true bounds implied by $\kappa$) if not tight? Overall, my current score incorporates my view that these two weaknesses are genuinely challenging problems, so I am somewhat sympathetic to the paper in its current state, but I would at least like to see more discussion of these challenges in the main paper. I'll discuss these weaknesses in more detail below. I am basing my score primarily on these points, though I also had an important clarifying question regarding the ability of the approach to "cover the entire identifiability spectrum". ## Details of major concerns (1) **Domain Knowledge for $\kappa$?** One of the appealing properties of the given approach is that it can interpolate between the extremes of identifiability (though see my clarifying question (3) below). However, there are many trivial ways to interpolate between e.g., point-identification and the non-informative bounds (e.g., by simply choosing a point between them). The value of sensitivity analysis approaches (in my view) lies in the ability to translate domain knowledge into informative bounds. In other areas of sensitivity analysis (for e.g., interventional queries in the presence of unobserved confounding) that rely on domain knowledge to derive bounds, there is an emphasis on creating additional tools to help end-users calibrate their choices of these hyperparameters by e.g., benchmarking (see Figure 2 of Cinelli and Hazlett 2019, full citation below). Informally, this type of benchmarking allows one to say "an unobserved confounder would have to be 3x as strong as the variable 'age' in order to change our conclusions", a statement that is a bit easier for domain experts to engage with versus reasoning about the value of an abstract hyperparameter. I'm aware that this example deals with a different type of causal query (i.e., counterfactuals vs interventional queries), and that the lack of identifiability in the current paper is fundamentally different from unobserved confounding - my point is that for any partial identification approach that derives informative bounds via "domain knowledge", like the current approach, a bit more effort is required to make it plausible for domain experts to convert their domain knowledge into the required (abstract) hyperparameters, like bounding the principal curvature of counterfactual level sets. I would be interested in any comments from the authors on this point. Citation: Carlos Cinelli , Chad Hazlett, Making Sense of Sensitivity: Extending Omitted Variable Bias, Journal of the Royal Statistical Society Series B: Statistical Methodology, Volume 82, Issue 1, February 2020, Pages 39–67, https://doi.org/10.1111/rssb.12348 (2) **Corresponding between computational bounds and theoretical bounds**: Supposing we know the right value of $\kappa$, Definition 2 defines an upper and lower bound on the quantity of interest (e.g., the ECOU), with the additional constraint (e.g., in Theorem 2) that all SCMs satisfy Assumption $\kappa$ and are in the class $\mathcal{B}(C^2, d)$. For the practical algorithm, we use $d = 2$. In an ideal world, the computational method (i.e., the Augmented Pseudo-Invertible Decoder in Section 5) would take a given value of $\kappa$ and return the bounds as defined in Definition 2. However, somewhat understandably, returning the exact bounds is not necessarily feasible. Instead, we are left with heuristics - we optimize an objective over a certain set of SCMs where one of the four terms relates to maximizing / minimizing the counterfactual outcome, and is controlled by the hyperparameter $\lambda_Q$. It was not clear to me how this term trades off with the other terms, e.g., the extent to which we can view the optimization problem as choosing a fixed constraint on $\kappa$ and then maximizing / minimizing the counterfactual subject to this constraint among all observationally-equivalent models. **The use of practical heuristics would be less concerning if this approach resulted in less informative bounds than those in Definition 2, but as I understand it, it may result in *more* informative bounds.** To clarify, this point is briefly discussed in Appendix E (discussion of extensions and limitations) under "Tight bounds". However, my concern is not that the bounds are insufficiently tight (e.g., too wide and could be narrowed), but that they might be *too narrow* (e.g., the theoretical bounds are actually wider than the bounds produced by the algorithm). I believe there is some evidence to suggest that these heuristics change the interpretation of the results. For instance, I would expect that as $\lambda_{\kappa}$ changes, the bound should monotonically increase/decrease. However, looking closely at Figure 7 suggests that in practice there is some crossover (see e.g., $y' = -0.5$, where the lower bound for $\lambda_{\kappa} = 1$ goes above the line for $\lambda_{\kappa} = 10$). I believe my observation below relating to Figure 13 in the appendix (where $\lambda_{\kappa} = 0$ still gives informative bounds) is also related to this point. Overall, it would be good to see a comparison, perhaps for a very simple example, between the ground-truth bounds in Definition 2 and the bounds returned by the approach. (3) **Clarification regarding non-informative bounds with $\kappa=\infty$**: It is stated in Corollary 4 that by setting $d = 2$ and varying $\kappa$, we can "cover the entire identifiability spectrum". I took this statement to mean that if e.g., there is no restriction on $\kappa$, then the resulting bounds should be non-informative (i.e., equal to $[l_a, u_a]$), and that for $\kappa = 0$, all values of $d$ yield the same point-identified solution. I'm deriving my understanding primarily from the explanation given in the introduction (see lines 72-73). Is my understanding correct? If my understanding is correct, then what is going on in Figure 13 in the appendix? It appears that setting $\lambda_{\kappa} = 0$ (corresponding to no constraints on $\kappa$) does not result in non-informative bounds. Is this due to the (perhaps necessary) mismatch between the computational approach and perfectly solving the optimization in Definition 2? ## Minor Points As a minor point, it may be worth clarifying where we are interested in taking a maximum/minimum (e.g., over observationally-equivalent SCMs) and where we are interested in taking an average (e.g., over the counterfactual posterior for a single SCM). E.g., when I first read the procedure for abduction on lines 341-344, I was concerned that the approach draws samples from the pre-image, as opposed to looking for $U_1, U_2$ which yield the maximum/minimum counterfactual outcomes, before I realized that this approach makes sense for a fixed SCM to compute a counterfactual expectation like ECOU / ECOT. I also found Figure 6 difficult to follow, especially the bi-directional arrows for "connections".

Questions

1. How would you go about helping domain experts choose an appropriate value of $\kappa$ or $\lambda_{\kappa}$ in practice? 2. Is there any guarantee that the bounds derived from this approach will be less informative than the theoretical bounds? Or, should we generally expect the bounds to be more informative, since we will inevitably fail to find the SCMs that yield the true maximum / minimum for a given $\kappa$? 3. Related to the previous question, is my understanding of "cover the entire identifiability spectrum" correct in Corollary 4? If so, it may be worth revising the theoretical statement to make this clear. If my understanding is correct, then what is going on in Figure 13, i.e., why does setting $\lambda_{\kappa} = 0$ still yield informative bounds?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

In the main paper, I would have liked to see more discussion of the limitations outlined in the "weaknesses" section above.