Quantifying Aleatoric Uncertainty of the Treatment Effect: A Novel Orthogonal Learner

Summary

The paper provides the orthogonal estimator for the distributional treatment effect (the conditional CDF of $Y[1]-Y[0]$).

Strengths

This paper is technically strong, demonstrating a high level of mathematic rigor and diligence. It presents an in-depth description of the proposed estimator. I think the proposed estimator is useful in practice.

Weaknesses

Despite the strong technical details, the paper is poorly written in overall. __1. Weak motivation__ A current shape of Introduction is weakly motivated. Firstly, the way that the paper motivates the problem is misleading: > Methods for quantifying the aleatoric uncertainty of the treatment effect have gained surprisingly little attention in the causal machine learning community. This sentence is incorrect, since there are literatures on quantile regression and semiparametric density estimation, as reviewed in Section 2. More importantly, the introduction doesn't provide the motivation of the problem against the following question: _why a community need the proposed estimator, given that the quantile estimator can capture the distributional treatment effect_. __2. Difficult to understand due to insufficient information__ Another issue that the paper has (especially in Introduction) is its lack of back-ground information that readers may need to comprehend. Specifically, in Introduction, the paper doesn’t provide any definition or clue what CDTE is. I understand that the CDTE is defined in the caption of Figure 1 as $P(Y[1] - Y[0] \leq \delta \mid x)$. However, $Y[a]$ is undefined, and this key quantity should be in the main body of the text. Also, even if Figure 1 aims to provide a whole summary of the paper, authors at Introduction have insufficient knowledge to comprehend it. In other words, Figure 1 is too detailed to be presented in Introduction section. Since Figure 1 can only be understood by those who entirely digested the paper from the beginning to the end, Introduction section is not the right position where the Figure 1 is located. I understand the goal of Figure 1, but it doesn't achieve the goal because of insufficient background information. The same issue happens to Figure 2 and Table 1. For example, in Table 1, technical terms like AIPTW, hold-out residual, optimization assumptions are undefined, so it's hard to appreciate the contribution of this paper. __3. Fuzzy description on contribution__ First, the terms like "Aleatoric uncertainty" and "distributional treatment effect" are used for denoting the same target quantity. Given that the "distributional treatment effect" is clearly describing the problem, I don't see why the authors want to use "Aleatoric uncertainty" as a title and employ these two words to denote the same estimand. Second, the contribution is wrongly described. Consider this sentence: > AU-learner solves all of the above-mentioned challenges 1 – 3 . The AU-learner doesn't address Challenge 1 and 2. Challenge 1 means that the distributional treatment effect is not identifiable. The Makarov's bound, _NOT_ AU-learner, is employed to address Challenge 1. Challenge 2 is actually the same as Challenge 1, since it means that there are no known nuisances-based functional for the distributional treatment effect. Again, the Makarov's bound is used to address Challenge 2. AU-learners are representing the approximated quantity of the target estimand in terms of nuisance functionals. Finally, third contribution "flexible deep learning instantiation of our AU-learner" is scarcely described only in Section 5. The description needs to be much improved. __4. Little focus on the real contribution__ The real contribution of this paper, compared to the existing works in Table 1, is to provide the doubly robust conditional distributional treatment effect for the bounds of the conditional CDF of treatment effects. However, little focus and efforts have been made for this contribution. For example, if developing a doubly robust estimator is a contribution, then corresponding results such as detailed error analysis, a closed form of estimators, a detailed recipe of the proposed estimator for the specified working model, how to minimize the losses in Equations (8,9), a simple example, and assumptions should be described.

Questions

1. Are rate-doubly-robustness and Neyman orthogonality violated when the scaling hyperpameters are not $1$? 2. Is the Makarov bound sharp? 3. What are the practical examples of the distributional treatment effect? 4. In line 178, is this phenomenon officially termed the selection bias?

Rating

3

Confidence

5

Soundness

4

Presentation

1

Contribution

2

Limitations

The paper is limited to the setting where the ignitability holds.

Summary

The authors propose a partial identification of quantiles of the individual treatment effect, which are not point-identifiable in general. The authors justifiably argue that characterizing the distribution of individual treatment effects gives a better idea of the aleatoric uncertainty in a causal-inference problem. The proposed method is doubly robust, works with heterogeneous treatment effects, and requires minimal additional assumptions about the data.

Strengths

The problem of aleatoric uncertainty in causal effects is clearly important, the contribution is significant, and the presentation is solid and concise. * Explanations of aleatoric versus empirical uncertainty are clear. * The numerous diagrams are informative. * Algorithm 1 is also easy to understand and succinct.

Weaknesses

* The benefit and/or novelty of the CA-learner is questionable (also reflected in the results Table 2) and I wonder if its exposition is taking up valuable space. It is an insightful point on lines 182--185 that learning Makarov bounds could benefit from inductive biases of lower heterogeneity than the conditional CDFs. However, it is unclear if the CA-learner loss really incorporates that inductive bias and if so, how much. * The conclusion is a bit grand. Arguably, previous papers like those highlighted in Table 1 have proposed robust methods for quantifying a version of aleatoric uncertainty of causal effects.

Questions

My main question has to do with recent related work. In particular, Ji et al. [52] appear to solve a similar problem, and the quick dismissal of that approach in this paper because they "made special optimization assumptions" needs further discussion. It would be helpful to spell out what these assumptions are in a concrete sense. In doing so, the authors could also discuss whether these two approaches have any fundamental commonalities, or if the partial identifications are expected to be materially different. Finally, it would be nice to see [52] appear as a baseline in the empirical evaluations, although I understand this could be difficult to implement, especially since [52] is relatively recent.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

N/A

Summary

In this paper, the authors propose a method to quantify the aleatoric uncertainty of the treatment effect. For this, authors estimated Makarov bounds on the CDF and quantiles of the CDTE, and then showed, how one can build a learner, which has properties of Neyman-orthogonality and double robustness. The authors proved the abovementioned theoretical properties of the resulting learner and demonstrated the usefulness of the proposed approach in a series of experiments on synthetic and real-world data.

Strengths

I think the paper has the following strengths: It - Addresses an important problem in the field of treatment effect estimation; - Proposed a theoretically grounded method, that has useful theoretical properties, to quantify the aleatoric uncertainty of the treatment effect; - Published code;

Weaknesses

I don't have critical concerns about the paper, but I believe the following can improve the paper: - I find the paper logically well-structured, but at the same time challenging to read, as it is too concentrated with technical details. I would suggest authors reconsider the narrative, concentrating more on the conceptual ideas in the main part, and moving technical details to the Supplementary; - The field of treatment effect estimation is quite special and narrow (from my point of view) among the machine learning community, and it is worth adding some clarifications to the terms and notation used. For example, what is $Y$? It is just said that it is a continuous outcome, without any intuitions of what it could be. - I feel that the paper missing the discussion of the alternatives of the proposed approach. For example, why specifically Makarov bounds were chosen, but not other possible alternatives? - In Table 2, it is worth explicitly writing that CNF corresponds to the Plug-in learner (as it was called throughout the paper). I would suggest Plug-in-CNF (like written for IPTW-CNF). - In Line 268 there is a minor typo: Should be Eq.13 and Eq. 14.

Questions

- What are the alternatives for Makarov's bounds? - Are such options like conformal predictions or plain confidence intervals somehow useful to estimate AU in the context of treatment effect? - Compared to the vanilla Plug-in estimator (say Plug-in-CNF), what is the additional computational overhead of AU-learner? - In lines 212-216, it is mentioned that CA-learned still has shortcomings (a) (and a new shortcoming (c)). In practice, how severe is the shortcoming (a) (selection bias) compared to AU-learner? Can you provide a toy example?

Rating

7

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

One of the limitations, not mentioned by authors, is that the approach requires (conditional) Normalizing Flows, and hence might not work well in the high dimensional scenario.

Summary

The authors introduce AU-learner, a method to estimate the conditional distribution of treatment effects (CDTE) and hence capture the variability in the treatment effect. They use Makarov bounds for partial identification and use conditional normalizing flows for estimation. Further, they show that AU-learner satisfies Neyman-orthogonality and double robustness.

Strengths

(S1) The paper is well written and the authors provide a good experimental validation of the proposed method. (S2) The authors are the first to propose a doubly robust estimator for the Makarov bounds on the distributional treatment effect.

Weaknesses

(W1) I would have liked a more comprehensive discussion of how the proposed bounds compare against existing ones in terms of sharpness. For instance, it is not clear to me if the bounds you get are sharper than [58] in the binary outcomes setting. (W2) I think it would be useful to compare the proposed methods against the non-doubly robust estimators. In practical settings both outcome and propensity models are likely misspecified, hence the main advantage of doubly robust estimators are the faster theoretical rates. In this setting (with both nuisances misspecified), it is not clear if the doubly robust version of the bounds is always better in practice and some more experiments would be nice to explore this. (Minor) In the extended related work on partial identification and sensitivity models (line 658) you could also mention more recent approaches that incorporate RCT data together with proxy and instrumental variables, e.g. the works of [1] and [2]. [58] Nathan Kallus. “What’s the harm? Sharp bounds on the fraction negatively affected by treatment”. In: Advances in Neural Information Processing Systems. 2022. [1] Falsification of Internal and External Validity in Observational Studies via Conditional Moment Restrictions. Hussain et al. AISTATS 2023. [2] Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. De Bartolomeis et al. AISTATS 2024.

Questions

(Q1) For a binary outcome, could you comment on how your bounds relate to Kallus [58]? Why don't you consider them as a baseline (say for estimating bounds on the quantiles of the treatment effect distribution)?

Rating

7

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

The authors study the distribution over the individual treatment effect for binary treatments, continuous outcomes, and observed, potentially high-dimensional confounders. The authors build up on prior work on Makarov bounds, to develop a new method to lower/upper-bound the conditional CDF and the quantile function of the treatment effect. They develop a doubly robust learner and perform a theoretical and empirical evaluation. In a series of experiments, They compare several instances of their method with a baseline based on kernel density estimation. While the bound-based methods clearly outperform the baseline, they yield rather similar results.

Strengths

The authors clearly state the problem and derive a viable solution. They first argue that learning estimators of the CDFs of the two potential outcomes, and plugging them into the bound computation does not yield optimal bounds. Instead, they then propose to learn the outcome CDFs which directly target the Makarov bounds. Third, they augment this loss with a term accounting for selection bias, which yields a doubly robust learner. Finally, the authors derive an implementation based on neural normalizing flows. The authors present a great set of supplementary material including additional discussion of related work, additional experiments, and implementation details. The material might be sufficient for a journal publication.

Weaknesses

The plugin estimator based on a conditional normalizing flow seems to perform rather well in the experiments. The CA learner w/o bias correction seems to perform as good as the AU learner (slightly worse in Table 2, slightly better in Table 4 in the appendix). The added complexity of the AU learner, in particular, the one-step bias correction which comes with an additional tuning parameter, renders the practical value questionable. It seems that a lot of work went into the AU learner; but it might still be worth focusing the presentation more on the simpler CA learner (e.g., discussing the role of g in the main doc).

Questions

I do not fully understand the role of the working model g \in G in the CA learner. The Makarov bound is a CDF and G may include all possible CDFs; so if G is rich enough, the CA learner may return the same CDF estimate as the plugin estimator. If we restrict G, why does this cause tighter bounds?

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

NeurIPS Paper Checklist is provided; no concerns.

Response

Overall, I am embarrassed by this response. The response is written as though the authors' categorization—(1) Aleatoric uncertainty of potential outcomes, (2) Distributional treatment effects, and (3) Aleatoric uncertainty of the treatment effect—has been clearly established in the paper, and my questions/concerns arise from a lack of understanding of this framework (e.g., " the distributional treatment effects are not the target of our paper,", where the term "distribution treatment effect" is just defined by this response, not the paper). However, as stated in the authors' response, this categorization is not mentioned in the paper, and no clues are provided regarding this categorization. Furthermore, this categorization seems counterintuitive -- does it really make sense to state that "distributional treatment effects" and "distribution of treatment effects" are _very different_? I do understand that they are different, and that the distribution of the treatment effect, P(Y[1] - Y[0]), is not pointwise identifiable due to the fundamental problem of causal inference. However, the categorization seems to come out of nowhere and requires justification. I do understand which research streams this work is located in. Using your framework, my questions and concerns (Q3 and W1) were about what practically interesting scenarios "Aleatoric uncertainty of the treatment effect" can capture that other research streams cannot. I don't believe your response has fully addressed this question/concern yet. I adjusted my score based on this response.