Response to Reviewer T6Zv
Dear Reviewer T6Zv,
Thank you for sharing this reference and giving us a chance to discuss it with you. We strongly agree with you that the kNN regressor can be a consistent estimator of target functions in some machine learning contexts. **However, such consistency property may not necessarily hold when fitting potential outcome surfaces due to the distribution shift between factual and counterfactual distributions.**
The paper you mentioned and Chapters 6 and 11 of [1] prove the consistency of the kNN regression estimator under certain conditions, such as $k/n \rightarrow 0$. However, note that **the consistency holds with the assumption that data are independent and identically distributed (i.i.d.)**, as discussed in the Introduction of [1]. Though the i.i.d. assumption is common in most machine learning scenarios, it doesn't necessarily hold when estimating potential outcome surfaces $\mu_t$ for $t \in $ {0, 1}. Please allow us to explain in detail below.
In the Rubin Causal Model framework, observational (factual) data are i.i.d. tuples $(x_i,t_i,y_i)_{i=1}^{n}$, following the factual distribution $P^{F}:=P(X,T,Y)$. For each pair $(x_i,t_i)$, there also exists an unobserved counterfactual outcome $y_i^{CF}$. The unobserved (counterfactual) data are i.i.d. tuples $(x_i,t_i,y_i^{CF}) _{i=1}^{n}$, following the counterfactual distribution $P^{CF}:=P(X,T,Y^{CF})$. As explained in Section 3.1 and GR1 of our top-page rebuttal, **$P^{F}$ is not identical to $P^{CF}$ in general**. Consequently, a model trained on $P^{F}$ may not predict well on $P^{CF}$.
Let's consider estimating $\mu_0(x)$ for further explanations. To infer the potential outcome $Y^0$ for treated ($T=1$) samples, the kNN regressor first approximates $\hat{\mu}_ {0}(X)$ using controlled samples $(X_i,Y^0_i) _ {i=1}^{n_ {control} }$, then uses $\hat{\mu} _ {0}(X)$ to predict $Y^0$ for treated samples, with the predicted samples being $(X_i, \hat{Y}^0_i) _ {i=1}^{n_ {treat} }$. However, $\hat{\mu} _ {0}(X)$ does not necessarily generalize well on $Y^0$ predictions for treated samples, because the source training data (controlled samples) do not have the same distribution as the target prediction data (treated samples), meaning that there exists a distribution shift problem $P(X,Y^0|T=0) \neq P(X,Y^0|T=1)$. Such a distribution shift (discussed in Section 3.1 and GR1 of our top-page rebuttal) renders kNN-based learner $\hat{\mu}_{0}(X)$ to be a consistent estimator of $\mu_0(x)$.
Therefore, the above analysis demonstrates that, if kNN-based learner $\hat{\mu}_ {0}(X)$ would be a consistent estimator of $\mu_ {0}(X)$, one of the following two conditions must be met:
* The factual distribution for controlled samples, $P(X_i,Y^0_i|T_i=0)$, should be identical to the counterfactual distribution for treated samples, $P(X_i,Y^0_i|T_i=1)$. This necessitates the absence of distribution shift between factual and counterfactual distributions, which in turn requires two key conditions: the unconfoundedness and access to infinite data from Randomized Controlled Trials (RCTs).
* The training samples should be $(X_i,Y^0_i)$ (including the whole samples) instead of $(X_i,Y^0_i)|T_i=0$ (only including controlled samples). This requires counterfactual knowledge, because the potential outcome $Y^0_i$ remains unknown for treated ($T_i=1$) data.
We hope this clarifies our perspective on the consistency of kNN in causal inference contexts. Note that the above analysis and conditions for consistency property are not exclusive to kNN but also extend to other machine learning methods employed in estimating potential outcomes or CATE. We will further incorporate the above discussions in Appendix, as they further underscore the importance and significance of our method in CATE estimator selection. The DRM method offers distinct advantages compared with previous selection metrics: it does not require counterfactual data and is capable of selecting estimators that are robust to the distribution shift between factual and counterfactual samples.
Thank you again for your active engagement and effort in helping us improve the paper! We are very welcome any further thoughts or questions you may have.
Reference: [1] Devroye, Luc, et al. A Probabilistic Theory of Pattern Recognition. Stochastic Modelling and Applied Probability.
Best regards,
Authors 4584