On related works and variance underestimation
Thanks for your reply!
- For W2, could the authors provide more discussion one when to consider this class of methods over the others added to the related works discussion?
Response:
_Shapley value-based methods_: Shapley value-based measures can be conceptualized as a weighted average of LOCO measures within our framework, accounting for the inclusion of a feature (or a feature group) of interest across all possible subsets (Williamson and Feng, 2020; Verdinelli and Wasserman, 2023; 2024). Inference based on Shapley values can utilize sample-splitting methods, as demonstrated by Williamson and Feng (2020). While extending our Zipper framework to accommodate this sample-splitting inference is feasible, it requires the evaluation of all covariate subsets, entailing significant computational resources due to the necessity of fitting numerous models. Moreover, as noted by Verdinelli and Wasserman (2024), many of these submodels may not contribute meaningfully to the overall measure, particularly when each is weighted equally in the definition. We recommend that if there is confidence in the algorithms' ability to approximate true submodels accurately, LOCO measures should be considered, where our framework can be effectively implemented.
_Conditional randomization tests_: CRTs offer flexibility in selecting variable importance measures beyond LOCOs. This method necessitates knowledge of the conditional distribution $X_\mathcal{S}\mid X_{-\mathcal{S}}$, from which data can be sampled for multiple model refitting and measure calculation (Candes et al., 2018). Subsequent research, such as that by Tansey et al. (2021), has aimed to mitigate these computational demands. However, when the distribution of covariates is either unknown or cannot be precisely estimated, our methodology presents a more viable alternative to CRTs.
- For your response Q1, you mentioned, "Under H1, it (the variance estimator) underestimates the actual unknown variance, which is advantageous for power enhancement as demonstrated." Could you explain why that is and where it is demonstrated?
Response:
_Variance underestimation under $H_1$_: As outlined in Lines 167 and 218, the unknown variance $\nu_{\mathcal{S},\tau}^2=(\nu_{\mathcal{S},\tau}^{(0)})^2+\tau\eta^2_\mathcal{S}$, where $(\nu_{\mathcal{S},\tau}^{(0)})^2=(1-\tau)(\sigma^2+\sigma_{\mathcal{S}}^2)$. Given that $\eta^2_\mathcal{S}\ge 0$ (with $\eta^2_\mathcal{S}=0$ under $H_0$ and $\eta^2_\mathcal{S}>0$ under $H_1$), it follows that $\nu_{\mathcal{S},\tau}^2\ge(\nu_{\mathcal{S},\tau}^{(0)})^2$. Proposition 2.3 and Line 221 confirm that our variance estimator is always consistent to $(\nu_{\mathcal{S},\tau}^{(0)})^2$ under both hypotheses. Thus, under $H_1$, the estimator tends to underestimate $\nu_{\mathcal{S},\tau}^2$.
_Power enhancement_: The aforementioned underestimation leads to an advantageous adjustment in our approximated power function, as outlined in Theorem 2.6 and Line 232. Specifically,
$$
G_{\mathcal{S},n,\alpha}(\tau) = \Phi\left(-\frac{\nu_{\mathcal{S},\tau}^{(0)}}{\nu_{\mathcal{S},\tau}}z_{1-\alpha} + \frac{\{n/(2-\tau)\}^{1/2}\psi_\mathcal{S}}{\nu_{\mathcal{S},\tau}}\right)\ge \Phi\left(-z_{1-\alpha} + \frac{\{n/(2-\tau)\}^{1/2}\psi_\mathcal{S}}{\nu_{\mathcal{S},\tau}}\right),
$$
This expression establishes a lower bound that corresponds to the power function of a test statistic employing a consistent variance estimator for $\nu_{\mathcal{S},\tau}^2$ under both $H_0$ and $H_1$. This explains how the underestimation enhance the test's power. Between Lines 231-236, we further elucidate that this lower bound outperforms the power achievable by conventional sample-splitting-based statistics, thereby underscoring the efficacy of our overlapping mechanism in boosting power.
Thank you once again for your insightful questions.
References
Brian Williamson and Jean Feng. Efficient nonparametric statistical inference on population feature importance using Shapley values. *Proceedings of the 37th International Conference on Machine Learning*, PMLR 119:10282-10291, 2020.
Isabella Verdinelli and Larry Wasserman. Feature importance: A closer look at shapley values and loco. *arXiv:2303.05981*, 2023.
Isabella Verdinelli and Larry Wasserman. Decorrelated variable importance. *Journal of Machine Learning Research*, 25(7):1–27, 2024.
Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for gold:‘model-x’knockoffs for high dimensional controlled variable selection. *Journal of the Royal Statistical Society: Series B (Statistical Methodology)*, 80(3):551–577, 2018.
Wesley Tansey, Victor Veitch, Haoran Zhang, Raul Rabadan, and David M Blei. The holdout randomization test for feature selection in black box models. *Journal of Computational and Graphical Statistics*, 31(1):151–162, 2022.