Variable importance assessment has become a crucial step in machine-learning applications when using complex learners, such as deep neural networks, on large-scale data. Removal-based importance assessment is currently the reference approach, particularly when statistical guarantees are sought to justify variable inclusion. It is often implemented with variable permutation schemes. On the flip side, these approaches risk misidentifying unimportant variables as important in the presence of correlations among covariates. Here we develop a systematic approach for studying Conditional Permutation Importance (CPI) that is model agnostic and computationally lean, as well as reusable benchmarks of state-of-the-art variable importance estimators. We show theoretically and empirically that $\textit{CPI}$ overcomes the limitations of standard permutation importance by providing accurate type-I error control. When used with a deep neural network, $\textit{CPI}$ consistently showed top accuracy across benchmarks. An experiment on real-world data analysis in a large-scale medical dataset showed that $\textit{CPI}$ provides a more parsimonious selection of statistically significant variables. Our results suggest that $\textit{CPI}$ can be readily used as drop-in replacement for permutation-based methods.
Paper
Similar papers
Peer review
Summary
In this work, the authors proposed a novel conditional permutation importance approach (CPI) that can well estimate the variable importance with the p-value provided under the condition that correlation exists across features. The authors theoretically justify the inflation of the feature permutation-based approach and justify the calibration of CPI. Empirically, CPI yields calibrated results across different settings with good power, which outperforms existing methods in the related field.
Strengths
This paper focuses on addressing two key issues in interpretable machine learning: statistical calibration and scalability. The proposed method, CPI, shows a significant improvement in statistical calibration compared to the naive feature permutation-based approach. The paper also provides theoretical justification for CPI's ability to effectively control type-I errors. In simulation datasets, CPI outperformed other methods that addressed feature dependencies in terms of the AUC that measures the consistency of feature ranking.
Weaknesses
The existing evidence is insufficient to convince me that CPI is well-calibrated. Several items can improve the soundness of the paper: 1. The authors should also provide the qqplot with a reasonable amount of tests (>=100) to demonstrate the calibration. 2. The authors used RF to approximate the feature j's conditional distribution; more justification is needed for why this is a good choice. (i.e., Does a simpler model achieve the same FPR and AUC level?) 3. The sample sizes in simulations are fairly small. It is unclear whether the well-controlled FPR was due to the calibration or the lack of power to detect inflation. A sample size of at least 10K is suggested w.r.t the given feature size: 50. If computation is a problem, I would suggest either precomputing the SVD of the covariance matrix and then transposing the data matrix to get $X$ with desired correlation. Or it would be even better if the authors could (also) perform the simulation on the UK Biobank genotype dataset (they have more realistic feature dependencies needed for this work).
Questions
1. It is surprising to see the significant difference in AUC between CPI and cpi-knockoff. Do the authors have insight into what makes the difference? 2. Getting from equation (2) to equation (3) is not intuitive to me. It would be great if the authors could provide more detailed proof. 3. How did the authors compute the actual feature ranking? Which scheme did they use (i.e. Did you take the integration of equation (4); use Shapley value; or rank by coefficients?)
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have addressed several limitations of this work: 1. The limitation of using RF to approximate the conditional independence 2. The limitation of sample size for the trainer Additionally, the theoretical calibration of the CPI relies on several hypotheses (A1, A2, A3, A4, B1, and B2) defined in the supplementary. It would be beneficial to have further discussion on the extent to which each hypothesis can be met in reality. Furthermore, if any of the hypotheses cannot be satisfied, it would be helpful to know the degree to which the calibration would be affected.
Summary
The authors argue for the importance of using conditional sampling in permutation importance (CPI) methods. They show that when features are correlated, using non-conditional sampling for permutation importance (on a finite training set) leads to a bias in the estimated feature importance. (Specifically, the bias is discussed in the case of null features having positive importance). As their main result, they show that under regularity conditions (similar to those of Williamson 2021), the conditional permutation importance estimator is approximately normal and therefore the resulting p-values / intervals would be valid. They produce an implementation of CPI, which they test against the permfit-DNN algorithm of based on Mi et al in a series of experiments. Here the main result is that indeed the non-conditional permutations introduce bias (when interpreted as p-values) and a little loss in "sorting" accuracy (that is in the relative rank of the null examples).
Strengths
Strengths: Clarity: The paper is in general well written and quite easy to follow. The arguments and proofs are surveyed well. The experiments and results are easy to follow. Quality: The theoretical derivations are convincing and seem correct. The experimental results are well set to show the point that methods can break if they don’t account for the correlation of the variables. The comparison to other explainability methods is quite comprehensive in terms of the survey. Novelty: There are some novel aspects to this paper: first and foremost, the method implements a code base for producing conditional inference without assuming a specific type of correlation between the variables, and with the learning model implemented at the same time. This might make the method attractive to users. In addition, we have two main theoretical results. The first is a detailed critical discussion of the assumptions of Mi et al, showing an unaccounted for bias term in their analysis. The analysis is nice, but I am not sure it holds the paper. The second is an extension of the Williamson result for condition permutation importance, showing that the resulting statistic is approximately normal under some conditions. IT is important for the estimator, but as far as I can tell, applies the proofs in the Williamson paper to our estimator. However, here it should be said that in my view the paper significantly overclaims, and does not describe well the current knowledge, See more in “weaknesses”. Significance: The problem of identifying important features that contribute to an outcome of interest is fundamental to scientific research, in particular in the biosciences. Recent years have seen many methods that try to adapt these ideas so they would work well for non-linear models. This paper proposes a new method (and software) for computing conditional permutation importance, and provides some theoretical and empirical evidence for the superiority of the conditional sampling over the marginal sampling. Because conditional sampling is harder, having out-of the box code for performing this analysis may be important.
Weaknesses
Weakness: The main weakness of this paper in my view is that it is not honest enough in representing its contribution to the literature. At a first read, and perhaps for researchers who are not experts in the field, it may sound as if the authors were the first to propose conditional permutation importance. (See the structure of the introduction - the idea of conditional permutation importance is not mentioned, except under their contribution). “””In this work, we introduce an improvement of permutation importance called conditional per46 mutation importance. This method uses a conditional sampling step to compute the conditional importance, taking into account the correlation between the variables. In section 3 we analyze the limitations of the permutation approach. In section 4 we describe a conditional sampling based variable importance framework and establish that it controls type-I errors. In section 5, we systematically compare the most popular variable importance measures on various classification 51 and regression tasks, with a focus on type-I error control.””” When they acknowledge other CPIs, in passing in Section 2, they only mention in the context of Strobl et al. In practice, CPIs have long been discussed in the general framework of feature importance. See for example Chapter 8.2 of Fisher et al in JMLR. More generally, the debate whether features removed (in a simple removal as in Covert 2022, or as in SHAP) should be imputed from the marginal distribution or from the conditional distribution has been ongoing. For example, the original definition of SHAP is designed to sample conditionally, but for computational reasons resorts to marginal sampling. In Tree-SHAP, the sampling is conditional. In implying that this paper introduces conditional permutation inference, the authors are also doing a disservice to the readers in neglecting to more clearly stake their actual innovations, allowing the reader to evaluate them. For example, is the Mi result important so that showing its fault is important? Articulate this. (In my view, the most important part here is the software with out-of-box estimators for the conditional distribution. But other arguments by the authors would make their case more clear, Currently, reading the abstract, I understand their main innovation to be introducing CPI, which obviously is not true. (Comment: My current very low grade is due to this weakness. A satisfactory rewrite of this would probably allow me to increase the score to the area of just below or above acceptance). Other than that, it seems to be a paper with rather modest innovation, outside of the software itself. Perhaps my view is dimmed by the overclaiming, and put in proper context I will be happy to re-evaluate. 3.Because in my view the most important part is the software tool, I would have loved to see more about how it works compared to other methods. In particular, more sophisticated relations between variables; how sensitive it is to an increase in p compared to other methods; An interesting question that goes unanswered is how much tweeking goes into making the method work well for new data; perhaps an analysis of multiple datasets would help here, (though I understand the added costs in writing that paper). Other comments: Out of the box methods for conditional inference could backfire if the relation between the variables is hard to model correctly, or if there are too many variables / not enough samples so that variables are almost completely determined by other variables. I would imagine that a conditional method would provide some quality control and safeguards regarding the accuracy of modeling one variable from the rest. Both the type-1 error statistics and the AUC are defined in terms of telling appart important and unimportant variables. Is there any sense where we can understand the accuracy or variability in ranking for the non-null variables? Feature important values can usually be interpreted for non-null cases as well. Smaller comments: Figure 4: color scheme is not consistent between left and right figures (same colors, different meanings). On the right color mean intersection of model results. On the left, the same colors mean source of features, whereas the intersection of model results are marked by the symbols. Also, crosses and circles almost identical; top rank is 1 not p which should be mentioned; Right side is a bit hard to parse. #### Changes #### Following discussions, the authors suggested specific changes to the writing and to the citations that I believe better reflect their contribution to the field. I will therefore update the score to borderline reject.
Questions
See above.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
I think one limitation not discussed is the difficulty in modeling the conditional distribution, and what happens if there are shortcomings in this model. (No societal impact problems :)
Summary
The authors present an approach that aims to tackle the problem with the default permutation importance technique of performing variable importance. The default setting is prone to detecting correlated variables even as important even when they are not (Type I error). The authors present Conditional permutation importance: an approach that fixes this problem by conditionally sampling the variables based on the correlations. The results indicate the method is robust to type I errors, model agnostic and computationally efficient.
Strengths
The paper has a number of strengths including a novel approach to solving the correlation issues with standard permutation importance methods, a rigorous analysis of the proposed method with clear results. The paper is very well written and formatted leading to a well oriented reading. The related works are discussed appropriately with similar methods compared as baselines. Its value to the field is clear and its applications broad. This is a strong paper.
Weaknesses
- The clarity of the experiments can be improved. Firstly the selections of models, datasets and settings can be motivated better. Secondly, the first paragraph of section 5 mentiones that “default behavior” was implemented. It’s not very clear what this default behavior is. - The contributions mention a computationally lean method, is this analysis presented somewhere? I cannot see it in the paper.
Questions
- The figure text is a bit small and hard to read if not zoomed in on a PDF - The paragraph formatting is strange is some places. For examples compare the strange linebreak in lines 26-27 with the standard Neurips format seen between paragraphs 74-75. - Is permutation performance (general setting) prone to type II errors for certain models? One can imagine a hypothetical setting where a variable, age, is duplicated, i.e. full correlation between the two variables. A certain type of model can learn to predict solely based on “age copy”, since it includes all the required information and “age original” is redundant. “Age original” will therefore have 0 importance from a PI analysis. Are these not considered as the models that collapse onto one feature are rare?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
There is a notable lack of discussion of the limitations of this method. One of the few weaknesses. The authors have many discussion points to touch on: computational complexity, sensitivity to the data structure, missing data, types of correlation, model configuration, just to name a few. The discussion section is fairly short, with a very shallow discussion on the limitations. With this part expanded and made more comprehensive, this paper could go from strong to very strong.
Summary
The paper proposes a statistical testing framework for more effectively determining variable importance in generic learning settings. Using a conditional permutation test, they show that correlated variables are better handled theoretically and empirically compared to existing unconditional setups.
Strengths
The proposed test is asymptotically and practically efficient, and while some of the theory (and assumptions) may seem strong, the resulting algorithm and test procedure are extremely clean. They take advantage of an interesting existing result in [Williamson et al 2021] and show that it can be implemented quite directly in modern learning settings. The summary of related work seems complete. The experimental evaluation is strong in supporting the claim that conditional importance is relevant and taking it properly into account leads to performance gains with respect to identifying variable importance compared to unconditional approaches, as well as the suite of selected other methods in the space.
Weaknesses
1. It's unclear to me how the performance of the method would stack up against aggregrated Shapley or LIME-like methods. As the authors say they are quite popular, and while "local" or "instance-based", it would be interesting to see if some naive (or sophisticated) application of those is interesting here or not. 2. Figure 4 and the related discussion around L283 is somewhat difficult to parse. It's also not clear what is meant to be taken away from this; is one in better agreement with scientific consensus? What is the interpretation here? 3. The discussion of computation time in Section 7 suggests that it is significant. The result in the supplement suggests it was an additional hour or so (~450-525 min). How does this scale with number of variables? Number of samples? The abstract claims "computationally lean", I think if the authors want to state this, support is necessary in the main paper. Minor: 4. The organization of the experimental section makes it quite difficult to read. I would suggest merging Sections 5 and 6 so that a reader does not need to flip back and forth between experimental setup and result. 5. Might be worth summarizing the power result from the Supplement in the main paper. 6. A few of the references are missing identifying information such as arXiv or DOI links, or venue.
Questions
How good is the normal distribution assumption in practice? It might be interesting or valuable to plot the estimated null distribution in a few different settings with no variable importance. I'm leaning towards acceptance for this work, if the above issues are well addressed. It is quite neat, and exposed this reviewer to a number of interesting ideas.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
It's not clear if some version of existing Shapley/LIME/etc estimators may achieve similar results. The efficiency claim is not well evaluated and does not seem true.
I appreciate the amount of work that the authors have put in responding to my and others' questions. I am quite satisfied with the authors' responses to my questions and the global response, and am happy to increase my score barring any other qualms that come up as I read other reviewer responses.
We thank the reviewer for the positive appraisal of our revision efforts. This feedback is encouraging us to include the extra plots shared for the revision in the supplementary material to allow future readers to benefit from the improvements obtained through our scientific exchange.
The authors have detailedly addressed the majority of my concerns in their rebuttal. One interesting observation the authors have shown is that as the max depth of the tree increase, the FPR becomes gradually conservative. My takeaway is that model overfitting will lead to conservative results, while an underfitting model can lead to inflation. But it can be tricky, in practice, to find such an estimator that well-fits the data. It would be great to understand the reason behind this, and I suggest the authors further justify their algorithm's robustness in FPR control or acknowledge this limitation. However, I understand the limited time the authors have in addressing all my concerns. The revised article has shown improvement compared to the original version, therefore I will adjust my rate to 6.
We thank the reviewer for the positive appraisal of the efforts we put in the revision. We agree that the subject of controlling the calibration of the p-values under the use of the Random Forest model as a conditional probability learner is important. We hope that the following consideration will help clarify how the depth experiment and our initial results are connected: In **PDF Fig.R3**, we demonstrated that an overly simplistic model (small depth) is losing its calibration and a more complex model (with high depth) is more conservative. Importantly, in **PDF Fig.R1** we reported satisfying calibration, which was not ad-hoc but achieved through hyper-parameter tuning of the Random Forest’s depth. Concretely, we found that, for our setting, 2-fold cross validation for the depth of the Random Forest model among a list of pre-defined depths {2, 5, 10} was satisfying to achieve this calibration. We used a small set of pre-defined depths not to lose the computational benefit of the method while maintaining the predictive accuracy. The 2-fold cross validation was performed for each sampling step (thus with new incoming data). To clarify the importance of tuning for model calibration, we will highlight the 2-fold cross validated depth for the Random Forest model and include these additional results in the supplementary material, so that the readers can benefit from the results of this conversation.
״The reviewer flags a potential misunderstanding. ״
I thank the authors for their answer. As far as I can tell, the issue with moving between unconditional and conditional sampling of the missing features seems to be well discussed in the literature, and as I said before I found the presentation in this paper lacking. One place to see this is in "Debeer, D., Strobl, C. Conditional permutation importance revisited. (2020)", which I think illuminates how the conditional and marginal importance reflect two different definitions as to the meaning of feature importance more broadly than in Random Forest. Another example for this discussion (in the CS side of the literature) can be seen in the SHAP papers which claim that it is better to sample the imputed features from the conditional distributions (though sometimes they settle for the marginals). They are critiqued by Merrick and Taly (The Explanation Game: Explaining Machine Learning Models Using Shapley Values, 2020, page 6), who claim that conditional sampling may lead to artificially inflated importance. (I think Debeer and Strobl's perspective may help understand this discussion). My point is that both marginal and conditional sampling for missing features are well known in the literature, and that their merits have been discussed. I think a new paper should acknowledge these discussions and be specific about the new contributions. ״״״ To address the concern, we carefully rephrased introduction, related work and method sections to clarify how we build on top of the CPI literature. ״״״ As I found this issue to prevail in the abstract and throughout the paper (as detailed in my review), could you be specific regarding the rephrasing? Thank you !
Elaborating conceptual views and proposed changes to the main text
We understand that the reviewer wants to ensure that this potential misunderstanding be resolved and so do we. We would first like to respond to the **conceptual point** regarding marginal and conditional importance. * It is important that conditional inference is well defined mathematically, but our point of departure is that the definition of CPI lacks consensus. In our work we proposed a more general formulation alongside a flexible implementation, which has allowed us to perform a more systematic study of the properties of different permutation schemes and variable importance estimation methods that was lacking so far. Indeed, both conditional and unconditional inference schemes can be useful, which is consistent with some of our results for settings with low to moderate correlation. * That permutation-based approaches are widely used does not mean they are without problems (as developed in section 2, Related work). The work by Debeer et al. (2020), emphasize the limitations of permutation approaches as follows : “Simulation studies (Nicodemus et al. 2010) have indeed shown that even when there is no dependence between the outcome and any of the predictors ($X_k ⊥ Y$ holds for all $X_k$ ), highly correlated predictors (i.e., $X_k ⊥ Z_{(−k)}$ does not hold) have a positive PI (...) when the outcome is independent from the predictors, the CPI does not show the preference for correlated predictors that has been observed in the PI.” We thank the reviewer for having made us aware of this paper, which we have now included in our literature review. ## Changes: Our continuing conversation gives us the opportunity to highlight some of the, potentially, most impactful changes we propose for the revision (which was beyond the character limits of the initial rebuttal). ### In abstract: We change “Here we present a Conditional Permutation Importance approach (CPI) that is both model agnostic and computationally lean.” to “Here we develop a systematic approach for studying Conditional Permutation Importance (CPI) that is model agnostic and computationally lean alongside reusable benchmarks of state-of-the-art variable importance estimators.” ### In Introduction: We reorganized parts of the introduction to form a paragraph explicitly stating the contributions: * In this work, we propose a general methodology for studying the properties of Conditional Permutation Importance in biomedical applications alongside tools for benchmarking variable importance estimators: * Building on the previous literature on CPI, we develop theoretical results for the limitations regarding Permutation Importance (PI) and advantages of conditional Permutation Importance (CPI) given correlated inputs. * We propose a novel implementation for CPI allowing us to combine the potential advantages of highly expressive base learners for prediction (a deep neural network) and a comparably lean Random Forest model as a conditional probability learner. * We conduct extensive benchmarks on synthetic and heterogeneous multimodal real-world biomedical data tapping into different correlation levels and data-generating scenarios for both classification and regression. * We propose a reusable library for simulation experiments and real-world applications of our method on a public Github repo {link provided in the paper}“ As a result, we have removed the potentially misleading sentence “In this work, we introduce an improvement of permutation importance called conditional permutation importance.“ from the paper to avoid the misunderstanding that we invented CPI. ### In sec 2 related work: As a result of our conversation, we have discovered additional relevant references, which we included in section 2 (Related work): * Hooker et al. (2021), Statistics and Computing; * Molnar et al. (2021), International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers. We conclude the section: “In summary, previous work has established potential advantages of conditional permutation schemes for inference of variable importance. Yet, the lack of computationally scalable approaches within a unified framework has hampered systematic investigations of different permutation schemes in comparison to alternative techniques across a broader range of predictive modeling settings.” ### In section 4.2 (Practical estimation): We change “we describe how to compute conditional permutation importance”, to “we present our proposed method for computing conditional permutation importance” ### In Discussion: We changed “In this work, we introduced *CPI-DNN* for estimating predictive variable importance, which was inspired by the limitations of the *Permfit-DNN* approach.” to “In this work, we developed a framework for studying the behavior of marginal and conditional permutation methods and proposed the *CPI-DNN* method, which was inspired by the limitations of the *Permfit-DNN* approach.” to avoid the misunderstanding that we invented CPI.
Changes proposed by the authors
Hi, I thank the authors for their proposed changes. I think they do a better job now in describing their contributions, and allow me to remove my strong insistence on rejecting and increase the score to 4 (borderline reject). Two more side comments on this: - In the reading of the original paper, it was not clear to me they were looking for a feature importance measure for biomedical applications. I think I now better realize their choice of data examples and motivations. - In reading your quote from Debeer, it seems to make sense to cite Nicodemus et al. 2010 as well, as a paper that highlights the issue of using unconditional permutation.
Side comments
We thank the reviewer for his insightful comments and we are happy that the proposed changes helped to address the misunderstanding flagged in the initial response. For the side comments: * As for the applied context of the paper, we want to point out that in the abstract we mention “An empirical benchmark on real-world data analysis in a large-scale medical dataset showed that CPI provides a more parsimonious selection of statistically significant variables.” and “Biomarker development is increasingly focusing on multimodal data including brain images, genetics, biological specimens and behavioral data“ (L.19-21) in the Introduction * Deber et al.’s work citing the work by Nicodemeus et al. 2010 is indeed a great work to add to our literature study. Yet, we’ve also mentioned two important works that highlights better the drawbacks of the use of non-conditional permutation, which we reported citing in our last response (we could not elaborate details given the character limitations): * “While naı̈ve permutation-based methods can be appealing, we have shown with some quite simple examples that they can give misleading results. We also identify the extrapolation behavior by flexible models as a significant source of error in these diagnostics. The precise biases that permute-and-predict methods produce will depend on the learning method employed as well as the specifics of the dependence of the features and the response function; see Bénard et al (2021) for approaches to this analysis.“, the work by Hooker et al.(2021) “Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model or There Is No Free Variable Importance“ * “When features are dependent, perturbation-based IML methods such as PFI, PDP, and Shapley values extrapolate in areas where the model was trained with little or no training data, which can cause misleading interpretations.“ section 5.1 in the work by Molnar et al. (2021) “General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models”
I thank the authors for taking my feedback into consideration. I will be keeping my overall rating.
Decision
Accept (poster)