Summary
This paper proposes a real-word benchmark to evaluate feature-based explanations of black box machine learning models. The authors collect six real-word datasets from three different domains to evaluate the performance of feature attribution methods in terms of pre-defined "expert features" $G^*$, i.e. a collection of feature sets that provide a meaningful interpretable "ground-truth" explanation. This is in contrast to existing work that focuses mainly on general metrics and synthetic examples without including domain knowledge specifically tailored to the given dataset and task. Given an instance $x$ and a collection of feature sets $\hat G$, i.e. the proposed explanation, the FIX benchmark computes essentially a heuristical score that determines how well $\hat G$ and $G^*$ align: For each feature $i$, the average "amount of alignment" of all elements in $\hat G$ that involve feature $i$ is compared with the ground-truth elements of $G^*$. This alignment is computed using the "ExpertAlign" function, which is either task-specific or simply the maximum intersection over union metric with the human-annotated ground truth "expert features". This score is then averaged over all features $i$ (Eq. 1). The benchmark covers the extraction of "expert features" for all six datasets, where for two datasets human-annotated are used, and the remaining four datasets have "implicit expert features", which are defined indirectly via a task-specific scoring functions. The paper then proceeds to evaluate several domain-specific pre-processing techniques (such as patches, watershed for images) and some feature-based explanations (such as Archipelago, CRAFT) to evaluate their score on the benchmark. From these results the authors conclude that current feature-based explanations are not sufficiently aligned with "expert features".
Strengths
1. The evaluation of feature-based explanations with respect to interpretability is an important topic, where human-annotated ground-truth explanations specifically tailored to a given task and domain are an important tool to evaluate approaches.
2. The proposed datasets cover multiple domains, and the proposal of a single score for each task is good for comparing methods. The applications seem interesting, although I cannot judge on the quality of any of the proposed expert features.
3. The paper is well-structured and easy to follow, and the purpose of the paper is clear
Weaknesses
1. **Lack of popular feature attribution methods:** The paper proposes a benchmark that has the potential to improve explanations based on groups of features. However, the benchmarking of methods is significantly under-developed. The paper claims (e.g. in the abstract) that feature-based explanations fall short in identifying the given "expert features". This claim is not sufficiently covered by the experiments. The authors do barely evaluate feature-based explanations. In the experiments, from current XAI literature only Archipelago is considered (and image-specific variants, such as CRAFT). While in the introduction feature attributions, such as LIME or SHAP are discussed, none of these concepts appear in the empirical evaluation. In contrast, the evaluation in Table 2 centers around pre-processing techniques rather than feature attribution methods. For improvement, to support this claim, I would suggest to include a variety of well-established feature attribution methods, such as SHAP, Integrated Gradients, LIME and their variants for higher-order explanations and evaluate their performance on the FIX benchmark. Moreover, it is unclear to me, why the proposed FIXScores reported in Table 2 are insufficient / unsatisfactory.
2. **Unclear modeling choices:** Given only the datasets, it is difficult to execute a benchmarking of XAI methods, since a black box model is required. The paper does not address the question on how to model these datasets in the first place. A potential improvement would be to clearly discuss the modeling aspect of the datasets, or incorporate the trained model in the benchmark. As of now, it is unclear if any results obtained by the benchmark is due to a bad modeling choice or a bad interpretability method. Moreover, it is even unclear if the "expert features" and actual model reasoning are aligned, which would be crucial for the benchmark (see Q1-Q3 below).
3. **Implications remain unclear:** The paper in its current state has limited impact on future research, since it does not provide clear evidence of limitations in current XAI methods (see above), nor where these are rooted in or how they could be mitigated in the future. It would be very helpful to clearly evaluate how these methods fail, why, and to propose improvements or directions for future developments.
Questions
1. What are the performances of the models trained for the given task? How difficult is it to train such models on these datasets? Are the performances sufficient to draw any conclusions about the quality of explanations or could these problems also be inherently incorporated in the trained models?
2. How do you ensure that the trained models actually align with the expert features in their reasoning?
3. What requirements should models satisfy to be eligible for an evaluation of explanations using these datasets? Would you include the models as part of the benchmark?