Summary
This paper focuses on a problem with Network Dissection and Compositional Explanation methods: these two methods explain the concept encoded by a neuron (or, more precisely, a convolutional filter) by only considering highly activated regions in the feature map. To address the problem, this paper proposes to divide the activation values into different ranges (clusters) and generate an explanation for each range of activation value. This paper also proposes a heuristic to accelerate the search for the optimal logical combinations of concepts in the Compositional Explanation method.
Strengths
1. This paper focuses on an important issue, i.e., previous concept-based explanation methods, such as Network Dissection or Compositional Explanation, neglect feature map regions with low activation values.
2. The authors theoretically prove that the proposed heuristic is admissible, which guarantees the optimality of the solution found using the heuristic.
Weaknesses
1. The contributions are incremental and miscellaneous. From my view, the contribution of this paper is mainly two-fold. The first contribution is that the paper considers different ranges of activation values to address the problem with previous explanations (Network Dissection and Compositional Explanation) that they only analyze the feature map regions with extremely high activation values. However, the proposed method is only a simple extension of previous methods, and the motivation of using clustering instead of manually dividing the activation values into different ranges is not well explained. The second contribution is to propose a heuristic searching method to accelerate the Compositional Explanation method. However, the second contribution is not inherently correlated with the first one, and I feel the paper is separated into two uncorrelated parts.
2. Many of the notations are inconsistent or confusing. For example:
(a) In Line 106, the intersection size is defined as $IMS(x,L)$. But in Eq. (5) to Eq. (7), it is denoted as $IMS_{[\tau\_1, \tau\_2]}(x,L)$. The meaning of the subscripts needs to be clarified.
(b) In Eq. (1), $\mathcal{L}^n$ should be $L$, in order to be consistent with Eq. (2).
(c) In Eq. (5)-(7), there is a notation $L_L$. I’m confused about what the two L mean respectively. If they refer to different meanings, please use different letters.
(d) In Eq. (5)-(7), there is no explanation for the subscript $R$ in the notation $L_R$.
(e) In many equations, such as Eq. (2), Eq. (11), Eq. (13), Eq. (14), an equal sign is missing.
3. The metric Activation Coverage seems redundant. Given the metric Intersection Over Union and the metric Detection Accuracy, we can actually derive the metric Activation Coverage by using the inclusion-exclusion principle $|A\cup B|=|A|+|B|-|A\cap B|$. Therefore, I doubt this metric is redundant.
Questions
1. In Table 2, when computing the desiderata qualities (or metric scores) of the proposed method, the authors simply average the scores from the 5 clusters. However, since different clusters correspond to different ranges of activation values, does it make more sense to assign different weights to the scores of these clusters? For example, it is natural to assign higher weights to clusters with higher activation values. Nevertheless, the authors are also encouraged to explain why doing simple averaging is a reasonable choice.
2. As noted by the authors in Line 259 to Line 276, the concept labels obtained from Cluster 0 and Cluster 1 usually represent the “default labels” to which the algorithm converges when the activations are random. This implies that the labels obtained from Cluster 0 and Cluster 1 actually cannot explain what the model has learned from the data. In this way, the metric scores of Cluster 0 and Cluster 1 are meaningless and should not be used to compute the final score of the proposed method.
3. Why do the authors set the number of clusters to 5? If there are more or fewer clusters, will the analysis and conclusion in Sections 4.3 and 4.4 change? I suggest the authors conduct ablation studies regarding the number of clusters and clearly explain how it will affect the empirical results.
4. In Table 3, why do the results with ReLU greatly differ from results without ReLU? Could the authors give some explanations or discussions on this issue?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Yes, the authors have discussed the limitations and broader impact of their work.