Decompose the model: Mechanistic interpretability in image models with Generalized Integrated Gradients (GIG)

### W1: **Guarantee of Finding Human-Understandable Representations with iERF** & W5/W6: **Detection of Non-meaningful Features** We acknowledge your concern regarding the guarantee of finding meaningful, human-interpretable representations for each concept using instance-specific Effective Receptive Fields (iERF). While iERF provides a localized understanding of how features contribute to model decisions, it is true that not every identified concept may align with human-defined semantics. While we acknowledge that some concepts may lack immediate semantic clarity, a substantial portion of the concepts identified by our method are indeed interpretable by humans, as evidenced by the visualizations in our qualitative experiments. These visualizations demonstrate that our method captures shared attributes across multiple classes, which can be intuitively understood by human evaluators. By focusing on dataset-wide patterns rather than isolated class-specific features, our approach aims to enhance the interpretability of deep models, even if some concepts are abstract or challenging to label. We will continue to refine our method to increase the proportion of human-understandable concepts, particularly by exploring additional evaluation techniques to bridge the gap between model-centric and human-centric interpretations. --- ### W2: **Limitations of Using ImageNet Validation Set** We chose this dataset primarily due to its widespread use and availability, allowing for reproducible and comparable evaluations with other future interpretability methods. However, we recognize that extending our analysis to additional datasets could provide a more comprehensive understanding of the model’s learned representations. --- ### W3: **Clarity of Section 3.2.1** We appreciate the feedback on the clarity of Section 3.2.1. The last two sentences in that section were intended to emphasize our strategy for handling the foreground-background imbalance in images when sampling PFVs. Specifically, we sample one PFV per image based on its contribution to the output logits, thus prioritizing features that are more predictive of the class. We understand that this explanation could have been clearer, and we will revise this section to improve its readability and precision in the final manuscript. - **Original**: Thus, we probabilistically sample a single PFV from each image in proportion to its contribution to the output, ensuring a more balanced dataset and solving the imbalance problem. - **Revised**: Thus, to address the foreground-background imbalance, we sample a single Pointwise Feature Vector (PFV) from each image. This PFV is chosen probabilistically, with a preference for those that contribute more significantly to the model’s output logits. This approach ensures that the selected PFVs are highly relevant to the class predictions, thereby creating a more balanced representation of important features across the dataset. --- ### W4: **Over-Segmentation and Under-Segmentation in Clustering** & Q2: **Number of Clusters** Thank you for pointing out the potential issue of over-segmenting or under-segmenting concepts during clustering. This is indeed a known challenge when applying clustering algorithms to high-dimensional data. To address this, we conducted an ablation study with different numbers of clusters in bisecting k-means. We provide these results in the comment provided to everyone (5) to offer clear guidelines for hyperparameter selection.

### **Q5: Explanation of the Blue Line in Figure 5 (Deletion from Layer 2.3 to Layer 3.0)** Thanks to pointing out the **anomaly between layer2.3 and layer3.0**, this observation led us to uncover an intriguing phenomenon.Specifically, when we zero out the layer activations (effectively deleting all concepts) and perform a forward pass using only the bias values (i.e., the 3rd batch normalization bias of **layer2.3** and the 1st, 2nd, and 3rd batch normalization biases of **layer3.0**), we observe that the norm of the PFVs at a certain layer (7.72 between **layer2.3** and **layer3.0**) is unexpectedly higher than the norm of the reconstructed vectors(4.64 between **layer2.3** and **layer3.0)**. In **Figure D-1 (projection)**, we illustrate the ratio of the maximum projection of the concept-deleted PFV to the maximum projection of the original PFV. Mathematically, this is represented as: $\frac{max(<\Omega',\bar{v}^b_t>)}{max(<\Omega, \bar{v}^b_t>)}$, where $\Omega$ is the output embedding with the complete PFV vectors, $\Omega’$ is the output embedding with the *deleted/inserted* PFV vectors, and $\bar{v}^b_t$ is the normalized version of the target concept vector $v^b_t$. As $<\cdot, \cdot>$ is the inner product operation, we could rewrite this as: \begin{equation} \frac{\max(cos(\Omega')\cdot norm(\Omega'))}{\max(cos(\Omega) \cdot norm(\Omega))} \end{equation} Here, we can rewrite this equation, where $\omega_i$ is the embedding of $\Omega$ at the $i$-th position, and the index $i$ has the $max(\Omega’)$, the index $j$ has the $max(\Omega)$. \begin{equation} = \frac{cos(\omega'_i)\cdot norm(\omega'_i)}{cos(\omega_j) \cdot norm(\omega_j)} = \frac{cos(\omega'_i)}{cos(\omega_j)} \cdot \frac{norm(\omega'_i)}{norm(\omega_j)} \end{equation} The cosine similarity term, $\frac{cos(\omega'_i)}{cos(\omega_j)}$, shows directional change after the deletion/insertion of the concepts, and the norm ratio term, $\frac{norm(\omega'_i)}{norm(\omega_j)}$, shows the norm ratio changed due to the deletion/insertion. We separately plot the cosine similarity term and the norm ratio term in **Figure D-2 (cos)** and **Figure D-3 (norm)**, respectively. - **Top:** Deletion experiments - **Bottom:** Insertion experiments Across all layers, the **cosine similarity** either decreases or increases depending on the concept deletion or insertion. However, at specific layers, the **norm ratio** notably increases after concept deletion, resulting in a sharp change in the **projection score**. Especially, during the final 25% of the insertion/deletion process. In conclusion, these findings demonstrate that concept vectors effectively influence the directions of the **target layer's concept vectors**, as scored by GIG. This observation validates that our method, GIG, accurately measures the importance of concepts by quantifying their contribution to subsequent layers.

### W1: Definition of "Concept" and Choice of clustering method and number of clusters **W1-a: Definition of Concept**: Please refer to the comments provided to everyone (2) above for further clarification. **W1-b: Choice of clustering method and the number of clusters**: We selected bisecting k-means due to its adaptability to the sparse and variably dense nature of PFV data, as discussed in our main text [lines 65-69]. Also, in Sec 4.2.1, we compared our clustering method, Bisecting Clustering, with other concept extraction methods including Dictionary Learning and Sparse AutoEncoder. Furthermore, to address your concern on choosing the number of clusters, we conducted additional ablation study, which is Figure B in supplementary materials. For detail, refer to the comment provided to everyone (5). --- ### W2: Computational Cost and Scalability GIG. We understand the concern regarding the computational cost associated with our method. However, we would like to clarify that the Generalized Integrated Gradients (GIG) approach is optimized for scalability, which also we shared to everyone (4). --- ### W3: Generalizability beyond ResNet50. Our method is not inherently tied to convolutional architectures. The **PFV-iERF framework** can naturally extend to Vision Transformers (ViTs) by treating each patch as an analysis unit similar to PFVs in convolutional layers. In essence, a convolutional layer can be seen as a fixed-attention Vision Transformer, where each PFV serves as a “token” within the layer. Thus, by replacing PFVs with tokens, our method can be adapted to Transformer-based architectures. We expect that the **interlayer concept attribution**, GIG, also can naturally extend to every models with overcomplete linear bases with their semantically meaningful features. --- ### W4: Potential Loss of Granularity in Concept Clustering You raised a valid concern about the potential loss of detail when clustering PFVs into broader concepts. We agree that concept aggregation might obscure finer nuances.: - **Overcomplete Representation:** We use an overcomplete set of concept vectors (eight times the number of channels), which preserves a high level of detail. This overcompleteness allows us to capture subtle variations within the data. As the ablation study on the number of concepts shows that we can control the fineness of the concept by adjusting $K$, we can increase $K$ for the tasks requiring detailed feature analysis. --- ### W5: Subjectivity in Concept Interpretability We acknowledge that the interpretability of concepts can be subjective, depending on human evaluators. To address this, we have taken the following steps: - **Top-10 Embedding Visualization**: To make the extracted concepts more interpretable and persuadable, we provide the top-10 embeddings with the highest cosine similarity to each concept vector. This helps identifying whether the extracted concepts correctly represent features. - **Objective Evaluation Metrics**: We have implemented quantitative metrics (C-Deletion, and C-Insertion) to evaluate the relevance and robustness of extracted concepts, reducing reliance on human judgment. These metrics are widely recognized for their ability to validate concept-based explanations across diverse models. --- ### W6: Lack of Robustness Testing for Identified Concepts Thank you for pointing out the importance of robustness testing for the identified concepts. Your insightful comment inspired us to conduct an additional set of experiments, which turned out to be both interesting and valuable. We are pleased to share the results of these robustness experiments in Comment provided to everyone (6), as we believe they hold enough merit to be shared with every reviewers. **W6-a: Adversarial Stability**: Please refer to the comment provided to everyone (6). **W6-b: Cross-Model Consistency**: Also, thank you again for highlighting the importance of evaluating cross-model consistency. In the paper, we used most widely-used pytorch resnet50 (weight=IMAGENET1K_V1) model. And now, we are currently in the process of conducting on recent version of resnet50 (weight=IMAGENET1K_V2). As both architecture is essentially same, we expect that our method can be applied well. We will update our findings as possible as we can.

Thank you for recognizing the potential of our proposed PFVs, ERFs, and GIG methodology in advancing the interpretability of image models. ### W1: **Novelty of our GIG and Theoretical Justification** This is the first to systematically trace the entire pathway from input image through all intermediate layers to the final output within the whole dataset. Existing mechanistic interpretability analyses in large language models, such as Google DeepMind’s Gemma Scope [6] or Anthropic’s work [7], do not seek interaction between concepts, though it is crucial to explain the model’s prediction mechanistically. Also, they typically treat hidden layer token vectors as input word of their position. However, this is not applicable (and possibly wrong) to the image field, where token vectors (or PFVs) do not represent individual pixels or patches directly. Instead, they correspond to receptive fields, and it is essential to consider instance-specific ERF as the representation of the interested vectors. Also, we position GIG as part of an evolutionary framework in the field of interpretability, from attribution methods (local explanation methods in Related Works), through concept attribution methods (global), to interlayer concept attribution methods (mechanistic interpretability). Here, we outline the progression from traditional attribution methods to our interlayer concept attribution: - **Traditional Attribution (e.g., IG, LRP):** Methods like Integrated Gradients (IG) [1] and Layer-wise Relevance Propagation (LRP) [2] have primarily focused on pixel-level or neuron-level attributions within a single layer, explaining model predictions by identifying which input features (pixels) contribute most to the output. These methods work well for instance-specific explanations but often fail to capture higher-level semantic concepts, especially in the context of image models. - **Concept Attribution (e.g., TCAV, CAT):** To bridge this gap, concept-based approaches like TCAV [3] and CAT [4] have emerged, focusing on explaining model behavior using high-level concepts rather than individual features. These methods introduced the idea of attributing model predictions to predefined or automatically extracted concepts, but they are typically confined to analyzing a single layer at a time, and also limited to class-specific explanations. - **Interlayer Concept Attribution (our GIG):** Our method advances beyond single-layer concept attribution by introducing interlayer concept attribution. GIG allows us to systematically quantify how concepts evolve and influence model decisions across multiple layers. This is particularly challenging in image models, which are often sparse and highly contextual. In summary, as our work is the first attempt at dataset-wide mechanistic interpretability to achieve interlayer concept attribution, there are no existing methods for direct comparison as far. We provide an additional evaluation of the robustness of our approach in the comments provided to **everyone** (6). [1] Sundararajan, M., Taly, A., & Yan, Q. (2017). *Axiomatic Attribution for Deep Networks*. In *Proceedings of the International Conference on Machine Learning (ICML)*, pp. 3319–3328. [2] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). *On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation*. *PLOS ONE*, 10(7): e0130140. [3] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2018). *Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)*. In *Proceedings of the International Conference on Machine Learning (ICML)*, pp. 2668–2677. [4] Fel, T., et al. (2024). *A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation*. In *Advances in Neural Information Processing Systems (NeurIPS)*. [5] Geva, M., Caciularu, A., Wang, K. R., & Goldberg, Y. (2022). *Transformer Feed-forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space*. [6] Lieberum, T., et al. *Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2*. *arXiv preprint arXiv:2408.05147* (2024). [7] Templeton, A. *Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet*. Anthropic, 2024. --- For both **W2: Definitions of Concept, PFV, and ERF** and **W3: Scalability and Computational Considerations**, refer to the comments provided to **everyone**(2-3, and 4, respectively) above for further clarification. ---

--- ### W4: **Evaluation on Larger Models (e.g., ViT, CLIP)** We appreciate your interest in the scalability of our method to larger models, such as transformers. However, we would like to clarify that the current study focuses exclusively on convolutional networks, specifically ResNet50, due to its widespread use and well-understood architecture. While transformers present a compelling future direction, the current scope of our research does not include them. We do, however, plan to extend our framework to vision transformers in future work, as our framework is inherently compatible with patch-based architectures such as ViTs. --- ### Q1: **Sensitivity to Hyperparameters in Bisecting K-means** We agree that hyperparameter sensitivity is an important consideration. To address this, we conducted an ablation study to assess the sensitivity of our method to the number of clusters in bisecting k-means, which is provided on the comments provided to everyone above (5). --- ### Q2: **Comparison of GIG with Other Attribution Methods** As mentioned in **W1**, our interlayer concept attribution, GIG, differs fundamentally from traditional attribution techniques. While attribution methods like Integrated Gradients (IG) focus on single-layer input-output mappings, GIG captures the hierarchical flow of concepts across layers, providing a more nuanced understanding of the model’s internal mechanisms. Of course, the computational cost of GIG is more expansive than the traditional attribution methods, but we can’t say that this is a trade-off as it offers deeper insights by tracing concept formation across layers.

Thank you for your thoughtful review. We appreciate your positive feedback on the organization, clarity, and extensive experiments that validate the effectiveness of our proposed method. Below is the response to the question you raised. We hope this clarifies any concerns. ### **W1: Diversity of Examples for Concept 1838 in Figure 3** We understand your concern regarding the limited diversity of examples for **Concept 1838** ("rounded cone"), as most examples presented were animal noses. To address this, we have included a new figure (**Figure A**) in the supplementary material, which shows the top 100 embeddings of **Concept 1838**. This expanded set of examples illustrates that Concept 1838 is consistently identified as a "rounded cone shape with a black tip," encompassing a broader range of objects beyond animal noses. The diverse examples include various cone-shaped structures, such as certain tools or parts of machinery, which better represent the general characteristics of this concept. --- ### **W2: Clarification of Contribution Values and PFV Decomposition** Before addressing your specific concern, we would like to clarify the distinction between the **Concept Contribution** and the **PFV Decomposition**. #### **W2-a: Concept Contribution Score** (Normalization Not Required But A Good Option) The **Concept Contribution Score** evaluates the impact of each concept vector on the classification logits or the subsequent layer’s concept vectors. Specifically, this score quantifies the extent to which a specific concept contributes to the model’s prediction confidence or the formation of more abstract concepts in the following layers. We intend to reflect the contribution score between layers itself as a confidence score, thus we normalize each score with their maximum value of the corresponding class. Yet, we acknowledge that your suggestion, **normalization (sum to one), is also a good option** for comparing inter-class concepts. #### **W2-b: PFV Decomposition Coefficients** (Absolute Magnitude Difference) The **PFV decomposition** is conducted with (overcomplete) linear basis, which is concept vectors in our case. The discrepancy between the norm of top 1 concept vectors, which you observed (e.g., the '39.4' and ‘7’ values in Figure 7), is due to **differences in the magnitude** of comprised PFV vectors. For instance, the PFV magnitude for the first sample (grapefruit) was **101.1618**, while the PFV magnitude for the bus sample was **42.2055**. These substantial differences in the magnitude of PFV vectors directly affect the decomposition coefficients. --- ### **W3: Explanation for the Huge Drop in the Insertion Experiment (Figure 5)** Although we provided an analysis of this phenomenon in the main text [line 513-516], we have added further clarifications. In earlier layers, the insertion process initially includes positively attributed concepts with the order of their attribution scores of the target concepts. Thus, after most of the concepts are inserted, **negative concepts** are inserted at the last order, which can lead to a slight decrease in the projection values as they may not be directly related to the target class. However, as the network progresses to higher layers (Layer 3.5 → Layer 4), the sharp decline observed is due to the insertion of highly specific, high-level concepts.

### 5. **Ablation Study on Number of Concepts ($K$)** We conducted an ablation study to assess the impact of the number of clusters. Here, we focused on a single layer (**Layer 3.5**) with ($n_{channel} = 1024$). We evaluated performance across various numbers of concepts, $(K = [512, 1024, 2048, 4096, 8192])$. The quantitative results are summarized in the **Table 1**. Table 1: Performance Metrics Across Different Numbers of Concepts | $n_{concept}$ | $\textrm{Rel}~l_2 (\downarrow) $ | $ l_0~\textrm{ratio} (\uparrow)$ | | --- | --- | --- | | 512 | 0.6771 | 0.9834 | | 1024 | 0.6598 | 0.9891 | | 2048 | 0.6403 | 0.9929 | | 4096 | 0.6181 | 0.9950 | | 8192 | 0.5891 | 0.9964 | As the number of concepts, $K$, increases, the relative $l_2$ error consistently decreases, indicating better reconstruction performance. Also, the $l_0$ sparsity improves with higher $K$, showing that the model uses a more fine-grained representation. These results indicate a clear improvement in performance with an increasing number of concepts, which is also seen in the following qualitative analysis. Additionally, we performed **qualitative analysis** to assess how broadly each concept applies as the number of concepts, K, varies. First, we selected two key vectors: the **concept vector (center)** and the **farthest vector** from the concept vector within the concept cluster. Using these two vectors as bases, we constructed a hyperplane. We then projected all vectors within the concept cluster onto this plane. From this setup, we drew a trajectory starting from the concept center towards the farthest vector, extending along this direction. As the trajectory extended, we iteratively selected the closest sample to the current point on the trajectory. In **Figure B** in the supplementary materials, we provide a concept vector “Dog Ear” and compare the concept clusters of varying number of concepts, $K$. With $n_{concept} = 512$, the farthest instance appears as a stone-like horn. At first glance, it might be unclear why this instance is categrozed under the concept of “dog ears.” However, by examining the trajectory, we observe that the concept vector of dog ear gradually darken, transform into goat-like horns, and finally become fully horn-like. This indicates that the concept cluster encompasses a spectrum from ears to gray horns, grouping them under a single broad concept. On the other hand, when $n_{concept}$ is $2048$, the farthest instance is a ear-like fluffy fur, indicating that the farthest concept is closer to the center than before. Additionally, the cosine distance decreases to 0.6332, reflecting a narrower range covered by this concept cluster. Finally, with $n_{concept} = 8192$, the cosine distance further reduces to 0.3801, indicating a very fine-grained range. In this case, even the farthest concept remains strictly confined to “dog ears.” We found that setting of $n_{concept}=8 \times n_{channel}$ yielded the most balanced results in terms of performance and interpretability. Although performance may continue to improve with higher $K$, we limited our experiments to a maximum of 8192 concepts. This limitation is due to the fact that our current dataset contains only 50,000 Effective Receptive Fields (ERFs), resulting in cases where the number of ERFs assigned to each concept could fall below one, which would undermine meaningful concept representation.

### 6. **Robustness Testing for Identified Concepts** We have performed robustness experiments to assess the stability of our concepts: **1) Targeted Adversarial Attack** [8]:We observed the targeted attacked input is more influenced by the concepts of the target class than those of the original class. Furthermore, we showed how the corrupted concepts formed. **2) Gaussian Noise Perturbation**: We found that the extracted concepts remained consistent, even under challenging conditions. This robustness suggests that our method captures genuinely meaningful patterns that are not easily disrupted by input noise. Moreover, our method can show how noise influenced the classification. For detail, please refer to the **[C_Robustness.pdf]** in the supplementary materials. Also, we will update our interactive page for this results as soon as we are ready. With these two experiments, we demonstrate the robustness of our method in explaining the adversarial attacks (which often fails with the traditional local, attribution methods) and preserving the underlying causal structure even with noise perturbation. [8] Carlini, Nicholas, and David Wagner. "Towards evaluating the robustness of neural networks." 2017 ieee symposium on security and privacy (sp). Ieee, 2017.