Understanding Visual Feature Reliance through the Lens of Complexity

Summary

This paper proposes a method to measure the complexity of features extracted by deep learning models. This method is based on $\mathcal{V}$-information, an extension of Shannon's mutual information that takes the computational capabilities of a decoder into account. The proposed measure of feature complexity is inversely related to the cumulative $\mathcal{V}$-information across a model's layers, the intuition being that features that become available at earlier layers will accumulate a larger $\mathcal{V}$-information, while features that only become available late will have a smaller $\mathcal{V}$-information. Features are extracted from a standard, ImageNet-trained ResNet50, from which an overcomplete dictionary is learned. These features are then clustered, and the mean complexity score for each feature is computed. Feature clusters with low, intermediate and high complexity scores are visualized, revealing that low-complexity features tend to be related to uniform colors or low-frequency information, intermediate-complexity features to local shapes such as eyes and noses and high-complexity features to highly structured shapes, such as insect legs. Using the CKA between the features in the dictionary and the activations at different layers within the main branch and residual branch of the network, the authors find that Easy features tend to emerge in early layers and then copied to later layers through the residual branch, while Complex features steadily increase throughout layers in both branches. Analyzing the dynamics of different features' emergence throughout training, complex features are found to emerge later than simple ones. Interestingly, analyzing the interplay between features' complexity and their importance in influencing the network's outputs, the authors find that (a) more important features tend to be less complex, and (b) the complexity of important features is reduced throughout training, suggesting that the network might be "compressing" the more important features.

Strengths

- The paper presents an elegant measure of feature complexity in deep neural networks, which takes into account the networks' computational expressiveness in different layers. - The paper is extremely well written, with detailed methods and supplementary materials, exhaustive analyses and clear visualizations. - The Related Works section is particularly exhaustive and thorough. - It successfully combines several recently proposed interpretability tools. - The analysis of the interplay between complexity and importance throughout learning was particularly interesting, suggesting a mechanism for the compression of important features across learning epochs in deep neural networks.

Weaknesses

- The proposed measure is closely related to the accuracy with which a feature can be linearly decoded from the layers of a network. I believe the paper would benefit from an explicit discussion of what, exactly, is the additional information provided by the proposed method which cannot be gained from directly looking at the "raw" decoding accuracy. - For simplicity, the authors restrict their analysis to a single ResNet model. While this is an understandable choice, I am not sure about what the implications are for models which do not include residual connections. The residual stream is found to be central in "teleporting" simple features to later layers. The authors should explicitly discuss what they predict the pattern of results would be for networks without residual connections. For example, would simple features be overall less relevant to the network's responses? Or would the network implicitly implement a residual-like stream? The limitations of using this architecture exclusively are briefly acknowledged in the Limitations section, but the specific role of residual connections is not discussed there. - The proposed measure is based on a loose assumption that the decodability of features tends to increase across layers. However, it is possible that certain features are _only_ available in early layers (for example, certain low-level image features might be discarded). These features would have a low cumulative $\mathcal{V}$-information, and might thus be erroneously classified as high-complexity. The authors should explicitly discuss whether this is a concern at all, and for what reasons. - The plot in Figure 5B shows that more important features tend to become less complex as training progresses. As the authors have access to the features which are subject to this process, and those which are not, it would be extremely interesting to visualize them, showing what exactly is happening - are the same features being extracted at earlier layers? Or are they actually changing, and starting to resemble simpler features such as colors or edges? The answer to this question would clarify more precisely what the proposed complexity measure is capturing. Is it possible that features we visually evaluate as "complex" can be extracted at early layers if they are important to the network's task? Or is layer depth always correlated with visual complexity?

Questions

No questions, beyond the requests for clarification listed in the weaknesses section.

Rating

8

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

Limitations are discussed adequately. Please refer to the weaknesses section for comments about limitations which I believe were not sufficiently discussed.

Summary

The current work proposes a method to assess the complexity of features in a representation space in terms of usable information. The measure of complexity is related to how far back in the layers of a trained network one can find information about the feature that is recoverable by a linear decoder. Equipped with a measure of complexity per feature, the authors grab 10k distributed features in the penultimate layer of a ResNet50 and then examine various relationships such as the growth in complexity over the course of training, the relative importance of simple features in determining the output, and the “flow” of simple features through the residual backbone of the network.

Strengths

The premise -- of a practical way to assess complexity via tiers of usable information, and then its employment as a route to a better understanding of learned features -- is very interesting. To evaluate the complexity measure, the authors cleverly use the trained layers in the network to approximate a hierarchy of function classes, such that the amount of usable information between the input and the feature must increase as you move deeper into the network, and at each level all that is needed is a correlation measurement. The complexity measure is thus reasonable and straightforward to measure in practice. The authors are upfront about a central assumption related to optimal processing by the trained layers. The writing is clear and the references to related literature are thorough and extensive.

Weaknesses

Many of the analyses are questionable, in my opinion, making the work seem to have a strong premise with a lackluster follow up. I will be happy to be rebutted on these points with justification and clarifications during the discussion period. Specifically, the “what” and “where” analyses are odd to me. My main gripes are below, with smaller points in the "questions" section. - Regarding the qualitative exploration of dictionary vectors, and their grouping into clusters (for “what”): is there a good reason to think relative positions mean anything in feature space, and therefore that clustering is meaningful? Were the dictionary vectors normalized beforehand, or else did the variable magnitude affect the UMAP and clustering? Why were 150 clusters used? Did a sweep over k or any other analysis indicate that the vectors are indeed clustered to some degree, and 150 is an appropriate choice? Unless I missed it, it’s not explained how the meta-clusters were labeled -- was it just visual inspection, and the remaining 120 clusters simply weren’t interpretable? - Regarding the use of CKA for “where”: The proposed complexity measure is already based on how far back in the processing the feature is still recognizable to a linear probe. Why show CKA instead of the usable information as a function of depth for simple vs complex features? It’s not intuitive to me to infer anything about the similarity of the “residual” branch aka simply the feature vec one layer deeper, compared to that of the delta (“main branch”), which is not used as a feature by the network. It seems far more straightforward to look at any changes in the usable information as a function of depth as arising from the delta at that layer. Why not just show the curves for each of the features? The curves would also show the way in which usable information is growing with depth -- is it gradual, or stepped? All said, it’s hard to find much insight from the results of Fig 4, and a straightforward alternative exists.

Questions

- In Appendix A: “The dictionary D was designed to encapsulate 10 concepts per class”. Does this mean the NMF was run separately for each class, and then the dictionaries were all combined? If that’s the case, are there multiple copies of some features (e.g. “wheel” being found for every vehicle class)? - How were the visualizations of Fig 1 produced, particularly the earlier manifestations z2 and z3? How should we interpret a feature’s identity in an earlier layer when it is not recognizable in that earlier layer (and therefore a function of many features)? - Fig 6: What are we supposed to infer from the epoch 1 results? If everything is still essentially random at this stage of training, then we're just seeing meaningless values plotted against each other, right? - The inhibitory/excitatory results (Fig 12) are striking -- **every single inhibitory feature has complexity less than ~0.5?** While the other findings are rough trends extracted from point scatter, this finding is relegated to the appendices and given minimal attention? Questions out of interest rather than needing to fill a gap in the paper: - Regarding the higher importance of simpler features: are they simply present more often, or are they more influential? I wonder if there's a frequency-complexity relationship that would be insightful. - Did you do any of the same analyses with the individual neurons in the penultimate layer (i.e. comparing local vs distributed)?

Rating

7

Confidence

4

Soundness

4

Presentation

2

Contribution

3

Limitations

There is a reasonable discussion of limitations in the Appendices.

Summary

The paper introduces a metric based on V-information to measure feature complexity in deep learning models. Using ResNet50 trained on ImageNet, it explores feature spectrum, training dynamics, network flow, and decision impact. The study also highlights the role of simplicity bias and the evolution of feature importance in neural networks.

Strengths

1) Clear and well defined objective 2) Informative Figures 3) Comprehensive supplementary material

Weaknesses

1) Not Easy To Follow On Math: The paper's mathematical sections are difficult to follow. Clearer and intuitive explanations would improve accessibility. 2) Limited Model Diversity: The study focuses solely on ResNet50, ignoring modern architectures like depth-wise separable CNNs.

Questions

1) The paper states that both $z$ and $D^*$ are positive due to the use of Non-Negative Matrix Factorization, which aligns with the nature of ReLUs. Could you elaborate on why negative values for $D^*$ wouldn't work, despite the positive nature of ReLU? It's not clear why allowing $z$ and $D^*$ to be negative would be problematic, especially considering that $(-z)(-D^*) = zD^*$. 2) In line 126, $z$ is defined as $z \in \mathbb{R}$. Given that $z$ is described as positive elsewhere, wouldn't $z \in \mathbb{R^+}$ be more precise? Or are you intentionally allowing negative values for $z$? 3) The nature of $z_1$, $z_2$, and $z_3$ in Figure 1 is unclear. If these represent the $i$-th element of $z$, they would be scalars, which doesn't seem to align with what's depicted. Could you clarify what these variables represent? 4) Figure 1 shows feature visualization across different layers, and the paper mentions earlier layers elsewhere. However, line 622 states that feature extraction was done only for the penultimate layer. Could you explain this and clarify how earlier layer information was obtained if extraction was limited to the penultimate layer?

Rating

6

Confidence

1

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have addressed limitations in the appendix.

Summary

The paper introduces a novel metric for quantifying feature complexity in deep learning models, specifically focusing on an ImageNet-trained ResNet50 model. This V-information-based metric captures whether a feature requires complex computational transformations for extraction. The study addresses four key questions: (1) The appearance of features as a function of complexity. (2) The learning timeline of these features during training. (3) The flow of simple and complex features within the network. (4) The relationship between feature complexity and their importance in the model's decision-making. The study reveals that simpler features dominate early in training and are transported through the network via residual connections, while more complex features emerge gradually and require more computational effort. Interestingly, the most important features tend to be simpler and become accessible earlier in the network, suggesting a sedimentation process.

Strengths

(1) Originality: The introduction of the V-information metric for assessing feature complexity is novel and provides a fresh perspective on understanding neural network behavior. The exploration of feature complexity across training epochs and network layers adds significant depth to existing knowledge. (2) Clarity: The paper is well-organized, clearly presenting its goals, methods, and findings. Visualizations and qualitative analyses effectively illustrate the differences between simple and complex features. (3) Significance: Understanding feature complexity and its impact on model performance and decision-making is crucial for developing more interpretable and efficient models. This work contributes to explainable AI by providing insights into how features are learned and utilized.

Weaknesses

(1) Generalizability: While the findings are insightful, they are based on a single architecture (ResNet50) and dataset (ImageNet). The results might not generalize to other models or tasks without further validation. (2) Complexity Metric: The assumption that each layer optimally represents features for downstream linear probes may not hold universally, potentially leading to overestimated complexity scores in some cases. More empirical validation across different models and tasks could strengthen the proposed metric's robustness. (3) Temporal Analysis: The focus on specific epochs (e.g., epoch 90) may overlook important dynamics occurring at intermediate stages of training. A more granular analysis could provide a clearer picture of feature evolution, i.e., Can you analyze the changes in importance during the training process？

Questions

(1) How would the proposed V-information complexity metric perform on different neural network architectures, such as Transformers or GANs? (2) Can the authors provide more details on how the complexity metric correlates with other existing complexity measures in literature? Which one provides better insights? (3) How does the model's simplicity bias, as observed in the study, impact its generalization performance on unseen data? (4) Are there specific strategies that could be employed to mitigate the simplicity bias and encourage the learning of more complex, yet important features? (5) Does the conclusion of section 4 still hold on networks outside ResNet? How to explain the architecture of the transformer regarding the residual connection? (6) Why does the horizontal axis in Figure 4 show several blocks instead of all blocks?

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

While the paper provides significant insights into feature complexity in deep learning models, there are several limitations that need to be addressed: Model and Dataset Scope: The study focuses solely on a ResNet50 model trained on the ImageNet dataset. This raises concerns about the generalizability of the findings to other neural network architectures (e.g., Transformers, GANs) and different types of datasets. Future work should validate the proposed metric across a variety of models and tasks to ensure broader applicability. Assumption on Optimal Representation: The complexity metric relies on the assumption that each layer in the network provides an optimal representation for downstream linear probes. However, this assumption may not always hold true, potentially leading to overestimated complexity scores. Further empirical validation and adjustments to the metric may be necessary to account for cases where this assumption is violated.

Summary

This paper proposes a new measure of feature complexity based on an information-theoretic metric, the $\nu$-information metric. Utilizing this complexity measure, the paper shows (1) visualization of features of different complexities, (2) simple features are propagated through the residual connections to reach the final layer, (3) simple features are learned earlier than complex features during training, and (4) simple features usually have a higher importance (weight) to the output score of the model.

Strengths

1. The paper is well presented and is quite easy to follow. I appreciate the summary of different experiments into the words “what” “where” and ”when.” 2. The paper provides abundant visualization to help readers grasp the intuitive behind simple features and complex features.

Weaknesses

My major concerns for this paper are its coherence and significance. 1. The paper lacks coherence because multiple components (e.g., metrics, algorithms) introduced in this paper do not come from the same theoretical framework. For example, the notion of feature complexity in this paper (the $\nu$-information metric) is drawn from information theory, while the method for extracting features (the Craft method) is based on non-negative matrix factorization and has little connection with information theory. The same issue applies to the feature visualization method for demonstrating features of different complexities, and CKA metric, and the importance metric $\Gamma(z_i)$. These metrics/algorithms are borrowed from different theoretical frameworks and contexts, so it is in doubt whether they can be used together. I would appreciate it if the authors are able to re-organize all components of the paper under a coherent framework, e.g., the information theoretic perspective (if at all possible). 2. My second concern is about the paper’s significance, since many claims have been discussed in previous works and appears non-novel. The conclusion that simple features are usually color/edge detectors that are located in earlier layers is not surprising, nor do the conjecture that simple features tend to propagate through the residual connection. For the conclusion that simple features are learned earlier than complex features, there are many studies leading to this conclusion, from the perspective of spectral bias [cite1], game-theoretic interactions [cite2], or frequency [cite3]. Overall speaking, I do not learn new insights from the paper. Nevertheless, as mentioned in the 1st point in weaknesses, if the authors are able to re-organize the paper from the purely information-theoretic perspective, it will be a more intriguing work. 3. Using K-means clustering to aggregate features into meta-features may lead to incorrect clustering results. K-means clustering is known for its poor performance for data clusters that are not circular-shaped and are of discrepant sizes. I am not sure how features are distributed in this paper’s experiments, but more advanced clustering strategies such as the spectral clustering or hierarchical clustering can be applied to mitigate this issue. [cite1] Rahaman et al. On the Spectral Bias of Neural Networks. ICML, 2019. [cite2] Liu et al. Towards the Difficulty for a Deep Neural Network to Learn Concepts of Different Complexities. NeurIPS, 2023. [cite3] Xu et al. Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks. ICLR, 2020.

Questions

1. The paper claims that simple features tend to propagate through the residual connection, while complex features tend to propagate through the main branch of the network. Then a natural question arises: how are simple and complex features propagate in CNNs without residual connections (e.g., AlexNet, VGG)? 2. About the visualization of simple and complex features in Appendix B. The feature *fences* is among the most simple features, while the features *whiskers* and *dotted texture* are among the most complex features. However, I do not see an essential difference between the features *fences*, *whiskers*, and *dotted texture* from the sample images in Figure 7 and Figure 8. For example, *fences* and *dotted texture* both seem to consist of lines that cross over each other to form holes. Similarly, the *whiskers* feature is also composed of lines, although the lines do not cross over each other but appear parallel. Why is *fences* a simple feature, but *whiskers* and *dotted texture* complex features? Another question is that previous studies show that CNNs are usually biased towards textures, i.e., they tend to learn textures as a shortcut solution, but why is the *dotted texture* here measured to be a complex feature?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

1

Limitations

NA