The Tunnel Effect: Building Data Representations in Deep Neural Networks

Summary

The paper offers an empirical study of deep neural networks. The focus is on the role of intermediate layers in building a representation that is linearly separable and can eventually solve the task. The work highlights the fact that this linearly separable, low-rank representation emerges at a depth that is a fraction of the total depth (usually 1/3 to 2/3 for common image classification setups). Layers before such point are named "the extractor", and layers afterward "the tunnel". The authors further investigate the role of "the tunnel effect" on transfer learning and catastrophic forgetting.

Strengths

The main strength of the paper is the thoroughness of the empirical study. Although only pertaining to image classification (as mentioned by the authors), the study includes a wide range of benchmark datasets and architectures, and shows results with plots that are easy to parse. Connecting the tunneling effect to practical recommendations for transfer learning is another interesting point.

Weaknesses

Although the authors put a strong emphasis on the novelty of their results, the fact that a linearly separable representation emerges well before the final layer is not completely novel. See e.g. [1] for an analysis using similar methods. Also, the reasons behind the so-called paradox that motivates the work are not completely clear. It's not evident how the fact that capacity increases with depth, is at stake with the fact that there is an "inductive bias" toward simple solutions. In my view, the motivation paragraph in the introduction would need to be clarified, and the conclusion should clearly state how this paradox has been addressed by the paper. [1] Feature learning in deep classifiers through Intermediate Neural Collapse, Rangamani, Akshay; Lindegaard, Marius; Galanti, Tomer; Poggio, Tomaso.

Questions

- Section 3.1 put emphasis on the fact that the tunneling effect emerges "early" during training. How do you assess what early is? Is the effect happening before/same time/after train and test losses start decreasing? It would be nice to have a comparison of these different timescales to better assess the claim of the section. - In the introduction, the authors claim to challenge the idea that deeper layer capture more complex and task-specific features. What are the results that question this idea? For example, the observation that the transfer learning performance deteriorates after neural collapse (i.e. in the tunnel part) is consistent with the idea that the representation in these layers becomes task-specific. The fact that more complex tasks require a longer "extractor" is consistent with the idea that more complex tasks require more complex representations to be built and more complex representations are built in a hierarchical manner through more layers. - I find the results of Figure 9 exemplificative of the fact that the network uses always the first layers to solve the task, independently of its total depth. It would be nice to have the same Figure for a CNN. - Minor: What is the difference between blue and orange dots in Figure 10 (top).

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors clearly highlight that their work focuses only on image classification and that a study for other data modalities would be interesting. I think this is a fair limitation and does not impact my evaluation of their work. Also, at first sight, the role of skip connection would be very relevant for the effect studied in this work, and possibly it would deserve to be discussed more in the main text.

Summary

The tunnel effect is described for deep overparameterized networks, whereby early layers form a linearly separable representation while later layers form a "tunnel" which passes this representation to the output without substantial change, other than compression (reducing its rank, a.k.a. discarding information). A large number of experiments show that this effect occurs in a variety of different models, datasets, and training times, and impacts out-of-distribution and continual learning settings. The effect implies that the capacity of a given architecture/dataset is fixed.

Strengths

This paper provides a clear and comprehensive argument for both the existence of its hypothesis and its effects on several relevant subdomains of DNN research. The experiments are thorough, well organized, and address many potential concerns (such as consistency of the observations across different models and datasets). Some subtle details are handled correctly (e.g. use of unbiased CKA estimator in appendix E). The effect is convincing and could point to significant improvements in how transfer and continual learning is handled.

Weaknesses

The procedure for computing the numerical rank should be given fully (e.g. the value of the threshold $\sigma$ is not explicitly stated). In particular, figure 6 indicates some issues (see Questions below). Also, figure 6 is missing shading for the tunnel region after training, which would help readers reference other figures. The difference in weights in figure 5 is not fully explained (e.g. $\tau$ should be defined). Also, while the difference is normalized relative to $\theta^0$ (the initial weight norm?, or is it the number of weights?), a better comparison would be relative to the norm of the mean change iterations/epochs, since the learning rate changes over training. As a result, the experiment cannot rule out the possibility that the tunnel is changing relative to other layers after the learning rate is reduced. Overall the evidence from CKA is lacking (figure 4) as it is only shown for a MLP. However, this is not a central piece of evidence.

Questions

In figure 6, why is the numerical rank for higher layers already less than that of lower layers at initialization? Given that random matrices tend to be near full-rank, this points to an issue with how numerical rank is computed (e.g. the threshold for determining rank should be layer-dependent, since different layers have different weight scales).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

ImageNet-scale networks and Transformer architectures are out of scope of this work. But given the importance of such networks to transfer learning applications, the obvious next step is to investigate the tunnel effect in these networks. Unlike the CIFAR/CINIC-scale models in this work, it is unclear what degree of overparameterization is occurring in ImageNet-scale networks and Transformers.

Summary

This paper proposes the Tunnel Hypothesis: Training of deep networks splits layers into two distinct phases: (1) extractor phase and (2) tunnel phase. Extractor phase learns the linearly separable features whereas the tunnel phase compresses the representations. The authors provide evidence towards degrading effects of this tunnel effect on OOD (out-of-distribution) samples. Further, better understanding of continual learning may be possible due to the proposed hypothesis.

Strengths

The paper has following strengths: 1. The OOD representation section was very interesting. 2. Applications to continual learning based on knowledge gained from the detailed analysis from this paper can also be useful. 3. Overall, a lot of work has gone into this paper (many experiments).

Weaknesses

The paper has following weaknesses: 1. Many of the observations are not particularly surprising. I think a lot depends on capacity of the network and the difficulty of the task at hand. It is not surprising that for a given task and a type of network, representations get learned up to a certain layer and then the remaining layers simply make the representations more compact. Indeed, if the task becomes more complex or more difficult, the tunnel length would reduce (since more layers would be spent trying to learn more complex features). This is clear from Table 1 where for ResNet-34, the tunnel length significantly reduces when going from CIFAR-10 to CIFAR-100 (from 20 to 29 layers). Thus, when the task became more difficult, more layers started getting used to learn better features. Similarly, for simple MLPs, we know that beyond a certain depth, depth does not help (this comes from many other studies, e.g., that analyze gradient properties, etc.: beyond a certain depth and without skip connections, adding more layers does not help due to vanishing gradients). The insight that “many later layers do not contribute significantly to accuracy” is also known and is precisely why “deep network pruning” literature is not able to prune later layers too significantly. 2. In the introduction section, the authors claim that “they challenge the commonly held intuition that deeper layers are responsible for capturing more complex and task-specific features”. I do not see any evidence that they changed this commonly held view. In fact, many of their experiments reinforce exactly the common viewpoint. Specifically, the authors show that later layers hurt the OOD performance. This indicates that the later layers got specialized towards the within-distribution task which is why they hurt the OOD task. Hence, if the commonly held view is being reinforced with the evidence provided by the authors, there is nothing particularly surprising about the findings. 3. I think the proposed work can have significant value in the field of continual learning and also multi-task/multi-modal learning if the observed insights can be used to guide novel architectural designs and/or loss functions. Unfortunately, the current work (despite a lot of hard work) only plays around with toy datasets in that problem space. If the authors can build further on the new insights and create new models/losses for the aforementioned areas, this can be an impactful work.

Questions

Please see above.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

1 poor

Limitations

Yes.

Summary

This paper shows an effect of deep neural networks when trained for classification tasks — the initial layers create linearly separable features, and the later layers collapse the features for the final prediction. This phenomenon is explored with extensive experiments.

Strengths

- The paper explored a very interesting phenomenon of how the features are learned dynamically through layers. - The paper performed extensive experiments to show how this "tunnel effect" affects the model performance under different settings. - The experiments are well-designed, and the results are demonstrated well.

Weaknesses

- This is not the first/only paper that discovered some similar effects under a similar setting, therefore, a more comprehensive comparison with them in the related work section as well as a clarification on the contribution should be added. e.g. [1] examines the feature "intrinsic dimension", [2] analyzes the generalization effects of feature neural collapse on in-domain and out-of-domain data. - The authors mentioned the network can be split into the extractor and the tunnel which compress the features, however, it can be difficult to systematically split the network, it seems the author also did not provide a systematically split of the network based on the numerical rank. [1] Ansuini et. al. Intrinsic dimension of data representations in deep neural networks. [2] Li et. al. Principled and Efficient Transfer Learning of Deep Models via Neural Collapse.

Questions

- Can the tunnel effect, or the numerical rank provide practical guidance on which feature we should use for different problems, e.g. out-of-distribution data, continual learning? - Following up on the previous question, if this tunnel effect can provide guidance, how can we use this tunnal effect while model training and/or model inference.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations of the paper is relatively thoroughly discussed.

Thanks for the response

I thank the authors for the response. As I said in the original review, I do find their OOD related findings interesting. Sorry for not providing concrete references in my review. Here are four references that may have made me think that the observations were not too surprising: 1. Pruning literature [Neurips 2019]: https://arxiv.org/pdf/1906.02773.pdf. See Fig. 1 and description in section 3.1 (global pruning paragraph) in this paper. It says that later layers can be pruned more. Hence, I thought that it was known knowledge that later layers have more redundant knowledge that can be removed. It is also known that if a network is too compact, it is hard to prune such networks (as seen by practical difficulty in pruning MobileNets, EfficientNets, etc.). Thus, if the task complexity is too much for the model at hand, many layers would play important role (and thus the tunnel would reduce). The tunnel hypothesis is essentially stating the same thing -- that the later layers do not contribute much new information if a task is relatively simple (thus a longer tunnel). If a task becomes more difficult, the tunnel reduces. 2. Deep Equilibrium Models (DEQ) [Neurips 2019]: https://arxiv.org/pdf/1909.01377.pdf. The whole motivation behind these networks was that after a certain layer, representation does not change much (i.e., they approach equilibrium). This can again be seen as an evidence towards representations not changing much after a certain number of layers. Please also check some other references on page 1 of the original DEQ paper above (refs [18,8,15]). 3. Feature visualization kinds of research [Distill 2017]: https://distill.pub/2017/feature-visualization. These are just some papers that talk in detail about how representations form (but from visualization perspective). This one may not have too much insight but it shows empirically how representations form. 4. How information content changes through the training: https://arxiv.org/pdf/1703.00810.pdf. This formally looks into how representations compress over the course of training using mutual information measures, etc. There was a lot of debate around this work, so I encourage authors to look at other follow up works also (e.g., ICLR 2018 https://openreview.net/forum?id=ry_WPG-A-). On the "more complex tasks requiring shorter tunnels" section above: I was only saying that given that features form slowly through the network layers (edges to complex shapes, as shown in the above Distill 2017 visualization paper), I imagined that for a more complex task, the network would spend more layers trying to learn the representations (and thus the tunnel would be shorter). And if the task is easy, several layers would learn redundant info (and, hence, layer output would not change between such layers...which is clear from your CKA analysis). Moreover, given the pruning literature, I always assumed this redundancy was happening towards the end. Maybe it is just me...but these are some of the reasons I was not surprised. If this paper had more theoretical results, that would make the case much stronger. For now, I will keep my rating. I will leave it up to other reviewers and AC in case I genuinely missed something.