Summary
This paper investigates the "tunnel effect" in NNs introduced in a NeurIPS 2023 paper. According to the “tunnel effect” the deeper layers compress representations, limiting OOD generalization. The authors challenge the prior assumption that the tunnel effect is universal. They imply that it's heavily influenced by training data diversity.
They analyze a large number of NNs on various datasets with varying resolutions and introduce three metrics to quantify the tunnel effect's strength. Using a SHAP-based approach, they disentangle the impact of augmentation, number of classes, sample size, resolution, and the architecture.
Their experiments suggest that increasing the diversity of the training data (more classes, augmentations, and higher image resolutions) mitigates the tunnel effect and improves OOD generalization. Depth and overparameterization have the opposite effect.
Based on these findings, they propose a revised tunnel effect hypothesis where its strength is inversely proportional to training data diversity.
Strengths
Originality: This paper is original, in the sense that it challenges existing assumptions on the universality of the tunnel effect hypothesis, which might have implications on understanding and improving NN’s OOD generalization ability. The authors propose a revised tunnel effect hypothesis which is a novel contribution.
Quality: This paper is scientifically well-executed. The hypothesis is clear and the experiments are extensive (several NN architectures and dataset (configurations), which greatly helps to support their hypothesis. The SHAP-based analysis they performed to disentangle the relative importance of different variables is solid (through rigorous statistical analysis with Wilcoxon signed-rank tests and Cliff's Delta to measure effect sizes) and provides a quantitative measure of the effects.
Clarity: The paper is very well written and easy to follow. Figures and tables are very helpful to understand the key findings.
Significance: The paper’s findings may have significant implications for the “tunnel effect” hypothesis, though I am a bit skeptical of the new insights this brings us in terms of designing algorithms that have better OOD generalization properties.
Weaknesses
Although the paper is a very solid work, the main worry that I have is about the confounding variables in the experiments and the revised hypothesis, which might limit the significance of the contribution:
* For example, Figure 1 claims that the increased resolution mitigates the tunnel effect. But intuitively, if you increase the resolution of the image, you also increase the resolution of the activation maps in the bottleneck layers. Still this doesn’t tell us anything about the architectures per se. Yes, the tunnel effect can be “sidestepped” if you increase the resolution, but then you run into the same issues if you increase the depth.
* OOD accuracy is measured on a number of datasets (NINCO [61], ImageNet-R [62], CIFAR-100 [60], CIFAR-10 [59], Oxford 102 Flowers [63], CUB- 173 200 [64], Aircrafts [65], Oxford-IIIT Pets [66], and STL-10 [67]). However, I wonder what such experiments tell us about the “kind” of OOD that we are testing for. For example, if one tests for compositional or systematic OOD generalization, it becomes very clear what the learnings are, and which particular flaws the models have. I feel that on these benchmarks it is hard to make the similar conclusions.
* Certain findings I find limited in the sense that they are already known, e.g. “Our metrics reveal that increasing the ID class count, higher resolutions, and using augmentations improve OOD generalization”.
* In these figures/results, typically normalized accuracy or performance retained are discussed. However, in ML it is also important to understand the actual accuracy, and these are not easy to find in the paper. Ideally for each figure/results discussion, the reader would be pointed to a table (in the appendix) that contains absolute numbers on the datasets. For example, it is claimed that "overparametrization negatively impacts OOD generalization .. since increasing overparametrization decreases OOD performance **retained**".
However I feel that without the absolute numbers this claim cannot be made. E.g. if both the IID and OOD performance improved by increasing overparametrization, but the IID performance improved more than OOD performance, one cannot claim that OOD performance decreased, quite contrary.
* L281: “Depth. Our SHAP analysis revealed that increasing depth impairs OOD generalization.” - Depending on how you increase depth (e.g. if you have pooling layers), to me this is an obvious consequence of additional compression that you may have in that case.
* The paper claims that their continual learning results contradict [9] and suggest the “tunnel” plays an essential role in mitigating catastrophic forgetting. However, they only replicate the experiment with ResNet-18 on ImageNet-100. To strengthen this claim, the authors could expand the continual learning analysis to include different architectures and datasets.
Questions
1. This citation has the wrong year and NeurIPS number: “Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miło ́s, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, 2024.”.
Please cross-check other references too.
2. SHAP papers should be cited the first time it is mentioned in the text.
3. Could you clarify a bit further the reasoning / logic behind the “Metric 3: ID/OOD Alignment”? I find it a bit hard to wrap my head around it in its current form. Specifically why you choose to have a product of the two values and the respective “baseline” values that are subtracted.
4. What do you mean by “more hierarchical features” in L136? Isn’t the tunnel effect reduced because the size of the activation maps is increased, so the bottleneck is decreased?
5. L275: Perhaps a more appropriate wording here would be "7x7 kernel size" instead of "7x7 stem"?
Limitations
The limitations and societal impacts have been addressed.