Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse

Summary

The paper provides a comprehensive understanding of gradient rank in deep neural networks (DNNs) and how architectural choices and data structure affect gradient rank bounds. The authors highlight the emergence of low-rank learning as an inherent aspect of certain DNN architectures and propose a theoretical analysis to provide bounds for training fully-connected, recurrent, and convolutional neural networks. They also demonstrate, both theoretically and empirically, how design choices such as activation function linearity, bottleneck layer introduction, convolutional stride, and sequence truncation influence these bounds. The study not only contributes to the understanding of learning dynamics in DNNs but also provides practical guidance for deep learning engineers to make informed design decisions. The authors also discuss the phenomenon of "Neural Collapse," where linear classifiers within DNNs converge to specific geometrical structures during late-stage training, and highlight the role of geometric constraints in learning beyond this terminal phase.

Strengths

This work theoretically analyzes the upper bound of the rank of gradients for different kinds of neural networks. The analysis can offer us great insight when we try to design new modules or activation functions. In addition, the authors provide a detailed practical analysis of the proposed methods.

Weaknesses

For the experiments for validation of the bottleneck, it would be better to add experiments on convolutional neural networks to make the validation more sufficient. For the analysis of the structure, it would be better to add analysis of some normalization techniques or regularization techniques.

Questions

Please refer to Weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

The authors present an investigative study into the learning dynamics of neural networks, specifically low-rank neural networks. The motivation is that training dynamics of neural networks are not fully understood. Low-rank models have practical advantages (time / memory). This work provides a theoretical and empirical overview of the learning dynamics of neural networks with a focus on low-rank models.

Strengths

The main strength of this paper is its significance, i.e. the attempt to explain learning dynamics of neural networks from both a theoretical and empirical perspective. This is a worthy and valuable endeavour which will be of great interest to the field if done correctly and thoroughly. Unfortunately, there are also significant weaknesses, as we will see, which weight heavily against the strengths.

Weaknesses

There are a number of critical weaknesses present in this paper that severely reduce its value. Perhaps there is a valuable message in there but the presentation, clarity and writing make it impossible to recommend this as an impactful paper. Here are some of the weaknesses detailed: - Flow and writing can be greatly improved. Very abrupt changes between sections (e.g. from 1 to 2). The sections do not have self-contained introductions to help readers orient themselves and understand the reasons and motivations for choices made. - Related to the first point, formulae are presented without sufficient discussion. Many sections (notably 2 and 3) read like a series of statements rather than a well-constructed and clearly motivated argument. I encourage the authors to improve the flow to help the reader understand the context - Formatting can be improved. What is *error*? Overview of results in lines 77-88 can be greatly improved with respect to formatting. - Missing definition, what is BPTT in line 104? - Lack of discussion on the connection between the presented theoretical and empirical results. - Stated contributions are not found in the paper (effect of stride is in supplementary material and I could not find sequence truncation experiments anywhere).

Questions

- Why are only leaky-ReLUs investigated, what is the motivation? And how does this differ from standard ReLUs? - What is the theoretical connection to bottleneck layers bounding the entire network? Why is this limited to bottleneck layers and not the narrowest part of the network? - What is the impact of training data on the learnt rank? Does the implicit dimension of the dataset influence the rank? With reference to [1] - Why is the PDF formatted as an image and not as a standard NeurIPS PDF output? The links to section labels and reference links do not work. [1] Li, C., Farkhoor, H., Liu, R., & Yosinski, J. (2018). Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

1 poor

Limitations

The authors discuss only one limitation of their work: the relationship between tasks and low-rank emergence. However, there are a number of limitations that need to be discussed. Not only have the experiments scratched the surface of the possible signals from learning dynamics we can collect and paint us a limited picture of what is happening but the results themselves also make us pose new questions that remain undiscussed. Refer to my questions to the authors for some possible discussion points that would improve this paper greatly and help place it within the existing literature and highlight the most obvious next steps to build on this work for future studies.

Summary

This paper studies the gradient rank of DNNs and examine how certain design choice, in particular architectural choices of the model and the structure of the data affect the gradient rank bounds. The paper mainly focuses on theoretical results with some empirical experiments to validate their empirical claims. The paper begins by studying the gradient rank of deep linear models before proceeding to explore leak-ReLU networks. They also extended their analysis to consider convolution and recurrent layers.

Strengths

The paper seems fairly novel, as I have not seen anyone explore the gradient rank although I think it is important as it can tell us more about the learning dynamics of neural networks. The theoretical analysis is simple, and this is advantageous in my opinion. One of the strengths of the paper is they have analysed a few neural network variants such as recurrent, convolutional and leaky relu networks. This demonstrates how general their analysis can be. The experiments are good and test well the bounds.

Weaknesses

it is not clear how to generalise the analysis to more complex non-linearities and can thus at most only seem to study them near a limited range of input values (such as near zero) where the network behaves more linearly. The analysis also doesn't look at more modern architectures like transformers.

Questions

DNNs are trained using gradient descent. It is possible to write the parameters of the network as a weighted sum of gradients, since we can bound the rank of the gradients with your analysis can we use that to bound the rank of some of the layers in a DNN.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors have adequately addressed and there is no clear potential negative societal impacts.

Summary

This paper explores the collapse of gradient rank in DNNs during training. Based on the simple linear network, the authors theoretically examinate the rough upper bound of the gradient rank for simple MLP, current network and CNN. Furthermore, they analyze the numerical effect of rank on Leaky-ReLU activation. To verify their results of the rank bounds, experiments are conducted with on two synthetic datasets and several real datasets (CIFAR10 and Tiny-ImageNet for computer vision and WiKiText for NLP) across pure linear networks and smaller-sized version of popular architectures, e.g., ResNet16, VGG, BERT, etc. Experimental results show that 1. Linear bottleneck layer architecture reduces the gradient rank; 2. the gradient rank of recurrent networks and CNNs is proportion to the size of sequence; and the negative slope of leaky-ReLU activation is related to the amount of gradient rank restored.

Strengths

The paper demonstrates a high level of originality in its exploration of gradient rank in linear network, recurrent networks and CNN and extra analysis of Leaky-ReLU effect.

Weaknesses

1. Poor writing quality: This paper has not been properly polished before submission, and there are numerous writing issues that have not been carefully checked. For instance, in line 132, the term "phi" should be "\phi". Additionally, in line 217, the notation "FIGURE1left" should be revised for clarity. In line 218, there are two issues: the terms "FIGURE1right" and "In figure" should be corrected ("in figure"?). Moreover, a missing "." in line 235 needs to be addressed. Another concern is the inconsistent section references throughout the paper, such as "sec 3.2.2" in line 226 and "&3.3.1" in line 236. Overall, this article resembles a draft rather than an official submission, and it requires further polishing and thorough review. 2. Unfollowed style: the bibliography style seems not to follow the provided template and please check your article format, because I cannot search and select your words in the main context and cannot jump to the referred figures. 3. Lack of practical network experiment: although the theoretical analysis is mainly conducted on the linear network and experiments are provided for 3-layer linear MLP and RNN network, more experiments and further analysis on popular networks are needed for providing insight for the understanding the gradient rank in practice. 4 Lack of citation and explanation for BPTT.

Questions

Because the paper is not well-written, most question is for clarification. 1. what does the "disjoint variable" mean in line 171? 2. what does the "original bound" mean in line 244? 3. what does the "restore" refer to in line 246-255?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

No potential negative societal impact of their work