On the impact of activation and normalization in obtaining isometric embeddings at initialization
In this paper, we explore the structure of the penultimate Gram matrix in deep neural networks, which contains the pairwise inner products of outputs corresponding to a batch of inputs. In several architectures it has been observed that this Gram matrix becomes degenerate with depth at initialization, which dramatically slows training. Normalization layers, such as batch or layer normalization, play a pivotal role in preventing the rank collapse issue. Despite promising advances, the existing theoretical results do not extend to layer normalization, which is widely used in transformers, and can not quantitatively characterize the role of non-linear activations. To bridge this gap, we prove that layer normalization, in conjunction with activation layers, biases the Gram matrix of a multilayer perceptron towards the identity matrix at an exponential rate with depth at initialization. We quantify this rate using the Hermite expansion of the activation function.
Paper
Similar papers
Peer review
Summary
A study of layernorm and Gramm matrix isometry is presented in both theory and emprical results. The results explain effectiveness of transformer (and similar) architechtures. Application of layernorm after activation seems to be a correct strategy when we are interested to isometry of output Gramm matrix.
Strengths
Good theoretical contributions in the field of explaining the effect of layernorm in deep neural networks via the concept of Gramm matrix isometry. Authors also added a large number of illustrative empirical results that help to drive home the main points of the paper.
Weaknesses
My main issue with the paper is not with the theory, but how it is presented in the main paper. The appendices appear to be fairly thorough mathematical exposition, but same rigorous exposition is absent in the main paper. Main paper should be readable without referencing to the appendices. Here under, I itemize places where I feel more rigour in exposition would be needed: From Theorem 1: "in isometry as a function of fluctuations in norms". What specifically is ment by "fluctuations in norms"? Isn't this statement obvious "corollary 2 shows that the isometry of layer normalization is deterministic and does not rely on the random weights." ? In the setup of Corollary 2, no weight matrix is applied. So how this question even would be raised? Completely different issue is that if you would compare I(G_l) and I(G_{l+1}), there weight matrix and activation function are applied. Next sentence about empirical proof also is out of place in this context. In Section 3.1. what is ment by the sentence "which implies orthogonalization of these results."? Orthogonalization of what? It would be good to expand the properties of Def 3 a bit more. Now we only have that it is "non-linearity strength for activations". Paper has half a page space for extra content. I would appreciate a few paragraphs of extra content in this context. About Eq. (7), you mention that G_*^l is a sequence that approximates the G^l. From these paragraphs it is hard to see where the sequence is coming from.
Questions
Gap between theory (Theorem 4) and observed in Fig. 4 is quite big. As a future work, is there a clear path for obtaining tighter bound?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
2 fair
Contribution
4 excellent
Limitations
-
Summary
The paper investigates normalization and non-linear activation functions from the perspective of isometry.
Strengths
- This paper provides interesting analysis into normalization and non-linear activation from the perspective of isometry. - The paper has provided interesting discussion on future research.
Weaknesses
- Overall the paper looks a bit rushed or not polished. There are many typos for example, for example line 31 "about the the". - The paper may benefit more from slightly larger scale data. For example, using MNIST could be a good way to improve the paper without too much additional work.
Questions
- One question while reading this paper is, why do we want to preserve the distance and angles between the input data points after mapping them to the feature space? Suppose in image recognition, the desired representation would not preserve the distance and angles in the input data points? - I think the previous question leads to another question which is also discussed in the paper: why would we want to use isometry to measure and explain layers? Despite many interesting points in the paper, I would suggest the authors add more motivation or probably a few past literature to better justify the motivation of this interesting work.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
Here are some minor comments: 1. I would recommend the authors polish more the paper. For example, for the figures, I recommend the authors add more captions for readers to better follow and understand the message. 2. I would also suggest improving the presentation a bit. For example, leaving two plots in the last page (page 9) may not be a good idea for the readers.
Summary
The paper is a theoretical analysis in the mean-filed regime the of the second-to-last gram-matrix of MLP. It proves that the presence of layer normalization in conjunction with non-linear activation functions biases the input-output mapping at initialization towards an isometry.
Strengths
This paper presents a novel theoretical contribution that aims to enhance our understanding of the interaction between layer normalization and non-linear activation functions at initialization. The proof relies on general assumptions and provides a justification for the utilization of pre-layer normalized representations in order to prevent training instabilities associated with the collapse of the rank of the input-output gram matrix in a multi-layer perceptron (MLP) network.
Weaknesses
The experimental tests are not well described, for instance the authors do not specify the datasets used neither in the main text and nor in the captions (see also questions sections).
Questions
1. What kind of input datasets are used in Fig. 2-3-4 ? 2. Why the unit-sphere in d dimension is denoted as $\sqrt d$-sphere and not $d$-sphere? 3. Minor: typo l 146: comma in place of full stop.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Not really. Perhaps the main limitation is that the author limit their analysis on MLP architectures. This makes the results of their paper not immediately applicable to transformers (due to the presence of the attention block) which are of much greater practical interest. Maybe this should be stated as a limitation of the current work.
Summary
This paper studies the isometry of Gram matrix under the effect of BN, LN and activation at initialization. For BN and LN, some results are obtained. For activation, most are empirical results.
Strengths
Study multiple factors that affect the isometry of Gram matrix.
Weaknesses
1) The novelty of the theory for BN and LN is small. 2) For activation, the conclusion is wekk. It is not strange to see different activation function have different effects. But so what and why? 3) The paper does not justify why study isometry. Its connection to neural networks, such as training speed or generalization, should be demonstrated either empirically or theoretically.
Questions
See weekness.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
See weekness.
Summary
This paper studies the isometric properties of a randomly initialized neural network. The authors show layer normalization and proper activation functions can mitigate rank collapse. In addition, they also quantify the normalization bias for different type of normalization layers. They use the Hermite expansion of the activation function to highlight the importance of higher order Hermite coefficients in the bias towards isometry. The paper provides theoretical results and empirical evidence to support its findings and discusses the potential implications for future research on neural network architectures and training algorithms.
Strengths
(+) This paper provides analysis of the isometry properties using the penultimate Gram matrix in neural networks. It complements previous study by analyzing the role of layer normalization (while previous work mainly focus on batch norm). It also proposes a few useful measurement of isometry bias. (+) The authors provide theoretical results and empirical evidence (mainly figure 2 and figure 3) to show that activation and normalization techniques can bias the Gram matrix towards isometry at initialization, which can improve the training dynamics of deep neural networks. (+) The paper discusses the potential implications of the findings for future research on neural network architectures and training algorithms, which can inspire new directions for improving the performance and efficiency of deep learning systems.
Weaknesses
(-) I feel this paper is not very coherent between different sections, though possibly because I’m not an expert in this field. I can understand each section but does not find the connection between different sections. (-) The empirical evidence presented in the paper is quite limited. Only very few activation functions are considered and it also assume the MLPs are of fixed width. The conclusion might be hard to generalize or be helpful to the practice. (-) The analysis of section 3 and section 4 seems to be two different systems. It does not explain how they are connected and lack a unified theory of them. (-) There are also a few confusing statements such as the authors show higher He coefficients have negative impact on isometry properties. However, in the abstract, they also state “ highlighting the importance of higher order (>2) Hermite coefficients in the bias towards isometry“, which implies higher order Hermite coefficients help isometry. Overall, I think this paper has limited applicability and does not inspire new ways on network design (either normalization or activation functions).
Questions
I have some questions regarding the coherence of this paper, for example: 1) what is the motivation of introducing normalization bias in Section 3.2, how does it related to the isometry bias? 2) Is definition 1 the definition for ‘isometry’, or a measure of ‘isometry property’? 3) Is the isometry gap (-logI(M)) ranges from 0 to -\infinity? (It is \infinity on line 100). I’m also curious how can the insights from this paper be used to design more efficient and effective deep learning systems? What is mainly the assumptions you make that present challenges of applying these insights in practice?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
This paper is unlikely to have potential negative societal impact.
I thank the author for providing this thorough rebuttal. The additional experiments on larger datasets seem very interesting and promising. Some concern left (which I think is out of the scope of the paper) is that does faster convergence (in the form of decreasing loss) means that the network is trained better. Previous literature and experiments have also demonstrated that faster drop in loss could lead to overfitting on the training dataset. This could be alleviated by reporting the testing accuracy of experiments in Figure 1 and Figure 2 of the additional material. The biggest concern is still that the paper, in the current form, does not convery the message too well. There needs to be quite substantial amount of work done on polishing the paper (e.g. adding motivation like the author wrote in the rebuttal and changing formats and wording of the work). I can only raise my score to borderline accept and I hope the author tries harder to polish this interesting work in the later versions.
> The additional experiments on larger datasets seem very interesting and promising. We sincerely thank the reviewer increasing their score based on our rebuttal response and appreciating the experiments > Previous literature and experiments have also demonstrated that faster drop in loss could lead to overfitting on the training dataset. This is an excellent point raised by the reviewer we will subsequently add the test loss plots to the camera ready version. > The biggest concern is still that the paper, in the current form, does not convery the message too well. There needs to be quite substantial amount of work done on polishing the paper (e.g. adding motivation like the author wrote in the rebuttal and changing formats and wording of the work). I can only raise my score to borderline accept and I hope the author tries harder to polish this interesting work in the later versions. We would like to point that our main contribution is the theoretical results about isometry under normalization and activation layers. None of the rebuttal responses imply any major changes to these parts (except for small typos and additional details in captions). The rebuttal response which was prepared in response to reviewers, which will be added as an additional page. This is accepted under formatting instructions for NeurIPS, as quoted below: > If your submission is accepted, you will be allowed an additional content page for the camera-ready version. Please let us know if there are any further questions or points that need clarification.
Thank you for your detailed reply. My question about 1) inconsistency of the effect He coefficients in experiments and abstract; 2) MLP of fixed width; 3) limited activation functions are addressed. I have two follow-up questions: Can you clarify the relationship between isometry gap, normalization bias, isometry bias, $-\log(I)$ ? In my understanding: isometry bias including isometry bias of normalization and isometry bias of activation. Is this correct? I have this question because figure 2 and 3 both show Isometry bias of normalization while the y-axis is "normalization bias" and "$-log(I)$" separately. What is the difference between the two figures? Another question is about identity activation function. I thought identity function would preserve the isometry property (it preserve the distance), but this seems to be contradict with Figure 3. Is this because identity actiavtion is not helpful on recovering the isometry gap (although it does not make it worse) while other activations are?
We highly appreciate the reviewer's response to our rebutal response. > Another question is about identity activation function. I thought identity function would preserve the isometry property (it preserve the distance), but this seems to be contradict with Figure 3. Is this because identity actiavtion is not helpful on recovering the isometry gap (although it does not make it worse) while other activations are? Yes the reviewer is correct. We thank the reviewer for making this nuanced observation. Please note that there is a distinction between preserving isometry (which is what identity achieves), and improving isometry (which is what non-linear activations achieve). We can see this in Theorem 4, equation (5), that isometry gap decays exponentially with depth $\ell$ with rate $\exp(-\ell \log\beta_0)$. Thus, for any activation with non-linear components $\beta_0 > 1,$ we have $\log\beta_0 > 0,$ while identity activation we have $\log\beta_0 = 0$, which affirms your observation. > Can you clarify the relationship between isometry gap, normalization bias, isometry bias, ? In my understanding: isometry bias including isometry bias of normalization and isometry bias of activation. Is this correct? The reviewer is correct in assuming that various components of neural networks such as normalization and activation layers, influence isometry. Let us clarify the definition of these terms and where they are defined in the main text: | Term | Definition| Where defined | Range | | -------- | -------- | -------- | ----- | | Isometry | $\mathcal{I} = \frac{G.M.(eigs)}{A.M.(eigs)}$ | Table 1 | $[0,1]$ | | Isometry gap | $-\log\mathcal{I}$ | Table 1 | $[0,\infty]$| | Normalization bias | $\frac{Var(\text{norms)}}{\text{mean}(\text{norms})^2}$ | equation 4 & 5 | $[0,\infty]$ | - Isometry bias of normalization: According to Theorem 1, a larger normalization bias causes a larger increase in isometry (or decrease in isometry gap) after passing through the normalization layer:o $$ \mathcal{I}(\text{post-normalization Gram}) \ge \mathcal{I}(\text{pre-normalization Gram})(1+\text{normalization bias}) $$ - Isometry bias of activation: According to theorem 4, isometry gap of MLP with activation non-linearity strength $\beta_0$ decays exponentially with rate $1/\beta_0$: $$ -\log\mathcal{I}(\text{Gram matrix of layer $\ell$}) \lesssim \exp(-\ell \log\beta_0) = (1/\beta_0)^\ell $$ > I have this question because figure 2 and 3 both show Isometry bias of normalization while the y-axis is "normalization bias" and "$-\log \mathcal{I}$" separately. What is the difference between the two figures? Figure 2 is plotting normalization bias, as defined in equation (4) and (5), but, in Figure 3 the y-axis is showing the isomtery gap, as defined in Table 1. > Overall, I think this paper has limited applicability and does not inspire new ways on network design ... We wonder if the reviewer finds our responses regarding applicability convincing or not?
Thank you for your reply. I think all the previous discussions (as well as rebuttal to other reviewers) make the paper much more clear. These should be incorporated in the revision. I think the applicability increases compared to the initial version. I'm still skeptical (but not negative) on its impact to practical network design but this is not a decision factor since I understand the focus is on theory and practical verfications is beyond the scope. I'm willing to increase my score to weak accept.
> I'm willing to increase my score to weak accept. We sincerely thank the reviewer for increasing their score, as well as for taking the time to give us constructive feedback. > I think all the previous discussions (as well as rebuttal to other reviewers) make the paper much more clear. These should be incorporated in the revision. I think the applicability increases compared to the initial version. We wholeheartedly agree with the reviewer that the review process dramatically improved the practical implications of our theoretical results. We will make sure to incorporate them in the final version.
I thank the authors for their response. They have addressed my concern regarding the experimental test on transformer architecture. I recommend that they include this explanation in the appendix if the paper will be accepted. I still have a small question about the notation used: I know that $\sqrt d$ is the radius of the sphere where data is projected after LN. My question was about how the authors denote this sphere. From what I know, and as shown in [1] and [2], the $d$-dimensional sphere is usually called the $d$-sphere. [1] https://en.wikipedia.org/wiki/N-sphere [2] https://mathworld.wolfram.com/Sphere.html
> the $d$-dimensional sphere is usually called the $d$-sphere. We thank the reviewer for bringing this nuanced notation issue to our attention. Thus we will replace these mentions by "$d$-sphere with radius $\sqrt{d}$" > They have addressed my concern regarding the experimental test on transformer architecture. We will make sure to include it in the appendix in the revised manuscript. We again thank the reviewer for this valuable and interesting suggestion. In light of these additional experiments, should you find it fitting, we would be grateful for any potential reconsideration of our score.
Thank you for the answer, I appreciate your prompt response. I will maintain my current score, which I think already fits the current value of the paper.
We sincerely thank the reviewer for their valuable and constructive feedbacks and appreciate their positive evaluation of our work.
Thank you for the rebuttal
Dear authors, thank you for providing a rebuttal. Some of the reviewers have already replied, so this is just to let you know that I am in contact with the remaining ones as well. Best, Your AC
We thank the reviewer for increasing their score. We would highly appreciate if the reviewer specified which parts of our rebuttal response or their concerns remain unresolved?
> I am quite satisfied with these explanations and I do not have any other issue. We greatly appreciate the reviewer for their positive evaluation of our work. We also sincerely thank the reviewer for their time and providing us with their insightful feedback.
Decision
Accept (poster)