Average gradient outer product as a mechanism for deep neural collapse

Deep Neural Collapse (DNC) refers to the surprisingly rigid structure of the data representations in the final layers of Deep Neural Networks (DNNs). Though the phenomenon has been measured in a variety of settings, its emergence is typically explained via data-agnostic approaches, such as the unconstrained features model. In this work, we introduce a data-dependent setting where DNC forms due to feature learning through the average gradient outer product (AGOP). The AGOP is defined with respect to a learned predictor and is equal to the uncentered covariance matrix of its input-output gradients averaged over the training dataset. The Deep Recursive Feature Machine (Deep RFM) is a method that constructs a neural network by iteratively mapping the data with the AGOP and applying an untrained random feature map. We demonstrate empirically that DNC occurs in Deep RFM across standard settings as a consequence of the projection with the AGOP matrix computed at each layer. Further, we theoretically explain DNC in Deep RFM in an asymptotic setting and as a result of kernel learning. We then provide evidence that this mechanism holds for neural networks more generally. In particular, we show that the right singular vectors and values of the weights can be responsible for the majority of within-class variability collapse for DNNs trained in the feature learning regime. As observed in recent work, this singular structure is highly correlated with that of the AGOP.

Paper

References (54)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer Ljpy5/10 · confidence 3/52024-07-08

Summary

Given the complexity of the process of neural network training, any understanding of robust phenomena that can be identified in the training process has potential value that can guide the design of models and algorithms. Neural Collapse (and its deep counterpart) is one such phenomenon that has been identified and reproduced across multiple model classes and datasets. This work shows that Neural Collapse also occurs for a recursive kernel-based model known as Deep RMF, when trained using an algorithm that is based on projection onto a matrix constructed from an outer products of gradients computed locally at each layer. Additionally, the authors present experimental results that document neural collapse in these models when trained on standard datasets. They also show that in standard neural networks, the projection of features onto the gradient outer product leads to neural collapse, rather than the effect of the nonlinearity.

Strengths

The paper is clearly written, and presents both theoretical results and some empirical results that complement them, since they apply to datasets that violate the assumptions under which the results hold. They prove that deep Neural Collapse can indeed occur in models beyond standard neural networks trained with gradient descent. The experimental results (specifically in Appendix D) demonstrate that the projection onto the gradient outer product matrix (AGOP) leads to neural collapse in standard models, motivating the further study of this object.

Weaknesses

Given that the main results apply both to a non-standard kernel method and a non-standard training algorithm, it is unclear what the implications of the results are for more well-known models and algorithms. If the authors believe that these results have implications of this form, they should be presented more clearly. Algorithms that are not based on backpropagation are interesting both as possible means of explaining learning in biological systems where backpropagation is unrealistic, and in distributed settings where backpropagation may incur a prohibitive communication overhead. However, the motivation of the algorithm used appears to be that it is a simple model that demonstrates certain phenomena that arise in the training of deep networks. The authors assume that the gram matrix of the data is full-rank. This requires assuming that the number of datapoints is smaller than or equal to the input dimension (which subsumes assumption 4.1). Standard datasets violate this assumption.

Questions

Are the models in appendix D trained with SGD? If so, they indicate the importance of the AGOP projection in causing neural collapse in standard models and I believe this result should be highlighted. That being said, this result may be of independent interest regardless of the analysis of Deep RMF.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations and societal impacts have been addressed

Reviewer uXDR6/10 · confidence 3/52024-07-10

Summary

This paper studies deep neural collapse (DNC) in deep neural networks (DNN) through the prism of the neural feature ansatz (NFA) and deep recursive feature machines (RFM). It is comprised of several results: - empirical evidence that DNC occurs in deep RFMs, - a theoretical analysis of DNC in a high-dimensional RFM setting, - a theoretical analysis of DNC in a kernel learning setting, - empirical evidence that the mechanisms which lead to DNC in RFMs and traditional DNNs are the same.

Strengths

This paper shows that deep neural collapse occurs in a similar way in deep networks and deep recursive feature machines. It thus provides a simplified setting in which to investigate deep neural collapse, which is an important research direction to further our understanding of deep learning. Specifically, it shows that neural collapse can be obtained just by iterating linear regression problems, without backpropagating through a deep network.

Weaknesses

My main issue with the paper is its writing, which makes it quite difficult to read. - The notations could be improved in several places throughout the paper (see minor points below). - I could not follow most of section 4.2, despite being rather familiar with kernel methods and their behavior in high dimensions. On a high level, I don't understand how a linear kernel could be the best setting for neural collapse. The text contradicts itself, as it simultaneously state that "if [$\lambda_k = 0$] [...], collapse will occur in just one layer] , but also that "this theory offers an explanation for why non-linear activation is needed". A linear layer can collapse within-class variability but also typically collapses class means together, and thus cannot induce neural collapse (see paragraph below). On a technical level, $k_\Phi$ and $\lambda_\Phi$ are referred to before being defined, and I do not understand the roles played by $k$/$\lambda_k$ vs $k_\Phi$/$\lambda_\Phi$. Assumption 4.2 is also referred to before being stated. - Section 4.3 is also slightly difficult to read. I took me several tries to guess that $k_M(x,x') = \tilde k_M(x,x')\mathrm{Id}$, which should appear in the paper. The terms "input-level" and "output-dimension level" kernels should be introduced for non-specialists in multi-task kernel learning. I also do not understand the point of introducing $M$ if it is dropped afterwards. Theorem 4.4 could simply be stated as "the optimal feature map for ridge regression is the one which already predicts the label: $\Phi(x) = y$". This result is not very surprising, and is not very integrated in the paper. I suppose that it is some kernel-level equivalent of the unstructured feature model, and suggests that weight decay might be instrumental in bringing about neural collapse? The normalization of $k$ should be restated in the definition of Problem 3 (otherwise the optimal loss is obtained when $k \to 0$). - The message of section 5 could be presented more clearly. What I understood was that it argues that RFMs and DNNs achieve neural collapse through the same means. I suggest making this point before introducing RFMs (in particular, stating the NFA correlations). I also did not understand why this mechanism is referred to as "denoising". My second issue is that I was not convinced by the claim that it is the right singular vectors and singular values which lead to neural collapse. By the same logic as lines 309-315, the right singular vectors do not change the DNC1 metric (with a "full" SVD where $U$ and $V$ are invertible). Similarly, if I were to divide operations in the network as $V^T\sigma$ and $US$ as opposed to $\sigma U$ and $SV^T$, I should see that it is now $US$ which is responsible for neural collapse (again with a full SVD). This conclusion also depends on the chosen metric for evaluating collapse. Why do the authors consider the ratios of traces of between- and within-class covariances, rather than the trace of their ratio (the Fisher linear discriminant)? It seems that it would reverse one of the conclusions of the analysis, since the trace of the Fisher discriminant ratio $\mathrm{tr}(\Sigma_W^{-1} \Sigma_B)$ is invariant to invertible linear transformations, and decreases under non-invertible linear transformations, so can only be improved through the non-linearity. If the conclusion of which part of the network is responsible for DNC depends strongly on the chosen metric, can we really ascribe meaning to the question? It seems to me that it is really the sequence of weights and non-linearity which _together_ induce DNC, and trying to separate their effects is not really possible. Finally, Proposition A.1 was first published by Cho and Saul in _Kernel Methods for Deep Learning_, NIPS 2009. Besides, the expression of the kernel in eq. (5) can be simplified with algebra and trigonometry (compare with their eq. (6)). Minor notes and suggestions: - I suggest using a so-called "diverging" colormap (such as "bwr") in Figure 1 to clearly separate positive from negative correlations, and use the same range for both datasets. - I suggest replacing "Gram matrix" with "(uncentered) covariance" to refer to $W^TW$, as weight matrices $W$ are generally decomposed in rows which correspond to individual neurons. - The notation $||X||$ to refer to the vector in $\mathbb R^N$ of column norms of a matrix $X \in \mathbb R^{d\times N}$ is never introduced (and clashes with the usual convention that this is a matrix norm). - Why is the last layer denoted $W_{L+1}$ instead of $m_{L+1}$? - The choice of layer-indexing is confusing and seems inconsistent throughout the paper. Contrarily to what is stated in section 3.1, isn't $X_l$ the features after $l-1$ network layers? I suggest to denote the input as $X_0$ instead of $X_1$ to simplify the notations. Also, it seems that $M_l^{1/2} X_l$ should be referred to as $\tilde X_{l+1}$ rather than $\tilde X_l$ given the chosen conventions. - Typo: missing a norm in the definition of $\bar H_l$ line 128. -In section 4.2, I suggest defining activations before the kernels, e.g., $\tilde X_{l+1} = \kappa^{-1/2} M_l^{1/2} X_l$ and $X_{l+1} = \Phi_{\rm lin}(\tilde X_{l+1})$. I also suggest choosing a different notation for $k_{\rm lin}$ and $\Phi_{\rm lin}$ which are confusing as they imply linearity, and to avoid the awkward "non-linear feature map $\Phi_{\rm lin}$". - Typo line 260: the output space of $k_M$ should be $\mathcal R^{C\times C}$. - I suppose that $\lambda = \mu$ in section 4.3. - In the caption of Figure 2, I suppose that "fully-connected" should be removed in the case of ResNet.

Questions

- Are the feature maps $\Phi_l$ and kernels $k_l$ unrestricted in Algorithm 1, or do they have to match in the sense that $k_l(x,x') = \langle \Phi_l(x), \Phi_l(x')\rangle$? - What is the motivation behind considering _normalized_ features _before_ the non-linearity in section 4.1? Could the authors clarify the role of these non-standard conventions? - Why are the setting different between sections 4.1 and 4.2? (relative to normalizations). Is neural collapse still empirically observed with the choices of section 4.2? It raises the suspicion that it could not the case, in which case Theorem 4.3 would not really explain the results of section 4.1 (e.g., because the asymptotic regime is not reached in practice).

Rating

6

Confidence

3

Soundness

3

Presentation

1

Contribution

3

Limitations

See weaknesses. In its current state, I think that the paper is slightly below the acceptance bar and would require minor, if not major, changes before it can be fully appreciated by the NeurIPS community. I would be happy to raise my score if the authors address the points raised above.

Authorsrebuttal2024-08-08

We thank the reviewer for their reply. Yes, we only study neural collapse on the training set. Note that there are mixed empirical results about whether or not neural collapse transfers to the test set [1, 2, 3]. Moreover, to the best of our knowledge no work has studied the emergence of neural collapse at test time theoretically. Perfect collapse at test time would mean perfect generalization, therefore only weaker forms of test-collapse are available [3]. We emphasize that neural collapse is a particularly structured form of interpolation that is not implied by low train loss, or overfitting, alone. In fact, there are many ways either neural networks or Deep RFM can interpolate the training data. Therefore, it is remarkable that the particular interpolating solutions found by these models exhibit DNC. Our paper, as well as all the other theoretical papers on collapse, study why among all interpolations for a given model class and dataset, the training procedure selects a solution exhibiting DNC. You are correct, the non-linearity is essential to achieve the collapse for linearly non-separable data. The ReLU plays an important role, as your proposed metric shows. We will make it clear in our revision that what we mean by saying that the linear layers are responsible for reducing within-class variability is not that they can achieve collapse on their own, but that in the solved model, it is the linear layers that directly decrease the NC1 metric. Moreover, the ReLU plays a critical role in enabling collapse, especially when the data is not linearly separable. [1] Xu and Liu. “Quantifying the variability collapse of neural networks.” International Conference on Machine Learning, 2023. [2] Kothapalli. “Neural collapse: A review on modelling principles and generalization.” arXiv pre-print, 2022. [3] Hui, Belkin, Nakkiran. “Limitations of neural collapse for understanding generalization in deep learning.” arXiv pre-print, 2022.

Reviewer uXDR2024-08-08

Thank you very much for these precisions and the references. I had wrongly assumed that neural collapse was known to occur on the test set, but I see the picture is much more nuanced.

Reviewer 1F1v6/10 · confidence 2/52024-07-12

Summary

The submission introduces a mechanism for Deep Neural Collapse (DNC) using the average gradient outer product (AGOP). The authors also propose the Deep Recursive Feature Machines (Deep RFM) model, which employs AGOP in its architecture to empirically and theoretically demonstrate DNC. The main contribution is that AGOP-based explanation is a data-based approach while prior work focused on data-agnostic explanations.

Strengths

* Using a data-based approach based on AGOP to explain DNC is novel to the best of my knowledge * The paper offers both theoretical analysis and empirical evidence supporting the role of AGOP in inducing DNC * The experiments are performed on different architectures and datasets

Weaknesses

* I found the paper challenging to read * I am unsure about the practical implications of this work

Questions

* Can the authors clarify if other metrics, such as the Neural Tangent Kernel (not the limit), would effectively predict this behavior? Or AGOP is unique in this aspect? * What are the practical implications of this work?

Rating

6

Confidence

2

Soundness

2

Presentation

2

Contribution

2

Limitations

See weaknesses.

Reviewer AtBe6/10 · confidence 4/52024-07-13

Summary

The authors study two effects associated with neural collapse: the within class variability going to zero and the orthogonality/tight-frame of the class means. They study the deep recursive feature machine model, and show that neural collapse forms in that setting as well, due to the projection of the data onto the average-gradient outer product (AGOP). They show both empirical and theoretical results on this phenomenon, leveraging high-dimensional gaussian equivalence of nonlinear random feature models. Further, they show that the right singular vectors of the weight matrices are responsible for most the within-class variability collapse, projecting onto the subspace spanned by the gradients.

Strengths

The writing of the paper is for the most part quite readable. The literature review is thorough and puts the results of this paper in a good context. The empirics are extensive and compelling. Moreover, the theoretical ideas leveraging the equivalence of nonlinear random features to a linear kernel with additional identity term make for a compelling argument about the mechanism for neural collapse in RFMs. Given the good mix of theory and experiment, I recommend this paper for acceptance.

Weaknesses

Section 4 is doing many things at the same time. It may be better to split it into an empirical evidence section, and then do a section on the theoretical results. In particular, it would be good to give an idea of where the theoretical ideas are going at the start of 4.2 before setting out to prove the deep neural collapse results. This would substantially improve the readability of this section. This goes double for section 4.3. The opening paragraph of that section is unreadable: *"Next, we show that the formation of the neural collapse is not only implicitly given by the specific optimization procedure of the Deep RFM, but is also implicitly regularized for in the parametrized kernel ridge regression, a model class that includes RFM (i.e., a single layer of Deep RFM)"* I don't really understand what this is saying, or even what you're trying to accomplish in the entire subsection. I tried many times to read it. The whole subsection should be rewritten. There are many sentences there that make no sense to me. Here is another one: *"Since we do not explicitly regularize M, we will drop the dependence on it, treat k as a free optimization variable and compute the optimal value of the following relaxed problem:"* This is certainly not something you can do generally. For example if I had a matrix parameterized as $A = M_1 M_2$ and optimized just the $M_i$ with no explicit regularization, there are many cases where this isn't the same as optimizing $A$. Maybe you mean to say something else but once again I can't understand what you're trying to say. The prior subsection was compelling enough that I am discounting this rather poor form of writing. Please rewrite this section. More generally, there are many sentences throughout the paper that are not well-worded and seem to run on. Improving the writing would benefit the quality, precision, and reach of this otherwise strong paper. If in your rebuttal you can provide examples of improved presentation, I may raise my score higher.

Questions

One can show that for a $\ell_2$ regularized deep network that the weights pick up rank one spikes proportional to $W^{\ell}_{ij} \propto \frac{\partial f}{\partial x^\ell_i} \phi(x^{\ell}_j)$ where $\phi$ is a nonlinearity and $x^\ell$ is the preactivation.. This usually means that the *left* singular values of the weight matrices should pick up terms aligned with $\nabla f$. See for example the update equation for $W$ in 2.2.4 of LeCun: http://yann.lecun.com/exdb/publis/pdf/lecun-88.pdf Is there any easy way to square this with the results on RFMs, that the *right* singular values align with $\nabla f$ terms? It would be good to be explicit and put a subscript below the $\nabla$s on the $\nabla f_\ell(x^\ell_{c i})$ in algorithm 1 to be clear what you're differentiating with respect to. I don't understand why 4.3 is called non-asymptotic analysis. I don't think you're proving non-asymptotic bounds compared to 4.2. If anything 4.3 seems completely unrelated. Can you please give it a title where someone can understand what you're trying to do? Once again the rest of the paper is quite readable but this subsection is a mess.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

This work elucidates an important phenomenon in deep learning theory. Developing a principled understanding of feature learning is likely to have implications for the interpretability and reliability of AI systems.

Reviewer uXDR2024-08-07

I thank the authors for their detailed response. They have addressed my main concern regarding the writing of the paper. As a result, I have increased my score. If I understand correctly, in this paper neural collapse is only considered on training data? That is, generalization is not required and a perfectly overfitting network would achieve neural collapse? I think this should be explicitly pointed out in the text. Indeed, it makes it very easy to achieve neural collapse, with a number of random features that is larger than the number of training samples, followed by a linear layer. Doesn't the statement "deep RFMs can achieve neural collapse" then reduce to "deep RFMs can overfit their training set"? A mystery of neural collapse that is not studied here is that the within-class variability remains negligible _on the test set_. Thank you for pointing out that the Fisher linear discriminant can decrease under noninvertible linear transformation, I was mistaken. I however maintain that stating something like "the linear layers are mostly responsible for neural collapse" is misleading, as removing all non-linearities from the network would prevent neural collapse from arising since the data is not non-linearly separable. So the non-linearities must have a role, as they "enable" the collapse of within-class variability by the next linear layer. Although this is not reflected in the metrics evaluated at intermediate points, non-linearities thus play an important role. To iterate on the question of the metric, if I replace NC1 with "the value of NC1 after applying a linear operator that minimizes NC1" (i.e. after a linear "probe" classifier), then only non-linearities can decrease this new metric.

Area Chair hkw92024-08-08

Dear reviewers, could you have a look at the authors response and comment on them if you have done so, yet. thanks in advance your area chair

Reviewer 1F1v2024-08-08

I thank the authors for their response. They have addressed most of my concerns. Therefore, I have increased my score. I look forward to the updated version of the manuscript.

Reviewer Ljpy2024-08-08

I would like to thank the authors for their detailed response. After consideration of their comments and those of the other reviewers I have decided to increase my score.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC