Noise Balance and Stationary Distribution of Stochastic Gradient Descent

The stochastic gradient descent (SGD) algorithm is the algorithm we use to train neural networks. However, it remains poorly understood how the SGD navigates the highly nonlinear and degenerate loss landscape of a neural network. In this work, we show that the minibatch noise of SGD regularizes the solution towards a noise-balanced solution whenever the loss function contains a rescaling parameter symmetry. Because the difference between a simple diffusion process and SGD dynamics is the most significant when symmetries are present, our theory implies that the loss function symmetries constitute an essential probe of how SGD works. We then apply this result to derive the stationary distribution of stochastic gradient flow for a diagonal linear network with arbitrary depth and width. The stationary distribution exhibits complicated nonlinear phenomena such as phase transitions, broken ergodicity, and fluctuation inversion. These phenomena are shown to exist uniquely in deep networks, implying a fundamental difference between deep and shallow models.

Paper

References (16)

Scroll for more · 4 remaining

Similar papers

Reviewer mL9n5/10 · confidence 4/52024-06-20

Summary

The paper studies the effect of rescaling symmetry in SGD and shows SGD tends to favor solutions with balanced gradient noises. The authors then derive an exact solution of the stationary distribution of a toy model trained by SGD. The derived solution shed lights on problems observed in deep learning such as fluctuation inversion and edge of stability.

Strengths

The paper contributes to the understanding of SGD properties. The noise balance theorem is novel and important. The analytical solution as well as the interpretation is interesting and insightful.

Weaknesses

The results of the paper are interesting and important, the writing needs refinement to improve clarity and precision. The conditions under which the results hold is sometimes omitted, leading to confusion. The language should also be made more precise. Minor points: 1. The first paragraph in related works appears to overstate the novelty of the results. Specifically, our result is the first to derive an exact solution to the stationary distribution of SGD without any approximation. (Line 55-56) This is a strong claim, but it seems inaccurate. There are previous results showing exact solution of stationary distribution of SGD (e. g. Liu Ziyin 2021). Corollary I.1 in arXiv:2306.04251 (2023) also states the stationary distribution on a deep learning setup similar to the D=1 model discussed in this paper. Also, the solution given in the paper is for a specific model. These should be made clear. 2. It seems that eq. 15 takes D=1, which has not been stated and thus is confusing. 3. It is unclear why the left figure of Fig. 5 has only two theory lines instead of three. Major points: 1. The related works on symmetry and SGD dynamics are insufficient. There are a few related works that are missing, e. g. arXiv:2309.16932 (2023). 2. The paper has not discussed convergence to the stationary distribution. The authors seem to assume convergence to stationary distribution and use interchangeably the SGD properties and the stationary solution properties (e. g. line 97-98). However, the properties of SGD can be very different from the properties of stationary solutions unless convergence to the stationary solutions is guaranteed. The authors should clarify this. 3. The authors fail to discuss uniqueness of the stationary solutions. For example, it is unclear to me why eq (3) is a necessary and efficient condition for stationarity. Eq (3) is a critical result in the paper, and it would be better to make it a theorem or corollary. However, since eq (2) cannot be interpreted as a deterministic ODE. The unique condition for a stationary distribution should be justified, especially considering that C1 and C2 are not constant but depend on u and w. 4. The equivalence of SGD bias and weight decay is not rigorous. (line 155-158) The C0 term is not constant but depends on u and w, while the weight decay rate is constant.

Questions

1. The entire paper is based on rescaling symmetry, but why tanh without rescaling symmetry is used in Fig. 3 and 5?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have listed limitations at the end. The major limitations are the simplicity of the model and lack of experiments on deep neural networks.

Authorsrebuttal2024-08-10

Reply

Thank you for your additional questions. **For the author’s response to my question, I would suggest replacing the experimental results with the ones run with ReLU activation. ReLU is widely used, and it still makes little sense to me that the author decided to run the experiments with tanh while the entire paper is based on rescaling symmetry.** Thank you for your suggestion. We will move the results of tanh nets to the appendix, and use the ones for ReLU nets in the main text to avoid confusion. **For the supplement pdf, I have two follow up questions.** **1. From my understanding, Equation 124 should still be considered a stochastic differential equation by nature. Then, how can the author arrive at Corollary D.2, which is a deterministic statement without out any probability condition?** This is a misunderstanding. There is no stochasticity in the evolution equation in Eq. 124. The diffusion terms in the dynamics of $\|u\|^2$ and $\|w\|^2$ cancel with each other due to the rescaling symmetry – even if $u$ and $w$ themselves are random. This (rather surprising fact) is essentially what the theorem is trying to prove. Also, please see our last reply to reviewer (56YC) for an alternative and more technical proof of theorem 3.1, which may be easier to understand for some readers. Now, because Eq. (124) has no noise term, one can further construct a deterministic dynamics that strictly upper bound Eq. 124, which is how one can arrive at the corollary. **2. In the right panel of the new Figure 6, why does the difference in the norm grows up again later in time?** First of all, Figure 6-right can be a little visually misleading. It looks like the neurons first have a decreasing $|\|u\|^2-\|w\|^2|$ and then an increasing $|\|u\|^2-\|w\|^2|$, but the actual situation is simply that roughly half of the signs of $\|u\|^2-\|w\|^2$ are changing during learning, and so in a log scale, it looks like they are first decreasing and then increasing, even though $\|u\|^2-\|w\|^2$ are quite monotonic in reality. Secondly, $\|u\|^2-\|w\|^2=0$ is not necessarily the stationary point and Corollary D.2 only guarantees a convergence to a neighborhood $B$ close to a stationary point, and so there is no reason to expect a convergence to any specific value, especially when the model is nonlinear. What happens the most often is that the parameters fluctuate around some small neighborhood, which is exactly what Figure 6-right shows. Please ask for additional clarification for any point that is not yet clear.

Authorsrebuttal2024-08-14

reply

Thanks for the reply. We will refine our statements. What you point to is a corner case. Technically speaking, $\lambda^*$ is indeed allowed to be infinity. The meaning is clear. When $\lambda^* =\infty$, the system diverges with probability 1 (here, an $O(1)$ neighborhood of infinity simply means infinity, and thus, divergence). This is not uncommon when the model has some sort of continuous symmetry. For example, in case of scale invariance, it is well-understood that SGD diverges along the degenerate directions (namely, the radial direction) when there is no weight decay (e.g., see https://arxiv.org/pdf/2102.12470)

Reviewer 56YC5/10 · confidence 4/52024-06-22

Summary

For ReLU networks trained by gradient flows, it is classical that a type of Minkowski inner product between the coefficients of consecutive layers is preserved. The authors demonstrate a monotonicity of the same quantity for stochastic gradient descent in continuous time. They use this to study the invariant distribution of parameters trained by (continuous time) SGD.

Strengths

The topic is well-chosen and the results - if correct - are very interesting.

Weaknesses

* In its current form, I find the article a bit unpolished and the results not easy to access. Many questions remained unanswered when I tried reading the article (see questions). * Important quantities are defined throughout the plain text. I understand that reading as a reviewer under time pressure is different from normal reading, but for instance in Theorem 3.1, I would have hoped for a more self-contained statement on relations and properties $L, C, \ell$ and the distribution of $x$ have to satisfy. As far as I can tell, the statement is fairly general and not specific to machine learning. * I have serious doubts about Theorem 3.1. It is derived in Appendix A from Itô's Lemma without the diffusion term. This is valid *in expectation over $\theta$*, but not pointwise in $\theta$. Pointwise in $\theta$, there should be white noise in the 'time derivatives', i.e. the ODE identity should be written as an SDE. In the proof, equations (27) and (28) appear to be wrong. * The authors do not pay any attention to whether solutions to the evolution equations exist (or are unique). Problems with regularity can sometimes be alleviated if the distribution in $x$ is sufficiently regular, but I would appreciate a short discussion.

Questions

* The noise matrix $C$ defined below (1) is an uncentered covariance matrix, while the centered version appears to be used below (and I believe that this would be correct). * In line 499, can you justify that $\ell$ is a function of only the rank one matrix, and is the function sufficiently smooth to take derivatives? Is it defined on the space of matrices, or at least in an open neighborhood of the set of rank 1 matrices so that the derivatives are well-defined? This is easy to settle in the linear settings in the main article, but a more thorough consideration is required in general. * The notation $\partial\tilde \ell /\partial(u_iw_j)$ is very unfortunate - for a single derivative, there should not be two arguments. This would be a derivative of $\tilde\ell$ in direction $z_{ij}$ on a space of matrices... Can you explain why the modification is needed? And could it be avoided in the statement of Theorem 3.1 to focus on the more intuitive quantities involving the original loss $\ell$? * Where in the proof of Theorem 3.1 is there any indication that $tr(C(u)) = tr(C(w))$? I have looked fairly closely and cannot find a justification for (3). I also assume that the covariance matrix of $u$ depends on $v, w$ and the covariance matrix of $w$ equally depends on all three sets of variables? I am also unsure how precisely this indicates that 'gradient noise between the two layers is balanced'. * What is the distribution of $x$ in the experiment of Figure 1? Please include code for reproducibility or describe the experiments which illustrate your point. * How does (2) imply that 'a single and unique point is favored by SGD'? * Would Theorem 3.1 hold for *all* neurons with leaky ReLU activation? * What do you mean by 'the difference between GD and SGD at stationarity is O(1)'? GD does not have an invariant distribution and the terminal state depends on (random) initialization. Is the message that a gradient flow analysis is only valid for a fixed finite time horizon? * Should the minimizer of (10) be $\beta_2 /(\beta_1+\gamma)$ or should $\beta_1 = \mathbb E[x^2] + \gamma$? * Below (10), what are the $\alpha_i$? Is this the same as in (8) for a different model? * I find that $$ \Delta = \min_v C(v) = 4\min_v \left( \alpha_1v^2 - 2\alpha_2v + \alpha_3\right) = 4\left(\alpha_3 - \frac{\alpha_2^2}{\alpha_2}\right) $$ is achieved for $v= \alpha_2/\alpha_1$. This differs from the value proposed in the article by a factor of $4/\alpha_1$. * I cannot find the proof of (12) in the appendix anywhere, and I am unsure what $\beta_1'$ is. More concrete references may be useful. * In Figure 3, what is plotted for $v$ in the depth 1 case? Is it the product of parameters?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Yes.

Authorsrebuttal2024-08-11

reply

Thank you very much for this very good question. It is true that our definition only defines a rank-1 subspace of the gradient vector of $\tilde{\ell}$, but this is all that is required for us. Since the same argument applies to $C_1$ and $C_2$, we focus on $C_1$. Essentially, we only need the quantity $u^T C_1 u$ to exist, and this only requires a rank-$1$ subspace of $C_1$ to be defined, which is defined as: $u^T C_1 u = \sum_{i,j,k}u_i u_j\mathbb{E}[\partial\tilde{\ell}/\partial (u_i w_k) \partial\tilde{\ell}/\partial(u_j w_k)] = \sum_{i,j,k}\mathbb{E}[ u_i \partial\tilde{\ell}/\partial(u_i w_k) \partial\tilde{\ell}/\partial(u_j w_k) u_j] $ (ignoring the second term of $C_1$ as it follows from the same argument). Meanwhile, the quantity $ u_i \frac{\partial\tilde{\ell}}{\partial(u_i w_k)} = u_i \frac{\partial{\ell}}{\partial(u_i)} \frac{1}{2 w_k} + \frac{1}{2}\frac{\partial{\ell}}{\partial(w_k)} $ (see our very first rebuttal), which is well-defined whenever the gradient with respect to the original loss is well-defined. Since each term of the sum is well-defined, the quantity $u^T C_1 u$ is also well-defined. Similarly, $w^T C_2 w$ is always well-defined. Lastly, the fact that we only require a rank-$1$ condition for the gradient of $\tilde{\ell}$ does not imply that $\nabla \tilde{\ell}$ when viewed as a matrix is rank-$1$. As an example, consider the case $\ell(u_i, w_i)= \sum_i u_i w_i$, and $\tilde{\ell}(Z) = \sum_i Z_{ii}$, such that $Z_{ii}(u, w) = u_i w_k$. Thus, we have that $\nabla_Z \tilde{\ell}(Z) = {\rm identity}$, which is full-rank and so its largest and smallest eigenvalues can still be well-defined.

Reviewer 56YC2024-08-12

I will increase my score. I still feel that this paper is not fully ready for publication and requires significant revision, but my concerns about mathematical correctness have been addressed. I thank the authors for engaging with the questions and providing thoughtful replies.

Authorsrebuttal2024-08-12

reply

Thanks for the reply. We will do our best to incorporate the feedback from the reviewers to improve the manuscript.

Reviewer HVit5/10 · confidence 4/52024-07-11

Summary

This paper tries to analyze the specific features that carry the noise of SGD (through a continuous model). The authors show that there is a certain 'law of balance' across the layers when some invariance is assumed. Going further, they derive a toy model to push their study, showing that there is an analytic stationary solution to it. They finally propose a phenomenology related to the role of the noise of SGD when analyzing this precise stationary distribution.

Strengths

The idea that a conservation law for the gradient flow implies an asymptotic balancedness condition for the stochastic flow is a good and striking idea. The one-dimensional examples that are given in the text are very pleasant to follow and they are good exercices to display the ability of the stochastic flow to diverge from the gradient flow. The example given in Eq.(13) is thoroughly analyzed.

Weaknesses

The paper present the following weaknesses: - The law of balance is an interesting phenomenon, yet considering it with a closer look, it seems that not much can be said generally and that one has to understand it case by case. In one dimension, sure, it is possible to conclude that balancedness will occur at exponential speed, yet in dimension more than $2$, it seems impossible to predict it surely. - I have to say that I was a bit bothered by the general overselling of the paper : - As said before the law of balance is truly valid asymptotically in one-dimension - The stationary distribution that the authors claim to be the first to derive is for a very specific model, which is not standard and does not resemble a diagonal network! - The fact that the stationary distribution can be computed is also very inherent to $1d$ calculation and is simply a recognition of a Pearson diffusion that already made in way in ML (at least in https://arxiv.org/pdf/2402.01382 and https://arxiv.org/pdf/2407.02322). Minor typos/flaws: - l.41: Fokker Planck is not inherently high-dimensional - l.44: Go to the line for new paragraph - l.165: The law of balance is not strictly applicable here since $\ell$ is not scale invariant because of the regularization. - Section **4.1 Depth - 0**: I think that $\Delta > 0$ is not currently the "most practical example" since it corresponds to a underparametrized model.

Questions

On top of the questions raised by the weakness section, here a some (more minor) questions: - l.65: Eq (1) is not the usual covariance matrix that is used for Stochastic modified equations. - l.95: Eq (3) seems important but it is difficult to follow where this comes from without intermediate calculations, can the authors develop? - l.110: Same thing for the equation $\lambda^4 = \frac{\langle w^*, C_2 w^* \rangle}{\langle w^*, C_1 w^* \rangle}$ (3): can the authors develop? - Thm 4.2 : Are they the only stationary distribution ? How to know to which one it converges ?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

As said before, all conclusion are drawn for models that live intrinsically in one dimension.

Reviewer 56YC2024-08-08

I appreciate the authors' response and explanation. * I looked at the proof of Theorem 3.1 and I am unable to fill in the gaps where diffusion terms should be. It is possible that the authors chose an unfortunate simplifying notation and that this is correct, but I cannot verify the result right now. I would also be surprised at true monotonicity for a stochastic quantity. In Corollary 3.3, the authors seem to use the result in an ODE fashion to conclude that |u-v|^2 in fact decays to zero. * It is still not entirely clear to me how to show that the loss only depends on the rank 1 matrix and how the 'lifted' function would be defined on the space of matrices (not just the set of rank 1 matrices, which does not form a linear subspace). An explicit construction or formula would be appreciated. I currently maintain my score and certainty. If the authors can provide a convincing answer to the two questions, I will most likely increase the score but drop the certainty, since I do not feel like I will be able to adequately assess its merit during the discussion period. For the record, I find the work interesting and I believe that it deserves to be published, but I have reservations about the correctness (or the presentation, if this is really a question of notation).

Authorsrebuttal2024-08-10

reply

**I appreciate the authors' response and explanation.** **I looked at the proof of Theorem 3.1 and I am unable to fill in the gaps where diffusion terms should be. It is possible that the authors chose an unfortunate simplifying notation and that this is correct, but I cannot verify the result right now. I would also be surprised at true monotonicity for a stochastic quantity. In Corollary 3.3, the authors seem to use the result in an ODE fashion to conclude that |u-v|^2 in fact decays to zero.** Thanks for this question. We clarify the details below. We chose a simplified notation in the original proof to avoid confusing the readers with too much technical details. More technically, the proof should start with Eq. (1), and the standard SDE formulation. The following is how one would arrive at the result in this route. Also, a core aspect of the result of Theorem 3.1 is exactly that Eq. (2) is an ODE! This is because the diffusion term cancel each other for the time evolution of $\|u\|^2-\|w\|^2$. The mechanism for this cancellation is essential to the proof: the gradient noice covariance is low-rank due to symmetry. Now, we outline the proof of Theorem 3.1 in a more technical and standard manner. By defining $\theta=(u^T,w^T)^T$, we can rewrite the quantity $\|u\|^2-\|w\|^2$ as $\theta^TB\theta$, where $B:=\quad \begin{pmatrix} I_u & O \\\\ O & -I_w \end{pmatrix} \quad$. The dynamics of $\theta$ is described by Eq. (1): $d\theta/dt=-\nabla_{\theta}L+\sqrt{TC(\theta)}dW/dt$, where $C(\theta):=\mathbb{E}[\nabla_{\theta}\ell\nabla_{\theta}^T\ell]-\mathbb{E}[\nabla_{\theta}\ell]\mathbb{E}[\nabla_{\theta}\ell]$. To obtain the dynamics of $\theta^TB\theta$, we use Ito's lemma: If the vector $X_t$ satisfies the stochastic process $dX_t=\mu_tdt+D_tdW_t$, then the dynamics of a function $f(X_t)$ of $X_t$ can be written as $d f(X_t) = \left(\nabla_X^T f \mu_t + \frac{1}{2}\mathrm{Tr}[D_t^T \nabla_X^2 f(X_t) D_t] \right)dt + \nabla_X^Tf(X_t) D_t dW_t$. Hence, the dynamics of $G(\theta)=\theta^TB\theta$ is given by $dG(\theta)/dt=-\theta^TB\nabla_{\theta}L+\theta^TB\sqrt{TC(\theta)}dW/dt+T\mathrm{Tr}[C(\theta)B]$. To simplify this, we use the infinitesimal form of the rescaling symmetry, which can be expressed as $\ell(\theta,x)=\ell(A\theta,x)$ with the matrix $A:=\quad \begin{pmatrix} (1+\epsilon) I_u & O \\\\ O & (1-\epsilon)I_w \end{pmatrix} \quad$. We expand the equation to first order in $\epsilon$ and obtain $\theta^TB\partial\ell/\partial\theta=0$. By taking expectation to both sides, we have $\theta^TB\partial L/\partial\theta=0$. In addition, $\theta^TBC(\theta)=\mathbb{E}[\theta^TB\nabla_{\theta}\ell\nabla_{\theta}^T\ell]-\mathbb{E}[\theta^TB\nabla_{\theta}\ell]\mathbb{E}[\nabla_{\theta}\ell]=0$. Therefore, we can see $\theta^TB\sqrt{C(\theta)}=0$ since $C(\theta)$ and $\sqrt{C(\theta)}$ share the same null space. By substituting $\theta^TB\partial L/\partial\theta=0$ and $\theta^TB\sqrt{C(\theta)}=0$ into the evolution equation of $G(\theta)$, we have $dG(\theta)/dt=T\mathrm{Tr}[C(\theta)B]$, which is nothing but Eq. (30). Here we can see the diffusion terms related to $dW$ vanish due to the rescaling symmetry. **It is still not entirely clear to me how to show that the loss only depends on the rank 1 matrix and how the 'lifted' function would be defined on the space of matrices (not just the set of rank 1 matrices, which does not form a linear subspace). An explicit construction or formula would be appreciated.** Due to the rescaling symmetry, the loss function only depends on this rank 1 matrix. To show this, we redefine a new function $\tilde{\ell}(u_iw_j,w_j,x):=\ell(u_i,w_j,x)$ to reorganize the parameters. Then the derivatives of $\ell$ can be rewritten as $\frac{\partial \ell}{\partial w_j}=\sum_{i}u_i\frac{\partial \tilde{\ell}}{\partial(u_iw_j)}$ and $\frac{\partial\ell}{\partial u_i}=\sum_{j}w_j\frac{\partial \tilde{\ell}}{\partial(u_iw_j)}+\frac{\partial \tilde{\ell}}{\partial(w_j)}$. However, due to the rescaling symmetry $\tilde{\ell}(u_iw_j,w_j,x)=\tilde{\ell}(u_iw_j,\lambda w_j,x)$, we have $\sum_jw_j\partial\tilde{\ell}/\partial(w_j)=0$. Since this equality holds for an arbitrary $w_j$, we always have $\partial\tilde{\ell}/\partial(w_j)=0$ for all $j$. Hence, the loss function only relies on the rank 1 matrix $u_iw_j$. We will clarify this in the revision. Please ask for additional clarification for any point that is not yet clear.

Reviewer 56YC2024-08-10

**I looked at the proof of Theorem 3.1 and I am unable to fill in the gaps where diffusion terms should be.** I thank the authors for the explanation. It makes a lot more sense now and I appreciate the detailed presentation. I believe that integrating it into the article will help future readers as well. **It is still not entirely clear to me how to show that the loss only depends on the rank 1 matrix and how the 'lifted' function would be defined.** I still do not follow. Let me explain my dilemma: The authors define the function $\tilde \ell$ on the space of rank 1 matrices by $\tilde \ell (u\otimes v) = \ell(u, v)$. In my understanding, they argue that it does not depend on the scaling between the vectors $u, v$ which are used to decompose $u\otimes v$ (unique up to scaling). Where I do not follow is when they take the derivative with respect to the ij-entry of the matrix, corresponding to a difference quotient $$ \frac{\tilde \ell (u \otimes v + h e_{ij}) - \tilde \ell(u\otimes v)} h. $$ However $u\otimes v + h e_{ij}$ is not necessarily a rank 1 matrix, meaning that $\tilde\ell$ is not defined there, as far as I can tell. This issue also arises in their previous response. I think this is the result of formal manipulations which may not be applicable here. If the authors can provide an explanation or alternative argument, I will be happy to increase my score.

Reviewer mL9n2024-08-08

I appreciate author’s response to my review! For the author’s response to my question, I would suggest replacing the experimental results with the ones run with ReLU activation. ReLU is widely used, and it still makes little sense to me that the author decided to run the experiments with tanh while the entire paper is based on rescaling symmetry. For the supplement pdf, I have two follow up questions. 1. From my understanding, Equation 124 should still be considered a stochastic differential equation by nature. Then, how can the author arrive at Corollary D.2, which is a deterministic statement without out any probability condition? 2. In the right panel of the new Figure 6, why does the difference in the norm grows up again later in time?

Authorsrebuttal2024-08-13

Thank you for the reply

Hi! We noticed that you have not responded to our previous clarification. It would be really helpful to us when revising and improving our manuscript if we could hear more from you. We would be happy to hear any additional thoughts or feedback you have on our work and on our previous reply!

Reviewer mL9n2024-08-13

I thank the authors for the clarifications! I am unable to arrive at Corollary D.2 without seeing the proof. Overall, I think the paper has novel contribution and the results are interesting. However, I feel like it still needs some refinement, especially for the language precision. For example, in the rebuttal pdf, $\lambda^*$ can take $\infty$, and it is unclear to me for Corollary D.2 what it means for an $O(1)$ neighborhood of $\lambda^*$ when $\lambda^*$ takes $\infty$.

Reviewer HVit2024-08-10

After Rebuttal

I thank the authors for the good rebuttal that has answered partially to my concern. For this reason I decide to increase my score by one, still thinking that the article lacks some convincing example to strenghten their claim.

Authorsrebuttal2024-08-11

reply

Thanks for your reply and update. We would really appreciate it if you can be more specific regarding "convincing examples." To be specific, which claim in the paper do you think requires more example to strengthen? Knowing this would really help us improve our work.

Program Chairsdecision2024-09-25

Decision

Reject

© 2026 NYSGPT2525 LLC