How to Scale Your EMA

Preserving training dynamics across batch sizes is an important tool for practical machine learning as it enables the trade-off between batch size and wall-clock time. This trade-off is typically enabled by a scaling rule, for example, in stochastic gradient descent, one should scale the learning rate linearly with the batch size. Another important machine learning tool is the model EMA, a functional copy of a target model, whose parameters move towards those of its target model according to an Exponential Moving Average (EMA) at a rate parameterized by a momentum hyperparameter. This model EMA can improve the robustness and generalization of supervised learning, stabilize pseudo-labeling, and provide a learning signal for Self-Supervised Learning (SSL). Prior works have not considered the optimization of the model EMA when performing scaling, leading to different training dynamics across batch sizes and lower model performance. In this work, we provide a scaling rule for optimization in the presence of a model EMA and demonstrate the rule's validity across a range of architectures, optimizers, and data modalities. We also show the rule's validity where the model EMA contributes to the optimization of the target model, enabling us to train EMA-based pseudo-labeling and SSL methods at small and large batch sizes. For SSL, we enable training of BYOL up to batch size 24,576 without sacrificing performance, a 6$\times$ wall-clock time reduction under idealized hardware settings.

Paper

Similar papers

Peer review

Reviewer nnfC7/10 · confidence 4/52023-07-06

Summary

In duality to previous work on learning rate scalings, the authors present a novel scaling strategy for the momentum parameter in teacher-student settings, i.e. how to adapt the momentum in the learning dynamics when the batch-size is changed. The work puts an emphasis on the large batch-size setting, which is of particular interest due to its computational efficiency in practice. The obtained scaling is validated through a theoretical analysis and numerous experiments in different settings demonstrate that (a slight modification of) the scaling indeed preserves performance at large batch-sizes.

Strengths

1. The paper is very well-written and easy to follow. The problem of large batch-size training is very relevant as it enables more efficient training (especially when distributing across multiple GPUs) and the resulting scaling rule is very simple. I believe that such insights can be of tremendous use to practicioners, especially if hyperparameter tuning is too expensive, especially in the setting of SSL where the setups are usually already very brittle to hyperparameter changes. 2. The experimental setup is quite broad, including various student-teacher settings such as Polyak-Ruppert averaging, pseudo-labeling and self-supervised learning. 3. The technique is nicely motivated through a theoretical analysis, strengthening the particular choice of scaling further.

Weaknesses

Some important aspects of the experimental setup are not well explained: 1. In the real data experiments (i.e. Fig 2, 3, 4, 5, 6), what does “No EMA Scaling Rule” entail? Does it still apply the “Learning-rate Scaling Rule”? If it does not, then it would be very hard to draw any conclusions regarding "EMA Scaling” as it is not clear how much of the “heavy-lifting” the learning rate scaling is doing. I hope the authors can clarify this. 2. One of the main motivation of the works is to make methods like SSL more computationally effective by ensuring that training them with large batch-sizes does not deterioate performance. In order to make it work in practice, the authors need to resort to “progressive scaling”, where the batch-size is only gradually increased after the warm-up phase. How long is the warm-up period exactly (during which a small batch size is used) and how quickly is the batch-size scaled up afterwards? How much time is lost compared to the “EMA Scaling without Progression”? 3. Another aspect which remained a bit mysterious is the effect of the baseline momentum. For Polyak-Ruppert averaging for instance, its role seems very crucial. If e.g. rho_base were set to 1, the teacher would always be equal to the student and at inference time I would not expect any sort of batch size effect, given that the learning rate scaling correctly works. I would appreciate if the authors could elaborate more on this initial choice of momentum.

Questions

1. The CIFAR10 results in Fig. 5 seem surprisingly bad (around 55%-58%). 2. Are you using a linear or a K-NN probe for the SSL experiments? 3. After the warm-up phase with progressive scaling, how well is the model already performing at this point? I.e. how much is still gained in the large (or maximal) batch-size regime? 4. I find it very surprising that the model is so sensitive to the momentum parameter in Polyak-Ruppert averaging. How well is the student performing at the end of training and how much is actually gained by averaging? I would be very surprised if the non-ideally averaged teacher would perform inferior to the student at the end of training.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have addressed the limitations of their work.

Reviewer 6nqM6/10 · confidence 4/52023-07-06

Summary

This work proposes a scaling rule for EMA in optimization and empirically evaluates its effectiveness across a range of tasks and optimizers. Authors find that proposed method enables large batch training using 24k batch sizes with a 6x wall-clock time reduction.

Strengths

- Idea of formulating a scaling rule for EMA is novel and not one I have seen much in the recent deep learning / optimization literature - Theoretical motivations are clear to understand - Strong empirical study

Weaknesses

- Applicability to modern optimizers is still not clear. Despite the authors discussing some connections to Adam, I think an in-depth discussion of how the theory/scaling rule relates to Adam is not fully clear.

Questions

- My main question is regarding the motivation behind such a scaling rule for EMA. Have the authors verified that such a scaling rule is infact needed since for most practical problems, hyper-parameters are just fixed and no such rule is used, still giving a good performance for most models. Is there a fundamental gap (that has potential to improve model quality or speed substantially) the authors are trying to address here? - It looks like there is still some tuning required for the scaling rule to work? Can the authors clarify how much and sensitivity to larger models and different architectures?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

1 poor

Limitations

- Key questions regarding broad applicability of this method are not answered (such as generality of the solution to varying model types, model scales, etc). Therefore it is not clear how impactful this method is.

Authorsrebuttal2023-08-15

Check for outstanding concerns

Dear Reviewer 6nqM, We thank you again for taking the time to read our manuscript and providing feedback. We hope the rebuttal and additional experiments we provided were helpful. Please let us know if there are still outstanding concerns regarding broad applicability, or any other concerns raised in your review. We will be very happy to address them. Thank you, The Authors

Reviewer 6nqM2023-08-21

Feedback after author feedback

Thanks to the authors for their response and detailed clarifications. I did another pass on the paper and based on author clarifications, I find the contributions to be well-supported empirically and theoretically. I still have some concerns about the presentation and organization of the paper, but overall I am willing to raise my score as my technical concerns have been mostly addressed.

Reviewer dfpr7/10 · confidence 4/52023-07-07

Summary

The exponential moving average (EMA) of model parameters is used in many contexts, including to improve robustness in supervised learning, and as an important component of self-supervised learning. Given model parameters $\theta_t$ at iteration $t$, the exponential moving average $\zeta_t$ is updated as $\zeta_{t+1} = \rho \zeta_t + (1 - \rho) \theta_t$, where $\rho$ is the EMA coefficient. This paper proposes a scaling rule that dictates how $\rho$ should be modified to preserve training dynamics when the batch size is modified. In particular, when the batch size $B$ is scaled by a factor of $\kappa$ yielding $\kappa B$, the EMA coefficient should be exponentiated to the power of $\kappa$, yielding $\rho^\kappa$. The authors justify this rule by analogy to the learning rate vs batch size rule from (Goyal et al., 2017), under the assumption that the gradients change extremely slowly between successive steps in training. They also give a justification using an SDE interpretation. Empirically, they evaluate the EMA scaling rule on several real-world tasks, including supervised learning for image classification and speech recognition, semi-supervised learning for speech recognition, and self-supervised image representation learning. In each case, they compare the loss curves of: 1) a baseline model trained with batch size $B$ and EMA coefficient $\rho$; 2) a model trained with batch size $\kappa B$ and the same EMA coefficient $\rho$; and 3) a model trained with batch size $\kappa B$ and modified EMA coefficient $\rho^\kappa$. They show that using the scaling rule $\rho^\kappa$ produces more similar loss curves to the baseline model than the un-modified $\rho$.

Strengths

* The authors present a toy example where they apply the EMA to a noisy quadratic problem, where the "batch size" is simulated by adjusting the gradient variance. On this task, they compute the optimal EMA coefficient $\rho^*$ for various scaling factors $\kappa$ up to 1024, and show that the $\rho^\kappa$ coefficient nearly matches $\rho^*$ with respect to the approximation error. I appreciate the inclusion of such a toy example. * The empirical evaluation considers diverse tasks, including both vision (ImageNet1K classification with a ResNet) and language (speech recognition with a Transformer) tasks. These experiments also consider different optimizers (SGD and Adam). * The authors also evaluate progressive scaling, in which they increase the batch size during training and simultaneously scale the LR and EMA coefficient. They found that in one case (training a BYOC Vision Transformer), this was necessary to obtain decent performance. * The plots are generally well-made and clear, with good labels.

Weaknesses

* One of the main drawbacks of the paper is that it is not clearly written. The core idea is very simple, and should be simply explained. However, most of the paper is needlessly obfuscated, making it hard to read. The majority of the definitions (2.1, 2.2, 2.3, and 3.1) are not novel, and are basic concepts that would not typically be presented as formal definitions. This reads as though the authors are trying to add math and formality to the paper, when it is not necessary. * The plots in Figure 1(a) are not clearly presented, as the legend overlaps with more than half the figure. * While the experiments use several architectures and datasets, they all measure the same thing, comparing the loss values of models trained with either $\{ (\kappa B, \rho^\kappa), (\kappa B, \rho), (B, \rho) \}$. There is very little diversity in this respect. Are there other things that can be done with the ability to match scaling behaviors, like doing hyperparameter optimization in the small-batch regime (which could be run in parallel on multiple devices rather than requiring a large amount of compute for a single experiment) to find the optimal $\rho$, and then scale it to the large-batch setting using $\rho^\kappa$? * For the Transformer experiment in Figure 3, it is hard to see the long-term behavior of the loss curves. It would help if the y-axis was log-scaled. Here, the red curve ($\rho$ unmodified) seems to do just as well as $\rho^\kappa$, especially in terms of final performance. * Throughout the paper, the notation $\times$ is used for simple scalar multiplication. This is non-standard in ML literature, and confusing to the reader, as $\times$ looks like a cross product. There is no reason to write $\eta \times \frac{1}{B}$ rather than simply $\frac{\eta}{B}$. * In Eq. 6, it is unclear what $\Theta$ and $Z_t$ are, as they are not defined. * Probable typo in L163: "scaled additive $b > 0$" --> "scaling $b > 0$" * The base results (black dashed lines) in the real-world experiments do not seem to have bands for the mean and std over three runs (only the red and blue curves have bands). **Minor** * The math font looks non-standard. * I do not think that the abstract does a good job describing the exponential moving average. * The term "model EMA" should be more clearly defined at the beginning, as "the EMA of the model parameters." * In the abstract, it is not clear what is meant by "Prior works have treated the model EMA separately from optimization." * In the abstract, what is meant by "optimally a $6\times$ wall-clock time reduction"? Is this the actual wall-clock reduction achieved in practice, or under some idealized scenarios? This should be clarified. * L32: "These weights are updated through by a momentum hyperparameter." --> Bad grammar "through by," and it does not make sense to say that the weights are updated "by a momentum hyperparameter." * To improve clarity, the introduction should introduce the mathematical form of the EMA, $\theta' = \alpha \theta' + (1 - \alpha) \theta'$. The expression for the EMA needs to be introduced before Definition 1.1 because otherwise it is not clear what $\rho$ corresponds to and how it is used. * Definition 1.1 should simply be labeled Definition 1, because there is no Definition 1.2. In Def. 1.1, it is confusing to say "scale other optimizers." Should this be "scale other optimization hyperparameters"? * On Line 64, write out "self-supervised learning" instead of saying "SSL" because in this context it could be misinterpreted as standing for "semi-supervised learning." * What is the purpose of including the third row in Eq. 4, which keeps the gradient $g$ unchanged? Is this just to highlight the assumption that the gradient changes very slowly, so that the first two rows hold? * Typo: L115, "typically use to" --> "typically used to" * L278 Typo: "a model EMA alters" --> "a model EMA that alters" * L307 Typo: "only when by the EMA" --> "only by using the EMA" * L210 Typo: "In summary, ASR" --> "In summary, this ASR"

Questions

* In many of the experiments, $\rho$ is very large: in Figure 1(a), $\rho=0.9999$, in Figure 2 $\rho = 0.9999$, in Figure 3 $\rho = 0.9995$, and in Figure 4 $\rho=0.9999$. Why are these EMA coefficients so large? Are these optimal values at the base batch size, and how were they chosen? How do these experiments behave when $\rho$ is smaller, like $\rho=0.9$? As a more general point, because $\rho$ is exponentiated, it seems that there could be numerical overflow or underflow issues depending on the initial value of $\rho$ and the scaling $\kappa$. * In Eq. 5, what form does the gradient noise distribution $\mathcal{E}\_{\sigma}(\theta_k, \zeta_k)$ take? Is it Gaussian? If so, it would be better to use standard notation $\mathcal{N}$. What does it mean that the gradient noise distribution is "assumed to be zero-mean and variance $\Sigma(\theta_k, \zeta_k)$ independent of $\sigma$"? If the variance is independent of $\sigma$, why use the notation $\mathcal{E}_{\sigma}$? * See also questions from the weaknesses box.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

* For the real-world experiments, it would be important to report scaling behavior beyond the factors that are shown in the paper. Why is the maximum scaling factor $\kappa = 8$? Why not continue the scaling to see if the scaling rule continues to hold, and determine where it breaks down? Does it work if $\kappa \in \{ 16, 32, 64, 128, \dots \}$? If the reason for not using larger factors is that the base model already used a large minibatch size ($B=1024$), then why not train a base model with a small batch size $B=16$ such that large $\kappa$ can be evaluated? In addition, because the scaling rule is meant to hold for both increasing and decreasing the batch size, why not have an example reproducing the results of a model trained with $B= 2048$ using a much smaller batch size like $B=16$? This would be a factor of $128$ decrease in batch size. * This also relates to the question about numerical overflow/underflow, because if the original EMA coefficient is $\rho = 0.9999$ and $\kappa = \frac{1}{128}$ then the adapted coefficient would be $\rho^{\frac{1}{128}} = 0.99999922$. What are the limits to the approach, and what are its limitations?

Reviewer bHsu7/10 · confidence 2/52023-08-01

Summary

This paper talks about the scaling rule for model EMA, and proposes that the momentum coefficient of EMA update should be scaled exponentially with the batch size instead of linearly. The authors first informally derive such rule (Equation 4) under the assumption that the gradients change slowly for a few steps. Then the authors theoretically validate such rule by considering the SDE limit of the optimization process (Equation 6) and prove that the optimization trajectory of different batch sizes could still have small difference after we exponentiate by the batch size ratio $\kappa$. For the experiments, the authors first use a noisy parabola and verify that exponentiating the model momentum term would approximate well with the empirical optimal momentum, as illustrated by Figure 1.b. The authors also consider ResNet and transformer for a supervised task, and the results are the same. However, there is a growing gap between the exponentiated momentum with large batch versus the original momentum with small batch (Figure 2), which is attributed to the breakdown of Adam scaling rule. In the final self-supervised experiments, the authors found a performance gap between the scaled model with EMA and the baseline, and thereby introduce progressive scaling which slowly increases batch size during the early stage of training to match the training dynamics at larger batch size.

Strengths

The paper is well-written and easily understandable, and the experiment section is well-designed that contain both simple convex task and a broad spectrum of deep learning tasks. The idea is built on top of Malladi et al. (2022)'s work but the contribution is clear. The application of scaling model EMA is broad, and I believe this paper make a nice contribution behind it.

Weaknesses

In figure 6 with subfigure progressive scaling kappa = 6, after applying the progressive scaling, both purple (without EMA scaling) and orange line (with EMA scaling) performs roughly the same. However, the other subfigures (that varies $\kappa$) in figure 6 show that EMA scaling is helpful besides the progressive scaling. I might misunderstand but could you provide an explanation why $\kappa$=6 would be quite different from $\kappa$=4 and $\kappa$=8? In line 210, section 3.2, could you elaborate more on the intuiton behind the sentence: *Inspecting the train loss leads us to conclude this is due to a breakdown of the Adam Scaling Rule*? Such clarification would be non-trivial as the gap is growing when we increase the scaling factor. An additional ablation study on both momentum coefficient + learning rate: The model EMA momentum coefficient used in this paper is too close to 1. It would be nice to have an ablation study on the EMA scaling rule vs. the optimal scaling factor when we vary the EMA momentum coefficient away from 1. Similarly, we would expect when the learning rate increases, the EMA scaling rule will break down, but it would be nice to quantify such phase change even with a convex task. Minor: In footnote 5, the eqn 68 might actually refer to eqn 74.

Questions

The questions are raised above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors acknowledge that there is still a $O(\eta * \beta_\rho)$ difference for the scaled model with the unscaled one, but didn't have an ablation of the impacts from the learning rate and the impacts from the base momentum coefficient. It would be nice to see such study somewhere.

Reviewer nnfC2023-08-15

Response

I thank the authors for their efforts and clear explanations! My concerns are addressed.

Reviewer dfpr2023-08-15

Response to Rebuttal

I have read the other reviews and the authors' rebuttal. I thank the authors for their responses, and for performing additional experiments in the rebuttal PDF. I raised my score to 7. I think that the EMA scaling rule is interesting and the experiments are good. The writing is still more obfuscated than is necessary, and the paper could benefit from some minor rewriting, which hopefully can be done for the final version.

Reviewer bHsu2023-08-21

Thanks for the rebuttal

I have read the authors' rebuttal and they have addressed all of my concerns. Thanks for the detailed explanation and well-designed experiments! I am willing to raise my score to accept.

Program Chairsdecision2023-09-21

Decision

Accept (spotlight)

© 2026 NYSGPT2525 LLC