Resolving Discrepancies in Compute-Optimal Scaling of Language Models

Summary

This paper aims to resolve the discrepant conclusion about compute-optimal model size drawn from (Kaplan et al., 2020) and (Hoffmann et al., 2022), two famous scaling-law papers that have guided the exploration of early large language models. The authors concluded that the discrepancies arise from last-layer FLOPS and warmup steps. And contrary to the claims of (Hoffmann et al., 2022), learning rate decay is not essential. As a byproduct, the authors also develop strategies to select hyperparameters, including optimal learning rates and batch sizes, and emphasize the need to use larger beta_2 when training with small batch sizes using Adam.

Strengths

1. This paper targets an interesting and important topic on the discrepancy of scaling laws. Given that scaling laws guide the design decision to train modern large language models, solid results on this topic can confirm appropriate scaling strategies and help consolidate the reliability of scaling laws for applications such as large-scale performance prediction. 2. This paper presents extensive and detailed experiments to show how to align two scaling laws for compute-optimal model sizes. 2. The discussion on hyperparameters selection, including learning rates, batch sizes, warmup durations, and beta_2 of Adam provide helpful guidance on reproducing scaling law results.

Weaknesses

1. Discrepancies in scaling laws indicate concerns about the correctness of scaling laws. Most of the paper discusses how to resolve the discrepancies between two scaling laws but has not fully addressed the concerns about the correctness of the two scaling laws as only limited results (only Fig. 4 with one dataset) show resolving the discrepancies generally leads to accurate prediction in large-scale experiments. 2. The conclusion of this paper is not clear enough. It would be clearer if the authors could conclude a standard pipeline to reproduce the scaling law experiments and enable reliable predictions with this pipeline. 3. This paper focuses on L(C), the compute optimal scaling laws, instead of L(N, D), which is a more general scaling laws that consider both model sizes and training data. Given that the latest large language models are overtrained to reduce inference costs, limiting to the compute-optimal setting may be less impactful.

Questions

1. One of my concerns on scaling laws is that the need for careful hyperparameter selection may impede its applications. Firstly, it makes fitting a law extremely complex, and making mistakes that lead to unreliable predictions is easy. Also, the need to tune hyperparameters increases the costs to fit a law, indicating that we need accurate extrapolation to orders of larger magnitude to pay off the costs. I would like to learn about the authors' comments on potential strategies to stabilize the process and reduce the costs of hyperparameter tuning. 2. Given that many scaling laws exist, and they are discrepant possibly because of different experimental settings or assumptions, could you summarize the condition for the scaling laws in this paper?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

This paper only explores the compute-optimal settings and provides limited extrapolation results based on their findings (only one dataset and only scales up to 3e19 flops).

Summary

This paper studies the discrepancy between the scaling laws for Kaplan et al. and Hoffmann et al. and identifies three factors behind it: last taking last layer cost into account, warmup duration, and tuning optimizer hyperparams with model scale. They show the transition from Kaplan et al. scaling laws before correcting for these effect to Hoffmann et al. scaling laws after correcting for these effects.

Strengths

The paper studies the important question of scaling laws for language models thoroughly (with the limitation of model scale). This fills an important missing piece in the literature and places the scaling law research on a firmer grounding. While running these experiments they also discover additional interesting observations such as the importance of tuning beta2 at smaller batch sizes.

Weaknesses

The main limitation of this work as also discussed by the authors is the scale of the models use (< 1B). A more stronger instance of this limitation is that the paper uses experiments on model of size 5M to 100M to fit scaling laws for learning rate and batch size, it is unclear if these laws would hold up for large models. It would have been useful if the paper did an check of this optimality at a significantly larger scale (~1B, not just 200M). This is less of a concern for the runs in the paper since all of them have model sizes <=1B and since Adam is known to be stable (Wortsman et al. 2023) across a range of learning rates.

Questions

1. Is the sweep described in appendix E done with constant learning rate or cosine decay? 2. While the papers fixing of warmup to be always less than the model size is certainly a step in the right direction as compared to Kaplan et al. is there. a reason to believe that this is optimal? or that the scaling law would not change with change with the warmup duration?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes.

Summary

The paper provides an explanation for the discrepancies between the scaling laws of Kaplan et al. and Hoffman et al. The authors start by reproducing the scaling laws of Kaplan at al. They then introduce incremental changes in the methodology: accounting for the last layer computational cost, setting a more reasonable learning rate warmup, and more thorough scale-dependent hyperparameter optimization. After introducing these changes, they recover the scaling laws of Hoffman et al. The paper provides evidence that careful learning rate decay is not necessary to resolve the differences between these two scaling laws. The paper demonstrates the importance of such scale-dependent hyperparameters for obtaining reliable scaling laws with small computational budgets.

Strengths

I believe the paper to be novel — I am unaware of prior work aiming to explain the differences between the scaling laws of Kaplan et al. and Hoffman et al., beyond the hypothesis put forth by Hoffman et al. which this paper directly addresses. The paper is very clearly written. Its experimental set-up is well-motivated in relation to the prior work of Kaplan et al. and Hoffman et al. The experiments are thorough, and the methodology sound. Beyond the scientific interest of studying why Kaplan et al. and Hoffman et al. arrived at such different scaling law coefficients, this paper provides valuable insights for researchers studying the scaling of language models, particularly regarding the importance of scale-dependent hyper-parameter optimization for small computational regimes.

Weaknesses

Hoffman et al. suggests that their fitted scaling law parameters differ from those of Kaplan et al. because Kaplan et al. include very sub-optimally trained models in their analysis, which bias the fitted parameters. That is, for certain (N, D, L) tuples, the loss L is substantially overestimated. They suggest that this is mainly due to the poor learning rate schedule used by Kaplan et al. Contrary to what the paper under review claims at times, I view this paper as supporting the broad hypothesis of Hoffman et al. rather than disproving it. Starting with training choices similar to those of Kaplan et al., the authors are able to reproduce their scaling law. They then show that improving these training choices — via a more reasonable warmup schedule, batch size, learning etc. — yields a scaling law that more closely resembles that of Hoffman et al. That is, the poorer training hyper parameters of Kaplan et al. explain the difference with Hoffman et al. While Hoffman et al. emphasize the learning rate schedule, I don’t read their paper as claiming that learning rate decay is necessary to obtain their scaling law —as the authors claim at times—, but rather that the poor learning rate scheduler used by Kaplan et al. is sufficient to mess up their derived scaling law. To me, the hypothesis of Hoffman et al. is consistent with the existence of different hyper parameter choices for which similar (N, D, L) is attainable with a constant learning rate — they do not claim that is is not possible. The authors decouple “warmup” from “decay”, whereas to me the learning rate schedule comprises both. Thus, when Hoffman et al. say that they set “the learning rate schedule to approximately match the number of training tokens”, I imagine that they mean adapting both the warmup and the decay period (e.g., to use heuristics such as warmup for 5% of the total training steps). Unfortunately, Hoffman et al. do not give details on what warmup schedule they used. However, looking at Figure A1, it is clear that the warm-up period that they use is a small fraction of the total number of training steps — unlike Kaplan et al. Broadly, I find it misleading to say “Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential” when Hoffman et al. say “Our work differs from Kaplan et al. (2020) in several important ways. First, the authors use a fixed number of training tokens and learning rate schedule for all models. […] In contrast, we find that setting the learning rate schedule to approximately match the number of training tokens results n the best final loss regardless of model size” — notice the difference in language, “learning rate schedule” (which is broadly understood to include also warmup) instead of “learning rate decay”. I ask the authors to please better scope their claim: “Counter to a hypothesis of Hoffmann et al. [21], we find that careful learning rate decay is not essential for the validity of their scaling law” — Hoffman et al. do not claim necessity, and they refer to “learning rate schedule” rather than only learning rate decay. Lastly, the authors may be underestimating the value of learning rate decay, since they only consider models that are reasonably close to Chinchilla-optimal. When considering, for the same model size, very different number of training tokens, cosine decay may play a more important role, at least if all other hyper parameters are kept fixed. The authors are aware of this and discuss it in the limitations section.

Questions

For Section 3.5, do you warmup the learning rate? How so? Hoffman et al. decay learning rate to 10% of peak, why do you use 1% of peak?

Rating

6

Confidence

4

Soundness

2

Presentation

4

Contribution

3

Limitations

The authors adequately discuss the limitations of their work.

Summary

- Identifies and eliminates discrepancies between the scaling laws of Kaplan et al and Hoffman et al via three interventions: accounting for un-embed layer FLOPs, correcting warmup hyperparameters, and tuning optimizer hyperparameters. - Also derives scaling laws for optimal learning rate and batch size as a function of model size

Strengths

- The discrepancy between the scaling laws of Kaplan et al and Hoffman et al has been a major source of confusion/frustration in scaling law research. This paper conclusively identifies + mitigates this discrepancy through multiple targeted interventions. - They also reproduce these results on two datasets, indicating that this discrepancy is not specific to the pretraining dataset. - Section 4 provides complementary findings that are helpful for speeding up scaling law experiments: scaling laws for optimal learning rate and batch size, role of AdamW $\beta_2$, constant learning rate schedule suffices to reproduce Chinchilla scaling laws (albeit with higher loss)

Weaknesses

The main limitation of this paper (as noted in the discussion) is that the range of models considered is fairly small compared to Hoffman et al. due to compute constraints. Nevertheless, it would be useful to have some sanity checks for checking if the estimated scaling laws for loss, learning rate, and batch size extrapolate reliably to 3B and 7B scale models.

Questions

- Why did you opt for the IsoFlop estimation approach (approach 2 in chinchilla) rather than the arguably more direct "approach 3" in the chinchilla paper? - In Appendix B, you mention that the main non-negligible source of error in approximating FLOPs with 6ND is the attention-related computation. This error can be quite non-trivial for small models with large context lengths, but would be small for large models. So, how would the scaling law exponents change if the FLOP estimator accounts for attention flops?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

None

Complete results of batch size sweep

We tabulated the full results of the batch size sweep below: | Batch Size | Validation Loss | |-------------:|:------------------| | 160 | 3.050 ± 0.002 | | 320 | 2.970 ± 0.002 | | 640 | 2.943 ± 0.002 | | 1280 | 2.955 ± 0.002 | | 2560 | 3.013 ± 0.002 | The results show that our predicted batch size is optimal among the values we have tested.