Scaling Data-Constrained Language Models

Summary

This paper studies an important empirical question: what is the optimal tradeoff between LLM size and the amount of data available for training it. The authors follow an existing line of work (e.g., Chinchilla), and additionally consider the role of data repetition (via multiple epochs), e.g., what is the effect of training a model for K epochs on N tokens each, compared to training it for one epoch on K*N unique tokens. The authors adjust the power law equation introduced in Chinchilla to take into account the “effective data”, which considers the lower value of repeated tokens, and similarly the “effective model size”, which considers the lower value of parameters trained on such data. They present an overwhelmingly large set of experiments, some VERY expensive (the total amount of compute spent on this work is 4.5 million GPU hours). This makes this study practically irreproducible for 99.999% of the community on the one hand, but very valuable still to those training such huge models. The conclusions reached by the authors are interesting: e.g., models can be trained on data that is repeated for up to 4 times, and reach a similar loss as models trained on a similar amount of unique tokens. They also study the effect of augmenting text corpora with code, and applying deduplication filtering steps.

Strengths

- An important empirical question. - A potentially more accurate modeling of the params/data scaling law. - A very large set of experiments covering a range of models and scenarios. - Some interesting and non-trivial observations. - Paper is generally well written

Weaknesses

- There is something unclear about using the validation or the test loss. The authors say (#42) that they are reporting test loss, but many of the tables and graphs say that they are showing validation loss. Which one is it? Reporting test loss is a problem, as it might lead to overfitting the test data (e.g., by others who want to adapt the proposed recipes). - The loss differences are often very small. E.g., in Fig1 (right), a difference of 0.006, or in Fig5 (left). Are these results significant?

Questions

- I am not sure I entirely understand the data repetition setup. In particular, the caption of fig2 says “Training runs with different epochs reuse subsets of the same data to ensure different training data is not a confounding factor.” What does subsets of the same data mean? Is the data sampled from the training data in each repetition? Doesn’t that mean that the data is not effectively repeated? Typos and such: - #132-133: I think R_D and U_D in these lines should be R_N and U_N? - #147: [65]partir

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

N/A

Summary

This paper demonstrates the scaling law both mathematically and experimentally in data-constrained situations. They analyze three experimental protocols: allocation (fixed unique data), return (fixed FLOPs), and parametric fit, and obtain new conclusions that were not found in previous research. For example, they demonstrate that increasing the data repetition (epochs) is more effective than increasing the parameters when the number of unique tokens is fixed, and the performance drop is shown to be negligible when there is a slight decrease in unique data within a predetermined number of FLOPs. They also analyze the effects of code data augmentation and data filtering strategies that can be applied in data-constrained situations.

Strengths

- The research is well-motivated and necessary. - They present scaling laws and propose optimal training strategies in a situation where data is limited. - They derive reliable and generalizable conclusions through extensive experiments.

Weaknesses

- Increasing the maximum parameter size (e.g. >= 10B in Chinchilla [1]) would further enhance comprehension of the scaling law in large language models. [1] Hoffmann, Jordan, et al. "An empirical analysis of compute-optimal large language model training." Advances in Neural Information Processing Systems 35 (2022): 30016-30030.

Questions

Since many large language models utilize IF (instruction fine-tuning) and RLHF (reinforcement learning from human feedback), do authors have plans to expand research on scaling laws of data-constrained language models in the IF or RLHF scenario, where the data includes instruction or human feedback data?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors adequately suggest limitations in various aspects.

Summary

This paper investigates scaling properties of languages, in the presence of *repeated data*: plenty of work has studied the scaling properties of LLMs (and ML architectures in general), both with respect to the number of parameters and amount of data, but they generally assume each new datapoint is unique. However, as newer LLMs start to reach the limit of available (internet) data, repeated use of the same datapoints sets will become more common and understanding the "worth" of a repeated datapoint is a very important research question. The authors conduct extensive experiments, training hundreds of model ranging from 10 million to 9 billion parameters and trained up for up to 1500 (repeated) epochs. They then propose a novel *data-constrained* scaling law by introducing a multiplicative factor the data term that decays with the number of repetitions and (briefly) validate it, showing a better fit than the typical Chinchilla scaling law in the presence of repeated data. They then use their proposed scaling to study two questions: 1. *Allocation: What is the optimal balance of (compute/data) resources?* Their results suggest that its better to first scale epochs/number of repetions, and only then parameter count: by repeating data, they are able to train a comparable model (in terms of performance and compute budge) with 27% less parameters. 2. *Return: What is the expected value of additional resources?* They discover that while there are diminishing returns to adding more compute resources in the presence of the same training dataset, there's a notable value in repeating data: training up to 4 epochs yields almost the same loss as training on unique data, but beyond that the value of adding compute resources to train further effectively drops to zero. Lastly, they explore several methods to address data limitations without the need to produce new natural language data. They found that incorporating code tokens into training data yielded a two-fold increase in effective tokens, even for tasks solely evaluating natural language. They also explored varying data filtering techniques, concluding that data filtering primarily yields substantial benefits on noisy datasets, and not so much on cleaner ones.

Strengths

This was very interesting read, and is quite well written. I think this work has the potential to have big impact in the LLM pretraining community: the limits of unique data we can get from the internet are already being reached, but the findings in this paper suggest that won’t necessarily be a problem for a while, and that repeating tokens a few times to obtain an optimal capacity-data compute allocation is probably fine. The experiments on filtering even hint that it might be better to even be more aggressive in current filtering and repeating the cleaner tokens. Their evaluation on Section 7 is also quite strong, using downstream performance, and the findings with code are insighful and also super relevant to the pretraining community, and further confirm “implicit” knowledge in the community.

Weaknesses

The only slightly more serious flaw I see is that the proposed scaling law is not well validated: from what I see, in the main paper only a single plot showing the fit is shown, and no comparable fitting metrics (like r2), commonly reported in scaling law papers, are given to quantitatively compare to the unique-data scaling laws. I think the paper could also be improved by making the theoretical discussion more concise and moving some of the (very interesting) findings from the appendix to the main text.

Questions

N/A

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

See above

Summary

This paper studies the scaling behavior of language models by repeating the training data to multiple epochs. The authors extend the recent successful Chinchilla scaling law to a data-constrained regime by adding exponential decay of data and model terms then fit the empirical observations. The key takeaway is that repeating data up to 4 times yields negligible changes to loss compared to having unique data, consistent with the observation of Galactica. The authors further studies additional methods like adding code data to 50% and find out this does not compromise model performance on text tasks.

Strengths

This paper studies a very important problem (what should one do to further scale up data when data is used up) and offers very practical advice (repeating the data 4 times and using code 50% to the training data). I believe this paper will make a clear impact in the area of language models and general AI.

Weaknesses

Although this paper has clear advantages in terms of practical guidance, I am concerned about the following weakness: The choice of the specific parametric form in equation 5 and 6 are not clearly explained and may be subject to debate, specifically: - Why the decay in equation 5 should be exponential (other than, say, polynomial) as when data points are small, exponential may look similar to polynomial (or other functional forms). - Why the decay in model scale (Eq. 6) should follow the same form as data (Eq. 5). For the original Chinchilla scaling law, the two terms taking symmetric forms are understandable due to their nature. But for repeating data, model parameter and data scale are not as symmetric as unique data setting, thus considering the same form of decay as data (though I do believe there should be a certain form of decay in model scale) might be less justified Different types of data may need different levels of repetition. - The Chinchilla model mostly considers webpages v.s. the Galactica model mostly consider academic papers. Intuitively, the level of complexity of webpages might be lower than papers, and one may want to repeat the complex data over simple ones. The conclusions hold for smaller models that are less than 10B may not hold for models that are larger than 65B - This is actually my largest concern with regard to this paper. Empirically, at least for fine-tuning, people have observed that larger models do not require as many fine-tuning epochs as smaller models. In general, models that are smaller than 10B has different behaviors than models larger than 65B - An alternative explanation for the repeating 4 times conclusion drawn by this paper could be: smaller models do not have enough capability to fit the data with only one pass, but larger models may be able to do so. The conclusion holds for data scale that is less than 100B may not hold for data scale that is larger than 1T. - Most of the experiments in this paper consider training models with less than 100B data. In the current practice, models (even if they are just 7B) are mostly trained with 1T data (either code or text). This means that in practice, the problem is whether repeating 1T data 4 times, rather than repeating 10B data 4 times. - As the data becomes large, a possible model behavior could be: with 10B data, because data is relatively small, repeating is beneficial, yet when data becomes large, repeating may not be as beneficial as the small data regime.

Questions

The authors reported that WebNLG and BaBI immediately improveSee about discussions as one mix in code data. I am curious about what kind of synergy between text and code data and what are the families of text tasks that would benefit the most from mixing code?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

See above discussion about weakness

Response

> If you have specific suggestions, we are open to revising the text based on them. I guess my (lazy) suggestion is just arxiv'ing a longer (8+) pages :)