Algorithmic progress in language models

Summary

The authors of this paper examine the performance improvements of language models over the past decade, and investigate how much of it can be attributed to algorithmic improvements of language models.

Strengths

I note here that, given that this paper proposes a method to evaluate the historical progress of language models, I believe that the primary goal for the paper would be to provide interesting insights for the field and outline current open questions. As such, I will structure the rest of my review with that in mind. - The topic of language models is more central than ever to the broad NeurIPS community, and I think that analysis on the historical progress of the field is of great interest. - The authors obtain valuable insight on the cause of language model improvements over the years. I find their conclusion that data/compute scaling has been a major driving force for improvement over the past few years interesting, if somewhat expected. - The result on the importance of the transformer is also interesting, and provides a nice retrospective justification of the broad adoption of this architecture. - The analysis of data from past works is also sound.

Weaknesses

- The most crucial weakness of this paper is that, despite the analysis of past trends, it does not provide clear insights or suggestions on where the field should direct its efforts, moving on. The authors acknowledge this limitation, but it seems to me that such discussion Is crucial, in order for this paper to be of interest to the community. - It is also not clear to me how the insights provided by the paper could extrapolate in the future. While there has been a lot of effort and improvements gained in performance from data scaling, this is not something that can reliably keep on - at some point, sources of data and compute limitations catch up. As such, it is not clear to me how the insight that the effective compute for language models doubles every set period of time can be useful for many years down the line. - On more minor note, I think some points can be improved for clarity of presentation: - The captions of Figures 1a and 1b are joined together - some spacing between them is required. - It would be better if the authors clarified the statistical models used in the main paper, rather than in the appendix. Overall, my key concern for the paper is that it does not provide enough insight for future directions.

Questions

I would be grateful if the authors could expand on the points I raised above, regarding the insight for future directions.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations of their work. Regarding negative societal impact, I do not foresee any arising from this work.

Summary

This paper investigates the rate of algorithmic progress on the task of language modeling, using a dataset of over 200 LM evaluations on WikiText and Penn Treebank between 2012-2023. The authors fit an augmented scaling law to the data and show that the models are requiring 2x less compute roughly every eight months -- a rate which supersedes Moore's law. Further, the authors find that more recent performance gains have been primarily due to compute scaling and that the contribution of the transformer architecture is roughly equivalent to two years of algorithmic progress in the field.

Strengths

1. This paper presents an interesting take on quantifying the progress on the task of language modeling using an analysis of data collected from over 200 model evaluations in the past 10-11 years. 2. The authors clearly specific the assumptions made in conducting the analysis, which makes the approach quite readable.

Weaknesses

1. There a number of assumptions made to elicit a quantification of algorithmic progress, and the nature of the task itself necessitates those assumptions be made. However, this also means that the resulting analysis is brittle and as such the technical contributions of the work don't rise to the level of a solid contribution.

Questions

1. Can you give examples of the ways in which the core results would change with different scaling law formulations are used?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The limitations were adequately addressed.

Summary

This paper presents an analysis of the relative contribution of algorithmic progress to overall LM performance gains over the window of 2012-2023. The authors evaluate a large number of potential equation variants for modeling algorithmic progress using leave-one-out CV. By making use of defined notions of effective data/parameters/computer, the authors estimate that 2x less compute is needed to acheive the same performance roughly every 8 months. The authors find that roughly 2/3 of the scaling progress is due to the scaling of physical compute resources, with the remainder being attributed to algorithmic progress. The singular contribution of the Transformer architectures is individually analyzed. Thorough analysis of the techniques applied and their limitations are presented.

Strengths

The paper is clearly written and well-presented. The depth and quality of the anlaysis are exemplary. The methodology of analysis is not highly original, but its application to algorithmic progress is a novel and useful contribution to the community. The limitations of this kind of analysis are well-discussed, which is a useful contribution in its own right.

Weaknesses

Nits: Figure 2: label the y axes, consider making especially the right plot a bit larger/more readable, the text is very small Fig 1: fix the spacing between the (a) and (b) captions, they are nearly overlapping

Questions

None.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The limitations are thoroughly discussed in the paper. No relevant missing potential negative social impacts.

Summary

This paper breaks down the driving force behind language models into two factors: scaling up the amount of compute used to train the models and algorithmic innovations. A statistical model, similar to the scaling law, is built and fitted to analyze the contributions of these two factors. The paper claims that models with the same performance level require half the compute every eight months, reflecting algorithmic improvement. The authors also find that the majority of scaling progress is due to the scaling of physical compute resources rather than algorithmic progress.

Strengths

The paper presents a very interesting and innovative approach to quantifying the algorithmic progress of language models. By covering over 200 language models from the past decade, the empirical foundation for the conclusions drawn is solid. Additionally, the author has performed extensive robustness checks to ensure the validity of their core results.

Weaknesses

While the paper makes several assumptions to quantify algorithmic progress (including the extra coefficients in the augmented scaling law), these assumptions and the many degrees of freedom undermine the robustness of the proposed model and the conclusions drawn. I have doubts about whether the statistical model built can inform future progress in language modeling. Minor issue: The caption of Figure 1 seems to be incorrect. It should address Figures 1a and 1b instead of 3a and 3b.

Questions

None

Rating

6

Confidence

3

Soundness

4

Presentation

2

Contribution

3

Limitations

I do not see any potential negative social impact of this work.

Thank you for your comments.

I would like to thank you for your response to my review. I understand better now how the findings of the paper can relate to future work. I believe that further highlighting the above points in the main paper will increase the paper's impact. Given that this was my main concern, I am raising my score.