Are Emergent Abilities of Large Language Models a Mirage?

Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance. We present our alternative explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities; (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show to choose metrics to produce never-before-seen seemingly emergent abilities in multiple vision tasks across diverse deep networks. Via all three analyses, we provide evidence that alleged emergent abilities evaporate with different metrics or with better statistics, and may not be a fundamental property of scaling AI models.

Paper

References (39)

Scroll for more · 27 remaining

Similar papers

Peer review

Reviewer ccWP7/10 · confidence 4/52023-07-05

Summary

The authors argue that purported emergent abilities of language models are mostly explained by the use of specific metrics, including non-linear or discontinuous scoring, and evaporate when linear or continuous scoring metrics are applied. The authors (1) present a toy model to explain emergence, (2) demonstrate that most abilities previously reported as emergent no longer appear emergent under an alternative metric, and then synthetically cause the appearance of emergence in an image reconstruction task.

Strengths

- Simply and clearly addresses the enigmatic phenomenon of emergence - Formalizes and quantitatively tests the hypothesis of [1] that emergence is due in part to metric - Compelling toy model to illustrate how emergence can appear with non-linear metric (accuracy) but vanishes with linear metric (token edit distance) - Convincingly demonstrates that emergence is (at least often) a result of the metric

Weaknesses

- As the authors point out in their related work, the idea that emergence appears due to the metric is not novel, and was proposed by [1], so the novelty of this paper lies in the quantitative evaluation of this hypothesis - There has been work on examining emergence and progress measures in similar contexts, e.g. grokking, which should be addressed in the related work [2–5] - There was no discussion of the relationship to alternative hypotheses. The authors mention in the related work multiple papers that suggest that abilities do, in fact, emerge with scale, but the authors should explain how their hypothesis relates - There was no discussion of the BLEU score metric, even though there are many tasks that have a very high score with this metric - Minor: a lot of the figures are small and therefore hard to read [1] Srivastava, Aarohi, et al. "Beyond the imitation game: Quantifying and extrapolating the capabilities of language models." arXiv preprint arXiv:2206.04615 (2022). [2] Power, Alethea, et al. "Grokking: Generalization beyond overfitting on small algorithmic datasets." arXiv preprint arXiv:2201.02177 (2022). [3] Liu, Ziming, Eric J. Michaud, and Max Tegmark. "Omnigrok: Grokking beyond algorithmic data." arXiv preprint arXiv:2210.01117 (2022). [4] Nanda, Neel, et al. "Progress measures for grokking via mechanistic interpretability." arXiv preprint arXiv:2301.05217 (2023). [5] Barak, Boaz, et al. "Hidden progress in deep learning: Sgd learns parities near the computational limit." Advances in Neural Information Processing Systems 35 (2022): 21750-21764.

Questions

See weaknesses

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors adequately address the limitations.

Reviewer eK8N8/10 · confidence 3/52023-07-06

Summary

This paper aims to debunk a possibly myth about the emergent properties of language models with regard to scale. The emergent behaviors are defined as abilities that are not demonstrated with smaller models, but emerge suddenly with an increase in the model capacity, and cannot be explained by smoothly extrapolating from the model capacity -- i.e., emergent abilities are sharp and unpredictable. The claim that the authors seek to validate is that emergent abilities are actually artifacts of the fact that the choice of evaluation metric, and that continuous metrics (instead of discrete) show smooth increases in performance with scale -- i.e., that "emergent" abilities are in fact smooth and predictable.

Strengths

This paper has many strengths: - First off, it is well-written. - The paper is timely -- large language models have only become moreso in the center of attention in the past year with abundant claims surrounding emergent abilities. - The claim that discrete metrics are a source of so-called "emergence" of new abilities is reasonable and makes intuitive sense. - The empirical evaluation is thorough, with evaluations of GPT-3/InstructGPT models, a meta-analysis using BigBench, and using discrete metrics to artificially induce "emergent behavior" in existing vision models.

Weaknesses

I may have misunderstood parts of the argument, but I am not entirely convinced that emergent abilities might not be real, when it is clear that they are real with respect to discrete metrics (that often have practical interpretations). The paper seems to question the basic premise of emergent abilities, i.e., whether or not they are real. The authors cite the following definition early on -- emergent abilities are "abilities that are not present in smaller-scale models but are present in large-scale models; thus they cannot be predicted by simply extrapolating the performance improvements on smaller-scale models,” which itself depends on an evaluation metric. Since the definition depends on an evaluation metric, why does changing the metric make emergent abilities a mirage? Shouldn't emergent abilities only hold, then, with respect to a metric? Per my understanding of this work, it seems to me that incorrect conclusions have been drawn in prior work by *only* considering discrete evaluation metrics, which is not the full picture -- pointing this out is the main contribution of this paper, and this is an important point to make. However, it still seems that these properties are real with respect to discrete metrics, and these discrete metrics are often desirable. Does this make the choice of metric bad, and emergence a mirage as a result? Or does this render prior analyses incomplete, since they do not consider metrics in which performance **does** smoothly change?

Questions

- Why are smooth evaluation metrics necessarily desirable if they are sometimes a relaxation of the real objective, which may actually be discrete (e.g. the 0-1 loss)? - NOTE: please switch to the NeurIPS 2023 template, I noticed that you are using the 2022 version of the template.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors address this.

Reviewer GXCv9/10 · confidence 4/52023-07-06

Summary

This paper shows that the widely-discussed phenomenon of "emergent abilities" in large language models is largely due to the choice of (discontinuous/nonlinear) metrics and underpowered analyses.

Strengths

- Well-written paper studying an important/commonly discussed phenomenon - Comprehensive, well-designed, and convincing experiments + analysis

Weaknesses

No major weaknesses - the paper was a pleasure to read!

Questions

- l227: Not sure what is meant by "multiple controls" - is the implication that the community should have more aggregate / multi-metric benchmarks like BIG-Bench? - comment: I found myself wanting to compare Fig 4 side-by-side with Fig 3 to understand the change from additional test data, but it was difficult to do so (especially with the diff y-axis scaling).

Rating

9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes

Reviewer Cfbn7/10 · confidence 4/52023-07-24

Summary

The paper shows that the apparent suddenness with which capabilities of language models seem to emerge with scale may be an artifact of the metrics by which they are measured. Theoretically, emergence should be more gradual if the metric scales linearly with log loss on some test set. Three sources of empirical evidence are used to support this claim. First, the authors study the scaling of the GPT-3 family on arithmetic tasks and show directly that emergence is smoother when a linear metric is used. Second, they conduct a meta-analysis of BIG-Bench, and show that gradual emergence only occurs with non-linear metrics, and that the effect is reduced when switching to a linear metric. Third, they artificially induce sudden emergence by using non-linear metrics for autoencoders on CIFAR100 and for transformers on Omniglot.

Strengths

The basic point made by the paper is an important one that has not been fully appreciated by researchers. It is something that everyone studying scaling laws should understand and incorporate into their future work.

Weaknesses

I think there are notable weaknesses in the empirical evidence given to support the basic point of the paper, and insufficient acknowledgement of the limitations of the evidence and other possibilities that could still give rise to sudden emergence. On the empirical evidence: - The main direct evidence that previous claims of sudden emergence are an artifact of the metric comes from studying those previous claims directly, which is done in Section 3. However, the evidence here is somewhat inconclusive: for example for 2-digit multiplication with target str len 4 (which is presumably most 2-digit multiplications?) the jumps from 350M->1.3B->6.7B are gradual while there is a much bigger jump from 6.7B->175B. Moreover, this bigger jump is not predicted by the theoretical model. So it seems like there is still some relatively sudden emergence (even though it is reduced by using the right metric). - The BIG-Bench analysis is helpful, but again on logical_args it seems like there may still be an unpredictable jump. And on the other tasks, the emergence is not that sudden even with the original metric. - The induced emergence experiments are also helpful, but they show that sudden emergence can be induced by using a non-linear metric, not that it is always induced by a using a non-linear metric. So this evidence is more indirect. - Overall I think the paper would have come to more correct (and probably more nuanced) conclusions if it had studied all of the tasks from Figure 1, where sudden emergence had previously been claimed, in detail, instead of studying just one of these tasks combined with other, weaker sources of evidence. - Nevertheless, for an initial exploration of the topic, I think combining multiple source of evidence, even if somewhat weak, is helpful, and it is reasonable to defer a more careful analysis to future work. On other theoretical possibilities: - There is no guarantee that because validation loss scales smoothly, cross-entropy loss on a specific sub-distribution must also scale smoothly. This depends on how the model allocates capacity to that sub-distribution. Therefore there is still an important theoretical possibility of sudden emergence, even with a linear metric. Indeed, this may explain why the data did not match the theoretical model in Section 3. - In some cases, it may not be possible to construct a linear metric. For example, for program synthesis, whether the program passes the unit tests cannot be easily converted to something like edit distance or log loss. - It is argued that log accuracy can be used in this case. But this is not necessarily feasible. For example, if current models pass unit tests with probability 1 in a trillion, it is not feasible to measure this accurately enough to perform any useful extrapolation. Therefore there may be sudden emergence for all practical purposes. Overall, I thought that the evidence was strong enough to support a conclusion that sudden emergence is partially explained by the choice of metric, and that researchers should prefer metrics that scale linearly with log loss when available. I also think that this conclusion is itself enough for the paper to be an important contribution. But I think the paper should take more care not to overstate its claims and acknowledge the limitations of the evidence and, space permitting, other theoretical possibilities. In particular, claims like "metric choice is likely wholly responsible for emergent abilities" (p. 9 line 204) are not justified and should be removed or reworded. I am recommending acceptance of this paper because of the importance of the contribution, but I think that some effort should be made to revise the paper prior to acceptance. However, I am confident in the ability of the authors to do this. Some presentation points: - I think it would be clearer to talk about "sudden emergence" and "gradual emergence" rather than "emergence" and "no emergence", since "emergence" can also be interpreted as "appearing despite not being explicitly trained for". However, I understand that the current terminology is taken from the existing literature and the paper cannot be blamed for this. - I suggest changing the title of the paper, because the current title can be misinterpreted as asking whether the _abilities_ are mirages, rather than whether the emergence (or sudden emergence as I prefer to put it) is a mirage. Any researcher who reads the abstract should not be misled, but unfortunately provocative titles that can easily be misinterpreted can be picked up on by the media and end up misleading the public. However, I acknowledge that choosing titles is hard and no title is perfect. More minor points: - p. 4 line 78 - I am not familiar with this Delta notation (or I have a vague recollection of it). It is clear from the context so probably fine, but consider alternative notation/clarifying the definition. - p. 5 line 115 - Why choose this task from among all the tasks in Figure 1, and why 2-shot / 4-digit? (Would be even nicer to include results on other tasks/other numbers of digits, but absent that, it would be good to clarify how you chose, to make it clear it was not cherry-picked.) - Figure 3: Target str len 1: how do you get a 1-digit answer from multiplying two 2-digit numbers? I assume you are counting things like "01 * 01"? Maybe clarify. - Figure 3: I think it would be clearer to include titles at the top of each column saying "Mathematical Model", "2-Integer 2-Digit Multiplication Task", "2-Integer 4-Digit Addition Task", instead of putting this in the caption. That way people can easily see at a glance what each plot is showing, without having to dig into the caption. - Figure 4: Same comment. - Figure 5: It would be nicer to group the metrics based on their mathematical form. E.g. it is unclear what "gender_bias_score" is. This also might help spot if there are more examples with emergence that didn't have high emergence score. - Figure 5B is a bit redundant. Maybe just point out in figure 5A which of the 5 did not count as emergence. It's also a bit unclear why this didn't count despite the emergence score. - p. 9 line 209 - I don't think this work really contradicts broken neural scaling laws. Even if sudden emergence goes away when you change the metric, one might still be interested in forecasting the original metric.

Questions

No questions other than those given above

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

4 excellent

Limitations

No limitations other than those given above

Reviewer GXCv2023-08-10

Acknowledgment

I've read the rebuttal response — thanks for answering my questions!

Reviewer ccWP2023-08-11

Thanks for the response. > However, respectfully, we feel the reviewer undervalues our paper’s novelty & contributions. I did not mean to suggest that this paper does not have many valuable contributions! I wanted only to point out that the idea that the fact that emergence-looking behavior can arise from a choice of metric is not new. In particular, in addition to the hypothesis of Srivastava et al. 2022, the work by Nanda et al. 2023 suggests that grokking arises similarly from the choice of metric, and that under a different metric (e.g., a progress measure, as they call it), the discontinuous jump in performance disappears. For this reason, I do believe that the work in grokking is relevant to your work (at least this paper, possibly others). > Are you requesting that we discuss the differences in greater depth? We would be happy to expound here, as well as in the paper. Yes, exactly, thank you for the further discussion. > We would be happy to put the appendix subsection back in if requested. The answer is that BLEU is also a sharp nonlinear metric. My main concern was that in Figure 5, it appears there are many tasks that leverage the BLEU metric with a high emergence score, suggesting that there may be emergence with the BLEU metric. Since there was no discussion of this in the paper, I was wondering how these tasks fit into your argument. Overall, I was mostly concerned with the discussion of the BLEU metric. With some added discussion of the BLEU metric, I think that the authors would have addressed my comments and I would raise my score.

Authorsrebuttal2023-08-13

Response to Reviewer ccWP's Response

> I did not mean to suggest that this paper does not have many valuable contributions! Ah! We are so sorry - we misunderstood your tone. We apologize for the miscommunication. Thank you for clarifying! > For this reason, I do believe that the work in grokking is relevant to your work (at least this paper, possibly others). Thank you for clarifying! We now better understand what you meant by progress measures, and what relationship you see. We agree that our perspective is similar to Nanda et al. 2023 and we will add an Appendix section relating emergent abilities with grokking, including appropriate citations to the grokking papers you referenced. We will also mention the Appendix section in our Discussion, but we do not foresee having space to discuss the relationship with grokking at length in our Discussion. > Yes, exactly, thank you for the further discussion. Sounds good. We will discuss Caballero et al. 2023 and Michaud et al. 2023 at greater length in our Discussion. > My main concern was that in Figure 5, it appears there are many tasks that leverage the BLEU metric with a high emergence score, suggesting that there may be emergence with the BLEU metric. Since there was no discussion of this in the paper, I was wondering how these tasks fit into your argument. That's a reasonable thought to ponder! To the best of our knowledge, despite our discovery that many BIG-Bench, tasks appear to score highly on the emergence metric under the BLEU metric, we've only seen one claimed BLEU emergent ability on a single task (IPA transliterate). We will explicitly state this in our paper and add our appendix. We will also clarify that the emergence score was introduced by Srivastava et al. 2022 as an approximate proxy for emergent abilities. > Overall, I was mostly concerned with the discussion of the BLEU metric. With some added discussion of the BLEU metric, I think that the authors would have addressed my comments and I would raise my score. We will also add back in our Appendix section explaining the BLEU metric. **Please consider updating your score if you think our agreed changes merit it!** (FYI: NeurIPS this year doesn't permit uploading a revised manuscript during the review process)

Reviewer ccWP2023-08-15

Any chance you could summarize here the content of the Appendix section you are re-adding? Also, regardless of which tasks have been previously claimed as emergent, it appears that are many tasks that have a high emergence score with the BLEU metric and I was wondering if you could comment on that, as well.

Authorsrebuttal2023-08-16

Response to Reviewer ccWP

> Any chance you could summarize here the content of the Appendix section you are re-adding? Of course! A number of changes will be made. These include but are not limited to: 1. In the main text, a new Limitations section 2. In the Discussion section, a longer commentary on the relation of our work to alternative hypotheses 3. In the Appendix, a new section "Relationship between Emergent Abilities & Grokking" 4. In the Appendix, a new subsection on BLEU as a metric > Also, regardless of which tasks have been previously claimed as emergent Let us clearly state that we are uncomfortable disregarding previous claims of emergence. Our paper is an investigation of previous claims, and we strongly do not want to invent & misattribute scientific claims to other researchers. Regarding BLEU, to the best of our knowledge, 3 out of 4 papers on emergent abilities didn't mention BLEU, and the 1 paper that does mention BLEU showed BLEU on 1 task (out of many) and did not emphasize the ability. We are hesitant to comment on BLEU because we don't think many authors are claiming emergent abilities under it. > Also, regardless of which tasks have been previously claimed as emergent, it appears that are many tasks that have a high emergence score with the BLEU metric and I was wondering if you could comment on that, as well. We indeed find high emergent scores under BLEU. However, despite this, the authors who studied BIG-Bench (Srivastava et al. 2022) do not claim that these are emergent abilities. We do not know why. Srivastava & colleagues briefly discuss this in their main text, but defer discussion to "Figure App.4" where they switch from BLEU in the main text to ROUGE-L-Sum in Figure App.4's caption. This is why we included our Appendix A4 on ROUGE-L-Sum to demonstrate how it scales. **We will add another appendix A4 discussing how BLEU scales.** Figure 5's X axis ("Emergence Score", called Breakthrough Score in Srivastava et al. 2022) is a rough proxy for emergent abilities. The authors who introduced the score clearly labeled it a heuristic metric. In our view, the Emergence Score should be viewed as "This _might_ merit further investigation" not "These _are_ emergent abilities." In our opinion, this Emergence Score is better thought of as a necessary but insufficient criterion. If you have specific experiments to suggest regarding BLEU, please do let us know! Lest the point be missed, **the key takeaway from Figure 5's left subfigure is that 34 out of 39 BIG-Bench metrics seem to display no possible signs of emergent abilities.**

Reviewer ccWP2023-08-18

Thanks for the response! > We are hesitant to comment on BLEU because we don't think many authors are claiming emergent abilities under it. I understand that from [33] only IPA transliterate was featured as emergent, but I don't think it's fair to say that authors aren't considering emergent abilities under it, considering that it is featured in their main figure. > We indeed find high emergent scores under BLEU. However, despite this, the authors who studied BIG-Bench (Srivastava et al. 2022) do not claim that these are emergent abilities. We do not know why. I think the fact that you find especially notable emergent scores under BLEU (and because it is only one of five metrics that you observe high scores on) warrants further investigation into the metric, even if prior work does not properly consider the metric. If you're looking for an experiment, one possible idea would be to reproduce something similar to Figure 9 (in A.4) but for the BLEU metric. If BLEU is a sharp metric, great. Even if it is the case that BLEU is not a sharp metric, I don't think this undermines your argument at all, as I understand your argument to be that the appearance of most *but not necessarily all* emergence arises from a particular choice of metric. Nonetheless, I do think that it's important to discuss BLEU in the paper to answer the immediate question that a reader might have when looking at Figure 5 and observing that the most notable metric (tasks with highest emergence scores) is BLEU, and not the other metrics that you do discuss. Considering how notable BLEU appears in Figure 5, I think it also might be worth including at least some discussion of this in the main paper, in addition to your proposed extension of the appendix. Overall, I think conditional on the added discussion of BLEU (regardless of whether BLEU is a sharp metric or not), I have been convinced that this is a solid paper, and so I will increase my score to a 7.

Reviewer Cfbn2023-08-12

Response to rebuttal

Thank you for the rebuttal. I am glad to hear that we are in agreement on several points. I continue to think that this is an important contribution, but also very hard to get comprehensive data on. While additional data would always help, I think the existing data is enough for the current paper and it's more important to be upfront about the limitations of the evidence as it stands. Therefore I am pleased to hear that a limitations section will be added. On the question of accuracy transitioning from ~1e-12 to ~1, I agree with the authors here. I was being a bit hyperbolic, but I think my point still stands: it's not so unusual for a 10x increase in model size (i.e. 100x increase in training compute, by [1]) to improve accuracy by ~30x (see for example 2-Integer 2-Digit Multiplication in the paper). Since compute spending is growing rapidly (see for example [2], which suggests a 10-month doubling time), that means that an ability could go from essentially undetectable (say 1e-3 accuracy, at which level it's hard to tell if it's a fluke) to useful performance (say 1e-1) over the course of a few years, or perhaps faster in some cases. (This isn't really a technical disagreement, it's more a question of how the result that log accuracy scales smoothly should be framed: I'm saying that even if this is true, we should still expect to be surprised by future model capabilities.) [1] https://arxiv.org/abs/2203.15556 [2] https://epochai.org/blog/compute-trends

Authorsrebuttal2023-08-13

Response to Reviewer Cfbn's Comment

Phew! We're so sorry for the dense earlier message and hope that it all made sense! A 6k character response is such a tight limit when a reviewer like yourself provides so much good feedback. > I am glad to hear that we are in agreement on several points. Indeed! We agreed with you on almost all points. I think that being more upfront about our paper's limitations by adding an explicit Limitations section will be an important addition. > I continue to think that this is an important contribution, but also very hard to get comprehensive data on. As a personal comment, an inability to obtain more comprehensive data has been one of the most disheartening aspects of working on this topic. We want to have a better scientific understanding of emergent abilities, but typically, models aren't public, nor are models' text outputs, and obtaining that data ourselves is slow and expensive. > I'm saying that even if this is true, we should still expect to be surprised by future model capabilities. Agreed. Once again, thank you so much for your comments & feedback earlier! It was quite obvious to us that you thought deeply about how our paper could be improved, and we very much value your perspective. We will update our manuscript accordingly. **Please consider updating your score if you think our agreed changes merit it!** (FYI: NeurIPS this year doesn't permit uploading a revised manuscript during the review process)

Reviewer eK8N2023-08-13

Thank you for the clarifications and helping me to understand what I missed in my review! With these addressed, I'm happy to raise my score to an 8. Great work!

Program Chairsdecision2023-09-21

Decision

Accept (oral)

© 2026 NYSGPT2525 LLC