Larger language models do in-context learning differently

Author response to Reviewer M5tP

Thank you for the positive review and encouraging feedback. We are glad that the reviewer appreciated our extensive experimentation and insightful findings that shed new light on in-context learning. > Do you think that studying the models of different sizes through the lens of mechanistic interpretability on the introduced ICL variations could provide valuable insight into the difference in behaviour in-between small and large models? Thank you for this interesting suggestion! We agree that studying why these differing behaviors in in-context learning occur is an exciting research direction. While we view this experimentation as out of scope of this work because this work focuses on robustly demonstrating the existence of these differences, we are still excited by the possibility for future work to investigate this direction. As such, we’ve added the following paragraph to the revised version of our manuscript in the “Why are larger models better at in-context learning” subsection of the “frequently asked questions” section of the Appendix. “One way in which future work could gain insight into why larger models are better at in-context learning could be to study models using mechanistic interpretability. For example, analyzing attention and activation patterns could reveal how models of varying sizes process in-context examples.Furthermore, circuit analysis could identify specific mechanisms that might allow larger models to utilize new information over existing priors. Further analysis in this direction could potentially provide insight on how the ability to override priors is implemented and what structural changes enable stronger in-context learning capabilities.”

Author response to Reviewer AN7L

Thank you for the encouraging review - we appreciate your positive comments on the presentation of our results, extensiveness of our experiments, and relevance of our topic. We’ve revised the manuscript according to your feedback - please let us know if you have any further comments. > The title is somewhat of an overstatement. The authors provide convincing evidence that larger models are better at ICL, but they do not provide evidence that these models do something fundamentally different from their smaller counterparts. Thank you for this insightful comment on our manuscript title. We agree with your point and have revised the title to “Larger language models are better in-context learners” which we believe accurately reflects the phenomena that language models seem to be more capable at learning from in-context examples at larger scales. Note that while we are able to make this modification to the revised manuscript, the revision process does not allow us to change the title of the submission itself, only the title in the paper. > In section 6, the authors perform the ICL linear classification experiment. This is an interesting experiment, but most of the results are relegated to the appendix (Section D.4). Since the appendix section is relatively small, and the authors have another page to spare, I would suggest moving the results to the main paper. Thank you for pointing out this detail and we are glad that you found the experiment interesting. Given the available space, we agree that we can move the rest of the linear classification results to the main paper and have made this revision in the manuscript accordingly. > While the authors perform a comprehensive study of the phenomenon (which is extremely valuable), they draw very few conclusions from the experiments. It would have been interesting to see a more in-depth analysis of WHY larger models are less tied to their semantic priors, or WHY instruct-tuned models become more tied to their semantic priors. [...] To be fair, these additions may be slightly out of scope for the paper. Thank you for this important feedback on analyzing the causes behind the phenomena that we observe in our work. We agree with you that this is an important question to understand, but that it is slightly out of scope of our paper. We primarily view our paper as providing extensive and robust evidence that this phenomena **exists**, rather than attempting to also explain **why** it occurs. As such, we’ve included discussion on this in the “Why are larger models better at in-context learning” subsection of the “Frequently asked questions” section of the Appendix. We are excited to see future work further delve into analysis on why the behaviors shown in our work occur.

Author response to Reviewer Cucr

Thank you for the thoughtful feedback. We are excited that the reviewer found our experiments to be extensive and to provide robust evidence for how in-context learning relates to model scale. > Is there any way to disentangle a model’s robustness to noisy input from reliance on semantic priors? The settings where greater than 0 but fewer than 100% of example labels are flipped might be interpreted in this way: The models learn the “true problem” they must solve from the regular examples and treat the flipped label examples as noise. One option would be to include an experiment comparing a model given N regular examples vs one given N regular and M flipped examples for various N and M. Thank you for this interesting comment. We agree that there is some subtlety in that flipping some labels may suggest to the model that the labels are noisy, rather than indicating to the model that it should start following those flipped labels. We believe, however, that this is still a function of reliance on semantic priors because models must decide when they no longer believe (a) that the original labels are correct and the inputs are just noisy and instead believe (b) that the original labels have actually been flipped, and the inputs are noisy in the other direction. This decision point is a function of the strength to which the model relies on its semantic priors, because a model that very heavily relies on its priors might treat a large number of flipped labels as “noise” before overriding its priors. This indicates to us that a model’s reliance on semantic priors directly affects its robustness to “noisy” input by influencing what the model considers “noise”. Please let us know if this helps clarify this nuance - we are happy to iterate more and are open to feedback/revising the paper if this is still unclear or if we’ve modeled this nuance incorrectly.

Author response to Reviewer 5sFD (1/2)

Thank you for these insightful questions. We are happy to see that you appreciated our extensive experimentation and analysis on instruction-tuned models. We revised our manuscript and added discussion following your suggestions. Please feel free to let us know if you have additional questions! > Could the behavior of overriding semantic prior be acquired not only by using larger models but also by using better training techniques and corpus? I’m curious if the conclusions about the sizes would still stand in the newly trained LLMs with better structures and training corpus. [...] Thank you for this important question. In our work, we investigated several model families to study how in-context learning behaviors differ across model scale. Namely, each model in the LLM model family is trained on the same training data and uses the same training protocol; the only difference is model size. This means that our findings on the LLM and IT-LLM models show the effect of purely scaling model parameter count.On the other hand, we also used GPT-3, InstructGPT, and Codex models; for these families, it is not publicly known if the models within each family are only scaling in terms of parameter count. It is likely that there are other scaling factors (e.g., better training data, a better model architecture) that increase the scale of that model. For example, code-davinci-001 and code-davinci-002 may actually have the same number of parameters, but code-davinci-002 could be trained on better data. In our paper, we assume that for these model families, the models are increasing in scale for some scaling factor, not just parameter count. Our experiments show that in-context learning behavior often changes with respect to model scale for all model families. This means that (a) purely scaling parameter count can result in the behavior of overriding semantic priors and (b) models that are larger for scaling factors other than parametercount (e.g., quality of training data) still exhibit the behavior of being able to override semantic priors when performing in-context learning. We can see (a) because the behavior exists in the LLM model family, for which models only differ by parameter count. We can see (b) because the behavior exists on the GPT-3, InstructGPT, and Codex model families, for which models do not necessarily only differ by parameter count (although we cannot confirm which exact factors are contributing because this information is not publicly known). We hope this helps answer your question. We’ve added this discussion in the “Frequently Asked Questions” section of the Appendix in the revised manuscript. > Are there any methods to control the semantic-prior overriding behavior in different cases (e.g., could we make larger models perform less in this way or vice versa? )? [...] I think a section discussing how to control the behavior of overriding the semantic priors [...] could make this paper stronger [...]. Thanks for this insightful question on how one might be able to control the behavior of overriding semantic priors when performing in-context learning. We agree that this is an important point of discussion, and as such, we have added the following discussion into a new subsection titled “How can one control in-context learning behaviors?” in the “Frequently-asked questions” section of the Appendix. “The behavior of overriding semantic priors can be both beneficial (e.g., teaching the model an updated fact) and harmful (leaving the model susceptible to false knowledge in a prompt). An open question is thus how one might be able to control the behavior of overriding semantic priors in language models. One example of a harmful side effect of language model’s susceptibility to override priors when presented with in-context examples is many-shot jailbreaking (Anil et al., 2024), where a large number of in-context examples of unsafe dialogues are presented to a language model in order to override its prior safety training. Anil et al. (2024) found that a useful intervention to prevent this strategy of jailbreaking is to finetune the language model on examples where the model follows its prior knowledge and ignores demonstrations given in-context. These findings suggest that some control over the extent to which large language models override priors with in-context examples can be gained via supervised finetuning. For example, if one desires a large language model that always conforms to its prior knowledge, supervised finetuning on examples where the final answer is independent of the few-shot examples may reduce the model’s tendency to override prior knowledge when shown in-context examples.”

Author response to Reviewer 5sFD (2/2)

> As mentioned in the limitations, more experiments on the generation tasks could make this paper stronger. One possible way would be inserting wrong/different facts from the semantic prior and see if the model would respond based on the newly inserted facts. Thanks for this neat suggestion! While we view generative tasks as out of scope for our findings because of the difficulties with evaluating the model’s answer in a generative setting and whether this type of injection of facts would work for non-instruction-tuned models (which may not understand that they are supposed to try to learn from the facts). As such, we’ve added your experimental suggestion into our existing “Would these findings translate to generative tasks” subsection of the “Frequently asked questions” section in the Appendix.

Author response to Reviewer J5hP

Thank you for this important review - we are glad that you found our paper interesting and appreciated our extensive experimentation. We revised the paper based on the feedback you gave. Please let us know if you have any further comments or suggestions. > The paper does not expose scientific novelty. Although the experiments and the (slightly crude) discussion are good, these experiments or something similar has been presented here before: https://neurips.cc/virtual/2023/76728.I would be grateful to the authors if they could highlight the new paper's substantial improvements. Thanks for bringing up this related work. We are happy to modify our paper to discuss this work and how it relates to our own findings. In our revision, we’ve added discussion on this work’s findings into our related work section. We believe that there is an important distinction between the work from Shi et al. and our findings; while both our work and Shi et al. examine the behavior of how larger language models perform in-context learning differently, our work focuses on **empirically** demonstrating this phenomenon across many task setups and model families, whereas Shi et al. focuses on theorizing about **why** this phenomenon occurs. Because our findings robustly demonstrate that this behavior **occurs** in the first place and Shi et al. provide explanations for **why** this behavior occurs, we are confident that our findings provide substantial novelty with respect to Shi et al.’s work. Here’s our addition to the “Related Work” section in the revised manuscript. We’ve aimed to reflect the important distinction between empirical demonstration of a behavior and theoretical explanations for why the behavior occurs. We welcome any feedback on how to best frame these differences and are open to suggestions. “Our work makes similar claims about the ability for language models to learn tasks via input--label mappings only, though it differs crucially in that we observe frozen pretrained transformers without any additional learning. Additionally, our work focuses on empirically demonstrating that larger language models are better at learning input--label mappings in-context. Related work from Shi et al. (2024) provides theoretical explanations for this phenomenon, proposing that when performing in-context learning, larger language models cover more hidden features, whereas smaller language models emphasize important hidden features.” We’ve also added a reference to this work in the “Why are larger language models better at in-context learning” subsection of the “Frequently asked questions” section of the Appendix in the revised manuscript.