Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. We also find that inference cost increases linearly in the many-shot regime, and frontier LLMs benefit from many-shot ICL to varying degrees. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.
Paper
Similar papers
Peer review
Summary
This paper explores the effectiveness of in-context learning with hundreds to thousands of examples, bringing the number of examples closer to the range one might use for supervised training methods. Experiments are performed on a large number of tasks and benchmarks using Gemini 1.5 as the LLM, in each case giving it a prompt containing varying number of dataset examples and observing its performance as the number of examples increases. These robustly demonstrate the effectiveness of using more examples, at times beating fully-supervised models on the same data. In addition to using the ground truth examples, two additional prompting methods are evaluated: Unsupervised ICL, which only adds additional inputs to the prompts, and Reinforced ICL, which adds inputs along with machine-generated responses filtered by correctness of result; both techniques limit the amount of "ground truth" responses required for adding prompt examples, and are also found to perform well for all tasks.
Strengths
* These experiments provide a very good overview of the effects on performance of including these numbers examples in prompts, surveying the effects in a wide set of tasks. * Additional studies, particuarly sec 4, are interesting, finding some behaviors that parallel those seen in supervised training, in the context of ICL with substantive benchmark tasks. * Unsupervised ICL and Reinforced ICL are useful techniques to limit the amount of gt reponses, and shown to be effective in these evalutaions
Weaknesses
* Only Gemini 1.5 was explored, though this was discussed by the authors in the discussion and limitations section, I think it's a significant weakness as it limits findings of many-shot ICL to describing the behavior of this particular model, not the typical range of performance behaviors for different systems with this approach, and whether these are automatic simply by extending context, or if there are model differences that can impact this technique. * I didn't see anything on the computational costs or runtime differences varying K, which would provide fuller picture of the behavior of many-shot ICL beyond just its end performance. * It's a lot of information to cram into the allotted space and many sections seem terse with the connections between experiments and how they contribute to the overall understanding often left hanging.
Questions
While reading this paper I ended up with lots of comments and questions that I've listed below. Aside from profiling computation increases as mentioned above, there are a lot of particulars around the individual experiments that I had questions about, as well as questions on what might be causes and effects in addition to raw performance numbers. - Including more examples has two effects: One of increasing information and distinct samples used, but another of increasing the context length. When viewed analogously to a training loop, a supervised optimization loop will repeat examples between epochs if there are fewer samples than steps. It would be interesting to start to separate these two effects, perhaps by extending context length by repeating the same examples instead of adding new examples. For example when going from 20 to 200, repeating the same 20 examples 10 times in the prompt. Is the resulting performance closer to using 20 or 200, or in between, and for which tasks? - Supervised finetuning comparison in 4.3 is only on the machine translation task; while this is a good experiment that I was on my mind from the start, the fact that it was evaluated on only one task limits the conclusion. - Reinforced ICL description could have more details, particularly since it is highlighted as one of the main contributions. The description "we select rationales that obtain the correct final answer" makes sense as a general filtering procedure, but I'm not sure how many rationales are generated or selected per problem (though I assume only one is used for each k-shot prompt), and it's not clear what happens if no generated rationales also contain the correct answer --- are these inputs thrown away entirely, and if so does this also have a beneficial or detrimental filtering effect on the inputs used? - sec 2.3 planning logistics: I don't agree that the figure demonstrates significant increase for many-shot as stated in this section. Near-maximum performance of just under 35% success is achieved by 10 examples, with a possible uptick at 800, as described in the caption. But this uptick is about as large as the downtick at 20, and is towards the end of the plot without enough points to establish a clear trend. It's unclear to me if this is due to the larger number of examples or other effects (e.g. which examples are included, as 800 will have a higher chance of including the most useful examples, or random variation). - Related: I'm not sure what error bars indicate; other figures' captions say it is stdev, but I don't see this described in the text. - sec 3.1 Fig 5: I'm a little unclear on what K=4 corresponds to in the ICL Ground Truth case. Since the Unsupervised ICL prompt also includes 4 gt examples at the end, does the ICL Ground Truth prompt contain 4 examples in addition to the final 4 (for 8 total), or only these final 4? That is, are these 4 at the end of the prompt counted in the K or not, and is this the same between all three prompt methods? If these final 4 are included in K, then the three methods should coincide at K=4. If they are not included, then ICL Ground Truth should have 8 total gt examples (K=4 + the four at the end of the prompt) -- is this the case? I also wasn't able to find example prompts for reinforced or gt ICL for tasks corresponding to the ones exploring unsupervised prompts. - fig 6: reinforced ICL at K=250 is missing from this figure - sec 3.3 l.194 "to reduce the impact of false positives, ..." --- why is this important? does it obscure trends in the results by having too much noice from chance, for example? - sec 4.1 Fig 8: really interesting the shape of these curves as a function of K is basically what one would expect to see for a supervised model preinitialized with the original task labels, as a function of the training step number. not sure if this would be worth mentioning in the text maybe also showing a supervised training run exhibiting similar behavior for qualitative comparison? - sec 4.3 comparing to SFT: this section lacks details on how supervised fine tuning was implemented: in particular, what was the base model, which weights/adapters are trained or frozen, and how many epochs over the training data were performed? And why were these choices made in particular? There is also a description of trade-offs between training and inference costs of the two approaches, but no estimates on what the computation resources for each are. - sec 4.4 on NLL: these are some interesting behaviors, though I agree it's unclear what to make of them exactly, but I appreciate presenting the observations. As for why GT ICL may be lower NLL than Reinforced ICL: could this be that Reinforced ICL specifically rejects some high-prob model generated responses that don't end up with the right answers, raising its NLL from this filtering step? whereas in contrast, if the gt or similar same-source examples was present in the pretraining data, it would be expected to have relatively low NLL due to being drawn from that datasource? - xsum weird dates after K=50: xsum was taken from webarchive articles, basically mapping html body text to title --- I wonder if somehow the model learned the task was to recover the webarchive page title and confused with other header data including the webarchive last updated time, from the pretraining data? the 2016 timestamps are roughly around to the document snapshot times in the xsum urls. e.g. if webarchive processing included writing a header with the timestamps, then title, then rest of the page, for example. this is also compatible with discussion on finding the relevant parts of pretraining for the task, for Unsupervised ICL intro at sec 3 l.144. - xsum could also use unsupervised ICL -- what is its performance just adding articles without summaries? For that matter, machine translation could also use unsupervised ICL just with the English phrases. I'd imagine it wouldn't help at all for MT but it's not impossible, and evaluating it might still be interesting just to verify this. In general using all techniques for all tasks would help round out the study (only reinforced ICL wouldn't make sense for tasks where there is no succinct way to do correctness filtering).
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
With the possibility of including more data, it is now more likely to include noisy or incorrect data (either accidentally or adversarially). This might also be mentioned and discussed in the limitations section.
[Continued] Clarifications about Reinforced and Unsupervised ICL, and remaining questions
> **Details about Reinforced ICL: (1) how many rationales , (2) selected per problem, and (3) what happens when no generated rationales, and is it detrimental?** - (1) Number of rationales generated was based on available problems for ICL. MATH has 7.5K problems and we generated one rationale per problem, resulting in correct rationales for about 3.5K problems (but we see plateauing with about 500-shots). For tasks with a much smaller number of inputs (250 for GPQA, 150 for BBH), we generated 10 rationales per problem at a temperature of 1, to maximize having at least one correct rationale per problem. - (2) For each problem, we randomly pick one of the correct rationales for the K-shot prompts. - (3) Inputs without any correct rationales are thrown away entirely, which is a limitation of Reinforced ICL to be unable to use such inputs. Moreover, as these inputs correspond to the harder problems which the model cannot currently solve, we might be throwing away valuable information. As shown in Figure A.17, doing another iteration of Reinforced ICL using only these “harder” inputs can further improve many-shot performance. > **l.194 "to reduce impact of false positives" --- why is this important?** Typically, we can only evaluate final answer correctness and not verify the CoT rationale. As such, BBH tasks with binary choices can result in rationales that obtain the right answer “by chance” with wrong reasoning (“false positives”) – our manual inspection indicated that model-generated rationales on such tasks were of poor quality. This is an inherent limitation of methods that rely on model-generated rationales, including Reinforced Self-Training method, which inspired Reinforced ICL. **We discussed this limitation in L134-137**, and will clarify it in the revision. > **fig 6: reinforced ICL at K=250 missing** We generated rationales with correct answers for only 129 problems (this was mentioned in L180-181, will further clarify). > **Unable to find prompts for reinforced or gt ICL for unsupervised ICL tasks** Figure A.9 shows the zero-shot GT prompt for GPQA and Figure A.11 shows the 4-shot GT prompt for MATH and GSM8K. Reinforced ICL prompts contain the same problems as GT prompts but use model-generated solutions – we’ll add an example to show the solution differences. > **Sec 4.4: why GT ICL may be lower NLL than Reinforced ICL .. rejects some high-prob model generated responses .. raising NLL from filtering step?** Certainly, the filtering step in Reinforced ICL can result in solutions that the base model considers as high NLL. Another hypothesis is that model-generated solutions can look very different from human-written ones, resulting in higher NLL on GT solutions. Notably, Reinforced ICL has lower NLL than GT ICL on model-generated solutions on test problems. > **sec 3.1 Fig 5: what K=4 corresponds to in ICL GT. Are these 4 at the end of the prompt counted in the K, and is this same for all three prompt methods?** The 4-shot GT ICL prompt uses only 4 examples, which correspond to the same examples as used by the 4-shot formatting preamble of Unsupervised ICL (Figure A.11). However, Unsupervised ICL prompt also used an instruction preamble (“You will be provided Problems similar to the ones below:”), leading to differences in results from 4-shot GT ICL Prompt. Reinforced ICL uses the same problems but model-generated solutions instead of GT solutions, resulting in different performance than GT ICL, even in the 4-shot setting. > **Machine translation could use unsupervised ICL with the English phrases .. it wouldn't help at all for MT .. and evaluating it just to verify this. Also, unsupervised ICL on xsum?** As expected, Unsupervised ICL on low-resource MT doesn’t help, as shown in Figure A.16. This was mentioned on L146-147 about limitations of unsupervised ICL. We also ran Unsupervised ICL on XSum but only observed a maximum rouge-L score of about 23.95 (with 250-shot unsupervised prompt), which is slightly higher than just using the 1-shot prompt with an article and summary. Looking at the generated summaries, they were more verbose than the abstractive target summaries. This might be mitigated to some extent with a better zero-shot instruction, but would be unfair as no such instruction was used for many-shot ICL with ground-truth examples. > **sec 4.1 Fig 8: the shape of these curves as a function of K is what one would expect for a model preinitialized with the original task labels, as a function of training step** Agreed, this is a nice connection and we’d mention it in text. > **With more shots .. more likely to include noisy or incorrect data** If many-shot examples in the prompt contain biases (e.g., stereotypes, unfair representations), the model can possibly amplify these biases. Moreover, many-shot ICL can be used for overriding safety-training biases, manipulating LLMs to behave in unintended or harmful ways. We’ll include this in the discussion.
Summary
In this work, the Authors investigate the performance of large language models on in-context learning (ICL) tasks when provided with a large number - in the order of hundreds or thousands - examples (many-shot ICL regime), enabled by recent increases in context window sizes. The Authors demonstrate significant improvements across various tasks when moving from few-shots to many-shots. They also introduce two methods, called Reinforced ICL and Unsupervised ICL, to mitigate the need for human-generated examples. The paper analyzes how many-shot ICL affects model behavior, including overcoming pre-training biases and learning high-dimensional functions.
Strengths
The paper examines many-shot ICL across a wide range of tasks including translation, summarization, planning, mathematical problem-solving. This broad scope illustrates very well the benefits of many-shots. The Authors introduce novel methods (Reinforced ICL and Unsupervised ICL) to address limitations of many-shot ICL, namely, the need for large amounts of human-generated examples, and show their superiority. An in-depth analyses is conducted of how many-shot ICL affects model behavior. For example, the Authors demonstrate that many-shot ICL can overcome pre-training biases (as shown, for example, in Figure 10 where performance on flipped and abstract labels approaches that of default labels with increasing shots). The paper presents a spund evidence for the benefits of many-shot ICL. For instance, in Figure 1, it is shown a consistent performance improvement across various tasks. By comparing many-shot ICL to fine-tuning, it is shown that a comparable performance is reached in some cases, suggesting that many-shot ICL could be a viable alternative to fine-tuning.
Weaknesses
As the Authors recognize, the study is limited to a single model (Gemini 1.5 Pro). While the Authors do include some results with GPT-4-Turbo and Claude-3-Opus, a more comprehensive comparison across different models would strengthen the generalizability of the findings. While the paper provides extensive empirical results, it lacks a theoretical framework to explain why many-shot ICL works so well. A theoretical analysis could provide insights into the mechanisms behind observed improvements. Another point in which a theoretical analysis would be highly desirable regards the following case. The Authors note that performance can in some cases degrade with more examples (e.g., in the case of MATH), but don't fully explain this phenomenon. They indeed state: "Our analysis found that negative log-likelihood trends are insufficient to explain this degradation, and future work should investigate new directions to shed light on the matter and improving many-shot ICL capabilities." However, this is point is correctly raised while discussing limitations. Finally, a more explicit discussion of the potential drawbacks and risks associated with many-shot ICL would be helpful to raise awareness.
Questions
I would like to aks the Authors what they think about the possible application of many-shots ICL in alignemt problems. The ability to include much larger contexts could - if I am not mistaken - be leveraged for improving alignment to specific ethical standards or legal frameworks (this seems plausible if one takes into account your findings about overcoming pre-training biases). By providing a large number of examples that demonstrate the desired ethical reasoning or decision-making process, it might be possible to steer the model's behavior more effectively than with few-shot prompting or fine-tuning. Do you think this is a potential application of many-shots ICL or you can already see limitations? For example, it comes to me that, since sometimes the performance can degrade with too many examples, and the ordering of examples can affect results, applying many-shots ICL to ethical alignment would require understanding how to structure and present examples effectively within the context window.
Rating
9
Confidence
4
Soundness
4
Presentation
4
Contribution
4
Limitations
While the paper addresses correctly the technical limitations, the potential downsides or risks associated with many-shot ICL are not discussed. I think that a brief remark on potential misuses would be appropriate.
Summary
Owing to the significant increases in context window lengths, the paper analyzes the efficacy of the Gemini 1.5 Pro LLM in the many-shot in-context learning (ICL) setting, where hundreds to thousands of exemplars can be provided to the model at inference time. ICL has generally been restricted to the few-shot learning setting, where only a small number of demonstrations are provided to the LLM. As this expansion to the many-shot setting can pose issues relating to large scale data collection, the authors propose two simple approaches: (1) Reinforced ICL, which switches human written rationales for demonstrations with chain-of-thought model generated rationales, and (2) Unsupervised ICL, where rationales are not provided in the ICL task. The authors conduct extensive experiments across a number of problem domains, ranging from summarization, machine translation, logistics planning, question answering, algorithmic reasoning, among many others, showcasing the performance benefits obtained by using a larger number of exemplars in ICL.
Strengths
- I believe the paper is a significant contribution to the field of ICL, as it analyzes the efficacy of LLMs in the not yet studied many-shot regime. The authors conduct a number of extensive experiments ranging from a diverse set of tasks and benchmarks, showcasing the benefits of many-shot ICL. The biggest takeaway from the paper would be that many-shot ICL could be a suitable alternative to supervised fine-tuning which would tune the entire set of model weights, albeit at the cost of increased inference time (which can be reduced via KV caching). - The paper is very well-written and I appreciate the large scale of experiments conducted on Google Gemini 1.5 Pro. - The two annotation-free many-shot ICL strategies (Reinforced ICL and Unsupervised ICL) proposed are simple, but clearly demonstrate improved performance on a wide variety of tasks. - Findings relating to overcoming pre-training biases, learning higher-order functions, and many-shot ICL vs supervised fine-tuning are also important results that strengthen the contributions of this work.
Weaknesses
- As this is an empirical analysis paper which spans many different tasks and benchmarks, can the authors confirm that for all the tasks, the full test splits were used for evaluation as in line with community standards and past work? If there are any exceptions, these should be listed. For instance, measuring summarization performance on XSum/XLSum for only 150 test articles seems far less in size than the actual test set for this dataset (~11k articles). Were the articles randomly sampled? - While the ablations are carried out with respect to the number of ICL exemplars provided to the LLM, I am somewhat unsure of what role the model size plays here. As shown in the cited Wei et al paper (https://arxiv.org/pdf/2303.03846), larger LMs might do in-context learning differently and can learn input-output mappings better than smaller LMs which for example, might not be able to reject pretraining biases. Thus, it is not clear if some of the performance improvements are just a direct result of using Gemini 1.5 Pro (which is a model on the larger end of the size spectrum). Conversely would a smaller LLM with a larger context also benefit from many-shot ICL (possibly not as many exemplars as Gemini 1.5 Pro, however). Do the authors have any thoughts on this and can they draw a distinction between gains attained due to an increase in the size of the model versus the many-shot ICL setting? - Do the authors have any intuition for why at times performance reduces as more exemplars are added, somewhat akin to overfitting? I noted that the authors discussed this as an open question in the paper but I think it would benefit readers if some more insight could be provided.
Questions
Please see the weaknesses listed above.
Rating
7
Confidence
4
Soundness
3
Presentation
4
Contribution
3
Limitations
Yes, the limitations have been addressed. More details can be provided in a revision.
Summary
This work conducts a comprehensive study on in-context learning (ICL). The experiments range from few-shot to many-shot scenarios, with up to 2048 in-context examples. It was observed that as the number of examples increases, the performance generally improves, even matching the performance of fine-tuned models. One limitation of many-shot in-context learning noted by the authors is the difficulty in obtaining high-quality in-context example pairs. To address this issue, the authors proposed Reinforced and Unsupervised ICL, which achieved results comparable to those using ground truth examples. Additionally, the authors explored many-shot ICL in the context of pre-training bias (distribution shift settings) and high-dimensional numerical settings, providing explanations for the results.
Strengths
The experiments are comprehensive, covering a wide range of tasks and comparisons. The paper is well-written with a clear structure. The problem itself is interesting and has broad applications.
Weaknesses
The experiments are limited to a single model, Gemini 1.5 Pro, as mentioned by the authors. This may narrow the scope of the results, as different models have varying pre-training data and biases. The performance of LLM under ICL shows significant variation (as seen in figure 6). It would be beneficial to include more seeds or utilize better statistical metrics to represent model performance, beyond just average accuracy. Additionally, as the number of shots increases, so does the number of tokens, making the computation budget (RAM of the machine) a potential bottleneck for further exploration and application. Given that many in-context examples have varying lengths, future work could focus on how to select better in-context examples that capture the task's nature while also being concise in terms of token length.
Questions
1. It's surprising that Reinforced ICL and Unsupervised ICL outperform the ground truth. Do you have any explanations for this? 2. The performance shows significant variations when different random seeds are used to select in-context examples. Could you explain why this happens? 3. Flipped labels and abstract labels seem to achieve the worst performance when K=8 or K=16. Is this a universal result across different model sizes, as related to [1][2]? 4. I am curious about what model is used for figure 11 in sec 4.3. Did you supervise fine-tune the same model and report its results on the dataset?
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Please see weakness and questions for limitations. There is no potential negative societal impact of their work.
I would like to thank the authors for their rebuttal and additional experiments. I believe these augment the work further and should be included in the revision. Finally, based on the merits I had listed in my original review, I believe the paper's contributions still currently constitute a "technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility". I will hence keep my score.
responses
Thanks for all of the responses and updated experiments, particularly measurements on runtime latency and repeated examples, and in clearing up some of the details around prompt setups, filtering, etc. I've raised my score to 7 as my concerns around these have all mostly been addressed. The repeated examples experiment to separate length from new information is quite interesting to see --- I'm not surprised that the MT task sees little (if any) improvement just by repeating examples, though I wouldn't have expected it to top out before just 2x with 25 examples. I wonder whether most tasks behave like this or if for some there may continue to be improvement for longer, that may be a good future study.
Decision
Accept (spotlight)