Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning

In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. However, existing literature has highlighted the sensitivity of this capability to the selection of few-shot demonstrations. Current understandings of the underlying mechanisms by which this capability arises from regular language model pretraining objectives remain disconnected from the real-world LLMs. This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models. On this premise, we propose an algorithm to select optimal demonstrations from a set of annotated data with a small LM, and then directly generalize the selected demonstrations to larger LMs. We demonstrate significant improvement over baselines, averaged over eight GPT models on eight real-world text classification datasets. We also demonstrate the real-world usefulness of our algorithm on GSM8K, a math word problem dataset. Our empirical findings support our hypothesis that LLMs implicitly infer a latent variable containing task information.

Paper

Similar papers

Peer review

Reviewer uvg25/10 · confidence 4/52023-07-01

Summary

The paper presents a new perspective to understand the in-context learning behavior of large language models (LLMs) from the angle of latent concept learning resembling topic models. Based on the generative process defined by input data, latent concepts, and labels, the authors propose a two-stage algorithm to first learn the latent concept and then use it to select the best-performing demonstrations that boost in-context learning performance. Experiments on eight datasets show that the method is able to consistently outperform random selection and selection based on semantic similarity.

Strengths

* Originality: Although the latent concept learning perspective is largely inspired by Xie et al., the goal and method proposed in this paper for selecting effective demonstrations is still sufficiently interesting and different from Xie et al., which seem novel to me. * Quality: The paper first states its generative assumptions defined by input data, latent concepts, and labels, based on which it then derives a two-stage method to first learn the concept and then select demonstrations. The theories and methods seem solid to me. * Clarity: The paper is overall clear and nicely written. * Significance: While it's nice to see that the method is able to consistently outperform simple baselines across the tasks selected, I generally feel that the setting considered in this paper is somewhat artificial, and that the evaluation is not comprehensive enough to testify the generalization ability of the method. These concerns weaken the significance of the paper. See weaknesses below for details.

Weaknesses

* Problem setting: The paper assumes access to a (relatively) large training set from which a few demonstrations can be drawn. With this amount of training data (e.g., 100), there could be better alternative choices than in-context learning. For example, one can tune an LLM with parameter-efficient methods and easily outperform in-context learning (shown in Liu et al.) without even having to consider how to select the best demonstrations. Of course, one can argue that in-context learning is applicable to non-open-source LLMs while training-based methods are not. However, given the recent growth in the availability of open-source LLMs such as LLaMA and Falcon, I believe it would be generally better to consider parameter-efficient tuning than in-context learning if the authors assume ~100 training samples are available. In summary, I feel that the problem setting of "carefully selecting good demonstrations from a larger training set" is somewhat artificial on its own from the very beginning. * Evaluation tasks: The eight downstream tasks selected for evaluation appear to be too easy for nowadays LLMs, and it's unclear given the current evaluation how generalizable the method is to more challenging tasks like MMLU and reasoning, potentially when combined with chain-of-thought prompting. * Typo: Line 104: "semantic analysis" -> "sentiment analysis" Reference: Liu et al. "Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning." NeurIPS (2022).

Questions

* What is the role of $\boldsymbol{\epsilon}$ in the generative process (Line 91)? It seems like it is not taken into account in the current modeling process. * How does the method work for instruction-tuned models (e.g., ChatGPT, Alpaca)? * Why are the results of LLaMA that bad (Figure 3a)? LLaMA is generally recognized as a better-performing model than OPT series under similar sizes.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Please see the Weaknesses section.

Reviewer 8JqR6/10 · confidence 4/52023-07-03

Summary

The paper introduces a novel demonstration selection method aimed to enhance performance of (few-shot) in-context learning. The approach is characterized by a two-stage process that includes latent concept learning and demonstration selection, each holding a unique significance. In the latent concept learning stage, the authors present a method for acquiring a task-specific token embedding set through prompt-tuning. Subsequently, the demonstration selection stage involves the process of selecting in-context samples. This process is based on maximizing the likelihood of post-fixing the previously acquired task latent. The efficacy of the demonstration selection has been evaluated across various language models, leading to improved in-context learning performance.

Strengths

This paper is clearly written with a clear definition of the task. The proposed method is presented with great clarity and detail, which significantly aids in understanding the overall procedure. The paper presented impressive performance results. The successful transfer of demonstrations selected from a smaller model (GPT-2) to other larger models is very impressive. There are extensive ablation and additional experiments for deeper analysis of each component of the proposed method.

Weaknesses

An area of concern lies in the assumption that the task latent derived through prompt-tuning is considered the “optimal task latent” (line 146). This assumption may not hold universally, and could be re-considered as the use of manual prompts - which could offer a more intuitive understanding of the task.

Questions

- Considering that the order of samples in the demonstration does not play a critical role in the proposed method, what motivated the decision to include a "re-ordering" (selecting the permutation) step in the demonstration selection phase? - How would the performance change if a larger model, as opposed to GPT-2, was employed for demonstration selection? Would this potentially enhance the overall performance?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are stated in the appendix C.

Reviewer sfNW5/10 · confidence 3/52023-07-06

Summary

This paper describes a framework to select demonstrations for in-context learning by using bayseian formulation for the data generation process. Similar to topic models, the formulation uses a "concept" variable and words in this sequence are conditioned over this concept variable and are conditionally independent of the other tokens in the generated text. The concept variable models the prompt/task instruction and is modeled by learning concept tokens by prompt tuning a small LLM. Since the concept tokens are made part of the vocabulary, the selection of in-context demonstrations can be learnt by maximizing the probability for the concept tokens. Experiments are presented on multiple NLP tasks and they indicate that selecting in-context demonstrations using this formulation works better than using random in-context examples.

Strengths

1. Simple formulation that aids better selection of in-context demonstrations 2. Experiments have been presented on multiple NLP tasks

Weaknesses

Modeling: I'm not sure the topic-model like bag-of-words assumption is accurate in the way the modeling has been described. As Line 56 in the paper states, the generation of tokens would be independent of the previous tokens but to truly model this wouldn't you need to modify the current latent concept learning setup to be sequence-agnostic? The method works empirically so perhaps its okay but some clarification would help here. Perhaps the authors can explain further in the rebuttal phase if I misunderstood it. Experiments: The experiment baselines refer to the "Similar" baseline but any discussion or analysis of its results is completely missing from the main paper. I also checked the appendix, and the results in Table 3 (appendix) do not correspond to the results reported in the main paper in the histogram plots. Notation: The notation is a bit hard to follow and could benefit from simplification.

Questions

See weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Reviewer XxTq5/10 · confidence 3/52023-07-17

Summary

This paper tries to study in-context learning through a Bayesian lens, namely treating LLMs as implicit topic models. It proposes an algorithm to select optimal demonstrations from a set of annotated data with a small LLM and then use the selected demonstrations with larger LLMs. 12.5% improvement compared to random selection is observed by adopting the proposed algorithm.

Strengths

The paper has extensive empirical results showing the proposed algorithm can effectively find helpful demonstration examples to boost the performance of in-context learning.

Weaknesses

Although we see significant performance improvement, it's hard to conclude LLMs are topic models. It could be possible that topic (or distribution of input text) is one of the important factors but there are other attributes affecting in-context learning. In some ways, this also conflicts with some previous work, e.g. "In-context Learning and Induction Heads" points out the pattern copying behavior is the key, "Robustness of Demonstration-based Learning Under Limited Data Scenario" finds out random tokens are also helpful.

Questions

Have you tried to compared with better algorithms to find demonstrations, e.g. "Selective Annotation Makes Language Models Better Few-Shot Learners"?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

yes

Reviewer waNU5/10 · confidence 5/52023-07-26

Summary

This work aims at proposing a demonstration example selection algorithm in in-context learning, using a formulation of topic models for language models. Specifically, the proposed algorithm applies prompt tuning on some prefixing learnable tokens to obtain latent concepts, and select demo examples that maximize the probability of inferring the learned tokens. The authors test the algorithm on small LMs and find generalizability to larger LMs, on a set of classification tasks.

Strengths

The writing of the paper is overall clear. The theoretic formulation is well-structured and points out where assumptions and approximations are made. The proposed algorithm is tested on a set of tasks and models. The ablation studies are also reasonable.

Weaknesses

The motivation and takeaway of this work are rather vague. As a demonstration example selection algorithm, the setup is not realistic enough since it uses prompt tuning with labeled task data to obtain the latent concept tokens. Subsequently, the comparison with the baseline methods using uniform or similar demonstration examples is not fair, since they do not assume prompt tuning with labeled task data. Some natural and probably necessary questions here include: (1) how does the performance of the proposed method compare with the vanilla prompt tuning performance? and (2) since labeled task data are used, what is the performance of directly (and independently) select ICL examples based on their contribution to the predictions of the labeled task data? Additionally, the current analysis is mostly on the performance of the algorithm rather than the selected demonstration examples themselves. What attributes do they share in common? (apart from the qualitative clustering shown in the tSNE figure) What label distribution do they have? (do they simply act as a *calibration* to the model's output distribution?) In other words, probably as the title hints, what are good demonstrations for in-context learning?

Questions

Please see the weakness section for my main questions to the authors. Additionally, is any generation task considered in this work apart from the classification tasks?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Reviewer waNU2023-08-19

Thanks for the response

Thanks for your detailed response. Below I'll first clarify my concerns over the selection of the baselines (Re: 3, 4) and then the overall takeaway of the work (Re: 1, 5). The authors described their method clearly in Figure 1, with two main stages: (a) Use prompt tuning to obtain concept tokens. During this process, a *labeled* dataset D is used (Algorithm 1). (b) Select k demonstration data from a candidate data set D^d (Algorithm 2). The size of D^d is 100 as mentioned in Line 230. However, it is not immediately clear to me what the size of D is, though the authors mentioned it is "limited data" (the method "requires tuning a small model on limited data"; can you perhaps clarify further?). My concern is that the two baselines compared in this work, random selection and selection based on similarity, did not utilize this labeled dataset D. Therefore in my review, I suggested two more comparisons: (1) Compare with prompt tuning as it uses the same labeled dataset D. (2) Since prompt tuning does not involve D^d in an ICL setup, I mentioned "since labeled task data are used, what is the performance of directly (and independently) select ICL examples based on their contribution to the predictions of the labeled task data". To clarify this a bit further, I meant to select/prepend demonstration examples from D^d based on whether they can maximize the probability of generating labeled examples from D (i.e., using the terminology from Algorithm 1 and 2 --- P(Y | X^d, Y^d, X) ). Having these two comparisons that also utilize the labeled dataset D will help the audience understand the importance of deriving the concept tokens theta more clearly. My second concern, apart from the demonstration selection algorithm, is on the takeaway of this work. The authors clarified that the goal is "elucidating the underlying mechanisms of in-context learning". In that way, I think a deeper analysis into the selected demonstration examples for each task would be necessary, to give the audience an interpretable picture of the mechanism of ICL. I agree that the transferability of the demonstration examples from small to large models is interesting and potentially useful (Re: 2). I have changed my scores accordingly.

Authorsrebuttal2023-08-20

Thank you for carefully reading through our rebuttal and responding to us. We appreciate your raise of the score. Below is our response to your concerns: 1. **About the size of D**: The size of D range from 346 (ETHOS-SO, ETHOS-R) to 1.6k (SST2, FPB, COLA, DBpedia, EmoC, EmoS, GSM8K), which is determined by the availability of the annotated data and then caped by 1.6k. We will state this explicitly in the revision. 2. **About prompt tuning baseline**: We involved the comparison with prompt tuning on D in Figure 8 in the Appendix and also in the new experiments with GSM8K, as detailed in point 4 in our rebuttal. We will add this in the main paper in the revision. 3. **About contribution-based baseline**: We now understand the second baseline proposed by the reviewer. Thank you for the clarification. We will run the suggested baseline and either post the results here if we can get it done before the discussion period ends on August 21 or we will directly add this baseline in the revision. 4. **About in-depth analysis of the selected demonstrations**: We will involve an analysis of the selected demonstration from all datasets, both from a qualitative perspective as shown in the last paragraph of our rebuttal, and from a qualitative perspective by analyzing the text distribution and information gain of the selected demonstrations, similar to [1]. We will either post the quantitative results here if we can get it done before the discussion period ends on August 21 or we will directly add this in the revision. [1] Han, Xiaochuang, et al. Understanding In-Context Learning via Supportive Pretraining Data. ACL 2023.

Authorsrebuttal2023-08-15

Dear Reviewers, Thank you for your valuable review. We have provided responses to your questions, and are committed to address further concerns. We would like to ask for your kind participation in the discussions. Please let us know if we have addressed your concerns or if you have additional feedback or suggestions. We highly appreciate your time and efforts and are looking forward to the discussions. Best regards, Authors

Reviewer sfNW2023-08-18

Acknowledgement of Rebuttal

Thank you for your response. I re-read my review in light of your responses. I had understood what the similar baseline refers to but I had missed its inclusion in Figure 2. I'm updating my review score

Authorsrebuttal2023-08-18

Thank you for reading our rebuttal and responding. We are glad that our rebuttal helped clarify your concerns. And we appreciate your raise of the score. Have a good weekend :)!

Reviewer uvg22023-08-19

I thank the authors for their response. I'm still relatively positive about the paper and I'm keeping my original rating.

Authorsrebuttal2023-08-19

Thank you for taking the time to read our rebuttal and respond to us. We appreciate that you still feel positive about our paper. We are wondering if our rebuttal has resolved all of your concerns. We are happy to clarify more if you have any remaining concerns. We are also wondering if you would like to consider raising your score in light of the new experiments showing the real-world use case of the proposed algorithm and the clarification in the rebuttal materials.

Reviewer 8JqR2023-08-19

I appreciate your insightful response. The authors' clarification regarding the concept of "optimal latent" and the inclusion of additional experiments have certainly contributed to my understanding of the presented paper. Still, I find that the theoretical assumptions and the detailed analysis of the conducted experiments, which support the reported enhancements in ICL performance, could benefit from further elaboration. In light of this, my assessment remains aligned with the initial score assigned.

Authorsrebuttal2023-08-19

As the end of the discussion period is approaching, we just want to make sure that you have read our rebuttal. It would be great if you can clarify some of your comments that we didn't understand (i.e. *‘select ICL examples based on their contribution to the predictions of the labeled task data’*) so that we can give proper responses. We are also more than happy to clarify if there are any remaining concerns.

Authorsrebuttal2023-08-20

As the end of discussion period is approaching, we are wondering if you have read our rebuttal and if you have any remaining concerns. We are happy to clarify more before the discussion period ends.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC