General Response to All Reviewers
To all reviewers,
We thank all the reviewers for taking the time to review our paper and their valuable comments. We are grateful that reviewers appreciate the effort of our work in analyzing the In-Context Learning of LLMs from a new perspective with comments such as Reviewer d42T "novel framework", Reviewer iqcg "meaningful explanations, comprehensive and convincing", Reviewer Ydk8 "systematic study, interesting results”.
We notice that a common concern raised by reviewers is the cause and inner-working responsible for the token reinforcement.
We have delved into the reasons underlying token reinforcement and conducted three important experiments to enhance our paper.
**As for the reasons, we believe that token reinforcement could be a result of the model's effort to maximize the likelihood of the training data.** Token reinforcement is hidden in the nature of pre-training corpus. In light of this, we conduct experiments to validate our hypothesis.
Given a commonly used large pre-training corpus (i.e., wikipedia-english(https://huggingface.co/datasets/olm/olm-wikipedia-20221220), 5.7B tokens), we compute the following statistical probabilities:
Exp_w P(w occurs | w is never observed in the context)
Exp_w P(w occurs | w has been observed before in the context for 1 time)
Exp_w P(w occurs | w has been observed before in the context for 2 times)
…
where w is any word, and context is the autoregressive context window of pretraining data, usually 1024 tokens.
Conceptually, these probabilities reveal whether a particular word $w$ is likely to recur if there have already been $n$ instances of $w$ observed in the context, resonating with our analysis of token reinforcement.
The statistical probabilities are shown below (n=[0,9]):
| Num Occurrence | Probabilities | Value |
| --- | --- | --- |
| 0 | █░░░░░░░░░░░░░░░░░░░ | 1.32% |
| 1 | ██████░░░░░░░░░░░░░░░ | 19.53% |
| 2 | █████████░░░░░░░░░░░░ | 35.63% |
| 3 | █████████████░░░░░░░░ | 47.95% |
| 4 | ███████████████░░░░░░ | 53.26% |
| 5 | █████████████████░░░░ | 57.73% |
| 6 | ██████████████████░░░ | 59.15% |
| 7 | ███████████████████░░ | 60.48% |
| 8 | ████████████████████░ | 60.78% |
| 9 | ████████████████████▒ | 60.79% |
The probabilities of a specific word occurring with an increasing number of instances perfectly align with the token reinforcement on LLMs.
The LLM manages to learn this inherent feature from the training data and generalize it to longer phrases and distant connections. This also elucidates the scaling with reinforcement, where larger models more effectively capture this feature from the training corpus.
To further understand the existence and inner workings of token reinforcement, we provide more experiments from two perspectives. The first one shows from learning process that **leveraging repeating features are effective in the maximizing likelihood**. We pretrain three conditional LMs from scratch and compare the converged loss of three strategies: (1) full context (2) random mask (3) mask repetitive features. Our results show that (3) mask repetitive features achieves the worst converged loss, which validates our hypothesis.
The second experiment analyzes the trained LLMs and shows that the **attention mechanism of attending preceding/adjacent token is responsible for the inner-workings of token reinforcement**. After masking out the adjacent token, the strength of reinforcement is reduced by 10 times.
For more details and experimental settings, we refer all reviewers to **Appendix C**. We highlight the revised parts of the paper with blue.
**Finally, our main contributions are:**
1. We propose a novel perspective to understand icl with repetitive text generations, where the key effect causing repetitions appears to be an important feature in ICL.
2. We empirically establish the existence of token reinforcement across various LLMs, alongside with the inevitable reason behind learning process and inner workings responsible for it.
3. We show that token reinforcement plays crucial role in ICL for both beneficial and detrimental effects. It constrains output space and desired patterns, but it is also responsible for spurious connections and possible failure of ICL. Our findings provide a novel research basis for analyze the mechanism of in-context learning.
*Based on our results, instead of the traditional understanding of input-label mappings, we view ICL as a combination of token reinforcements, even though some of them only appear once and do not get reinforced much.*
Due to the time limit in the discussion period, we are committed to incorporating these new results about the reasons and inner workings of token reinforcement into the main content, as well as the highlighting our contributions and implications in the next version.
We hope we have addressed all your concerns and would greatly appreciate it if you could re-consider your score about our paper based on our responses.