Bias Amplification in Language Model Evolution: An Iterated Learning Perspective

Summary

This paper investigates iterated learning with large language models (LLMs). The authors present theoretical analyses demonstrating how bias is amplified when LLMs are chained in an iterated learning process. Moreover, they conduct experiments with LLMs that confirm the amplification of bias during iterated learning. When the weights of LLMs are allowed to be updated during this process, the authors find that manipulating the information passed through intergenerational transmission can train LLMs to exhibit a desirable bias, such as providing responses that are both helpful and concise.

Strengths

The experiments were conducted in three successive steps: in-context learning with both explicit and implicit hypotheses, followed by in-weight learning. I appreciate this structure, although the results are somewhat difficult to interpret (as detailed below). These three experiments are quite novel.

Weaknesses

The paper devotes extensive text (Sections 3 and 4) to presenting their key theoretical results: (1) iterated learning with Bayesian agents leads to a stationary distribution that effectively samples from the agent’s prior, and (2) in-context learning can be understood as an implicit Bayesian inference process. I found both results to lack novelty as they are essentially paraphrasing existing findings. For instance, Griffiths & Kalish (2007) have already provided the result (1), and both Xie et al. (2021) and Zhang et al. (2023) have presented the same result (2). Therefore, I do not believe the theoretical results should be featured in such detail given the established prior work. Due to lack of clarity, I am also uncertain about the results from Sections 5 and 6. The results from Section 7 are the clearest to me, where controlling the transmission channel to pass on more helpful and concise data allows the LLM to amplify this good bias. However, I am unsure if this is a viable long-term approach, as it still involves training a model using data generated by the same model. Numerous studies, including Shumailov et al. (2023) and Lu et al. (2020), have shown that this method can eventually lead to model collapse. The authors might not have observed model collapse because the iterated learning chain was not run for a sufficient number of iterations. Furthermore, the evidence that imposing too strong of a constraint (line 384) negatively impacts the model’s helpfulness during iterated learning seems to support this idea.

Questions

First of all, the font sizes in the tables and figures are too small to read. Second, the results in each table and figure are not self-explanatory. Terms like “Corr”, “r_20”, and “BOTH” in Table 1, as well as the numbers in Table 2, lack clear definitions and justification. What do these terms mean specifically? Are smaller values in Table 2 better? The text does not explain these terms either. Due to the extensive theoretical analysis, the key experimental results received insufficient attention in the main text. Moreover, the paper ends abruptly without sufficient discussion and lacks sections on limitations and future directions.

Rating

3

Confidence

2

Soundness

2

Presentation

1

Contribution

1

Limitations

This work contributes useful empirical knowledge to the field of LLMs. However, I would like to suggest that the authors place greater emphasis on the empirical results and ensure these results are clearly presented. A key missing piece is the important comparison between in-weight iterated learning and model collapse. It would be beneficial to identify when in-weight iterated learning is effective and when model collapse is likely to occur. References Griffiths, T. L., & Kalish, M. L. (2007). Language evolution by iterated learning with Bayesian agents. Cognitive science, 31(3), 441-480. Xie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021). An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080. Zhang, L., McCoy, R. T., Sumers, T. R., Zhu, J. Q., & Griffiths, T. L. (2023). Deep de Finetti: Recovering Topic Distributions from Large Language Models. arXiv preprint arXiv:2312.14226. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023). The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493. Lu, Y., Singhal, S., Strub, F., Courville, A., & Pietquin, O. (2020, November). Countering language drift with seeded iterated learning. In International Conference on Machine Learning (pp. 6437-6447). PMLR.

Summary

The paper discusses how the widespread adoption of Large Language Models (LLMs) and their iterative interactions can lead to an evolutionary process similar to human cultural evolution. It leverages the Bayesian Iterated Learning (IL) framework to explain how subtle biases in LLMs are amplified over iterations. The authors outline key characteristics of LLM behavior in this framework, supported by experimental verification, to predict and guide the evolution of LLMs.

Strengths

1. The application of the Bayesian-IL framework to LLM evolution is innovative and offers a new perspective on understanding and guiding LLM behavior. 2. The paper provides a solid theoretical foundation and validates its claims with comprehensive experiments across various LLMs. 3. The proposed strategies for guiding LLM evolution can be highly beneficial for designing more effective algorithms for bias mitigation and alignment.

Weaknesses

1. The framework relies on several assumptions that may not hold in all practical scenarios, potentially limiting its applicability. 2. While the framework is validated on specific LLMs, its generalizability to all LLM systems remains to be fully explored. 3. The paper primarily addresses explicit biases, and while it acknowledges the challenge of implicit biases, it does not provide concrete solutions for detecting and mitigating them.

Questions

1. Can you provide more details about the experimental setup, specifically the choice of datasets and evaluation metrics used to validate your framework?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See weaknesses above.

Summary

The main idea of this paper is similar to Xie et al., as both attempt to incorporate in-context learning of Large Language Models (LLMs) into the framework of Bayesian inference. This paper extends the concept to include more general multi-agent, multi-round self-improvement of LLM systems within the framework of Bayesian Iterated Learning. Based on the proposed Bayesian-IL framework, the authors prove LLM's bias amplification during self-improvement and propose a potential solution by introducing hypotheses. Experimental results confirm the existence of bias amplification during LLM self-improvement and demonstrate the effectiveness of the proposed solution in specific tasks.

Strengths

1. The attempt to conceptualize self-improvement of LLM agents with Bayesian Learning is relatively novel. 2. The proposed framework provides a systematic perspective for understanding bias amplification in LLM agents and potential solutions. It conceptualizes the design of agent-based LLM systems into initialization, imitation, and interaction phases, providing guidance for reducing bias amplification. 3. The experiment section presents comprehensive empirical evidence of bias amplification during LLM's self-interaction. The experiments are well-designed to cover diverse setups, task scenarios, and base LLMs. Results confirm the existence of bias amplification during LLM self-improvement and demonstrate the effectiveness of the proposed solution in specific tasks.

Weaknesses

1. The presentation of this paper could benefit from reorganization for better reading flow. Specific comments include: a. Provide a running example in the introduction, such as the ACRE experiment, to help readers better understand the concepts discussed in the theoretical proof. b. Reorganize Sections 3 and 4 to make the argument more concrete and concise. There is no need to repeat similar proofs available in previous literature (e.g., Xie et al.). Focus on the novel aspects of the proposed work, such as bias amplification. c. Add a connection paragraph before Section 5, summarizing the planned experiments and their connection with the proposed theoretical framework. Describe task environments and their nature, or refer readers to the appendix if needed. For example, the compositional language experiment should not be mentioned without context. The nature of ACRE tasks should not be discussed in the middle of a sub-experiment. d. Move Appendix A into the main text, as it contains valuable discussions highlighting the theoretical and practical value of this work. 2. Both the theory and experiments do not consider multi-agent scenarios, where embodied LLM agents might be competing/collaborating, heterogeneous/homogeneous, or have partial/global observation 3. In the proof of Proposition 1, the authors claim that $h \in H_{eff}$ can be guaranteed by adding constraints during imitation and interaction phases. This process might not be applicable to all Bayesian agents during iterated learning in general. For example, if agents are unaware of task completions, it is non-trivial to "rule out" unsuitable hypotheses at the end of each iteration. Consider multiple embodied LLM agents interacting with a task environment with sparse rewards, where they might not receive feedback at each timestamp. 4. In the proof of Proposition 2, the Markov property of hidden hypotheses and the LLM sampling process used in the second line lacks proof. Can we always find a hidden hypothesis $h$ such that $P(d|d_{t-1}, h) = P(d|h)$, meaning LLM's sampling only depends on this variable instead of the prompt input? 5. There is a gap between the theoretical framework and empirical evidence. As claimed in Appendix A, most of the assumptions used in the theoretical proof are violated in the experiments. How do experimental results support the proposed Bayesian-IL framework? The claim in line 314 that the proposed framework can predict LLM's behavior is less accurate. The framework actually predicts the phenomenon of amplifying bias during LLM's self-improvement, rather than individual LLM agent's behavior (e.g., when they will make mistakes given certain inputs). The proposed framework is one explanation of bias amplification among many others. 6. The solutions authors propose to reduce bias amplification all depend on domain knowledge about the task. What if the intrinsic biases of LLMs are unknown and the criteria for task completion are difficult to measure?

Questions

Questions are asked in the previous section.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Limitations are discussed in the appendix.

Summary

The paper explores the evolution of Large Language Models (LLMs) through the lens of Iterated Learning (IL), drawing parallels with human cultural evolution. The authors propose a Bayesian framework to analyze how biases in LLMs are amplified over generations of learning. They introduce the concept of Iterated Learning in the context of Bayesian agents, demonstrating how subtle biases can be magnified as knowledge is transmitted across generations. The authors argue that understanding the evolutionary process of LLMs can help in designing more effective algorithms for alignment, bias mitigation, or amplification.

Strengths

- The paper has a good theoretical basis. - The paper's focus on bias amplification in LLMs is timely and important, given the increasing concern about fairness and bias in LLMs. - It is interesting from the perspective of Bayesian framework and iterative learning.

Weaknesses

- Do you think that the Bayesian framework and iterative learning are effective and robust across different large language model agents? Could these methods lose their effectiveness with changes in model architecture or parameter scale?

Questions

- In lines 51-52 of the introduction, you mentioned, "We believe that our analysis can enhance our understanding of LLMs, and aid in designing more effective algorithms for alignment, bias mitigation or amplification, and similar tasks." Could you provide more concrete examples to demonstrate how this analysis plays a role in downstream tasks and applications to strengthen this claim?

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

NA

Official Comment by Reviewer Zowd

Thank you for your response, which addresses most of my concerns. I would keep my rating.