Transcendence: Generative Models Can Outperform The Experts That Train Them

Summary

1. Definition of Transcendence: The paper defines transcendence as a phenomenon where a generative model, trained on data from human experts, achieves capabilities that surpass the abilities of those experts. 2. Theoretical Framework: The authors provide a theoretical analysis of the conditions under which transcendence can occur. They prove that low-temperature sampling is necessary for transcendence in their setting. 3. Chess as a Testbed: The researchers use chess as a domain to empirically test their theories, training transformer models (called "ChessFormers") on datasets of human chess games with various skill levels. 4. Main Findings: - Low-temperature sampling enables transcendence: Models trained on games from players rated up to 1000 and 1300 were able to achieve ratings above 1500 when using low-temperature sampling. - Dataset diversity is crucial: The model trained on games up to 1500 rating did not achieve transcendence, likely due to less diversity in the dataset. - Denoising effect: Low-temperature sampling helps by shifting probability mass towards better moves in key situations. 5. Theoretical Insights: The paper draws connections between their findings and existing literature on model ensembling and the "wisdom of the crowd" effect.

Strengths

The paper studies an interesting questions theoretically and shows the insights relate on a nicely accessible empirical problem namely chess. One strength of the paper are also the simple theoretical arguments presented, which seem relevant in non-trivial settings. Paper is quite well written (one could improve here quite a bit) and results nicely presented, although the paper seems quite rushed.

Weaknesses

I think the theoretical section could be improved by integrating the proofs, when given the extra page if accepted. Also I wouldn’t call the theoretical insights „theorems“ but rather propositions. I think the paper would greatly benefit from a toy model which allows to study when transcendence is possible, theoretically and practically. I think the major weakness of the paper is the missing understanding when transcendence can occur and when it can’t. Didn’t have much time to review this paper unfortunately, will study it in more detail in the next weeks and consider raising my score, after the authors feedback.

Questions

See weaknesses Typo in line 100 we make assume Typo in line 123 outputs a correct but noisy prediction(s). Reference in line 201 - reference to the current section Typo in line 237: to the right? Figure 4, improve legend and generally readability/visualizations of the Figure, E[F] two times, not clear what the difference is Typo in line 249 This is matches Typo in line 265 We, we Reference Theorem line 142

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Looks fine, would be great to discuss future research directions.

Summary

In this work the authors focus on formalizing and investigating the concept of *trascendence*, i.e. the capacity of a model to outperform the experts it was trained on. To do so, they train transformer-based models for chess limiting experts used for training to a certain maximal ranking. In this setup they postulate and prove a connection of trascendence to low temperature scaling and validate it empirically. Then they continue their analysis by showing that the gain in performance due to low temperature scaling is to be attributed to large improvements over a relatively small subset of states. Finally, a link between trascendence and dataset diversity is drawn.

Strengths

- The topic of the work is relevant, and the paper provides interesting insights with solid empirical validation. - The choice of chess as an experimental setting is very smart for this study. This setup also allows the authors to perform targeted ablations and validate specific claims. - The experiments section is clear, with convincing results, and it is easy for the reader to find and understand the evidence that backs up specific claims in the paper. - The paper is well-structured and well-written.

Weaknesses

- Chess is the only experimental setting considered. While the experiments are thorough in this setting, validating the claims in the paper on multiple and diverse experimental settings would be stronger evidence. - Theorem 2, basically easily follows from the assumption that $\hat{f}_{max}$ is better than the best expert. Since it is so central, can the authors elaborate more on the assumption that $\hat{f} _{max}$ is better than the best expert? Is it always realistic to assume? - The connection between majority voting and low temperature sampling (line 116) should be more formally proved, rather than left to intuition. One could substitute the expression for $\hat{f} =\frac{1}{k} \sum_{ i=1}^k f_i$ in $\text{softmax}(\hat{f}(\cdot|x); \tau)$ and show that for low $\tau$ (e.g. taking the $\lim _{\tau \rightarrow 0}$) it results in majority voting. In addition, I am not sure how much this connection is precise. In fact, looking at the Appendix C, doing majority voting of the single experts would not lead to the same outcome as low temperature sampling. It seems more precise to say that low temperature sampling results in a sharp (i.e. low-entropy) distribution with its peak corresponding to the action that is put more mass on by the consensus of experts. - To my understanding, in Section 3.4 the mathematical definition of the i-th expert, that performs well on the subset $\mathcal{X}_i$, should rather be $f_i(y|x) = \biggl( \frac{\delta(y \in Y^\star_x) \delta(x \in \mathcal{X}_i)} {|Y^\star_x|} + \frac{\delta(x \notin \mathcal{X}_i)} {|\mathcal{Y}|} \biggr)$. Am I missing something? - In key parts of the work there are typos/inaccuracies (see above and Questions section), which hamper the clarity of the work and confuse the reader.

Questions

- In Eq.4 there appears to be a typo, since the two terms subtracted on the right-hand-side are exactly the same. In my understanding it should be $y \sim f(.|x)$ in the second expectation. - In line 237 it seems that "left" should be replaced with "right".

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors are throughout the paper clear in the assumptions that define the scope of their claims, hence implicitly outlining some limitations of the study. Limitations are also explicitly addressed in the last section.

Summary

The paper explores the phenomenon where generative models, trained to mimic human behavior, can surpass the performance of the experts generating the training data. The study focuses on autoregressive transformer models trained on chess game transcripts, demonstrating that these models can outperform the best human players in the dataset. The key to this transcendence is identified as low-temperature sampling, which effectively denoises human errors and biases through a majority-voting mechanism. The paper provides theoretical proof and empirical evidence supporting this phenomenon and discusses the necessity of dataset diversity for achieving transcendence.

Strengths

Some Strengths: 1. Novel Concept: The paper introduces and formalizes the concept of transcendence in generative models, providing a new perspective on model performance. 2. Theoretical Foundation: The authors offer rigorous theoretical proofs that support their claims, grounding the concept of transcendence in solid mathematical principles. 3. Empirical Validation: The study includes extensive experiments with chess models, demonstrating the practical applicability of their theoretical findings. 4. Comprehensive Analysis: The paper not only shows that transcendence is possible but also delves into the mechanisms (low-temperature sampling) and conditions (dataset diversity) required to achieve it.

Weaknesses

Here are some weaknesses: 1. Limited Scope: The empirical validation is confined to chess, a well-defined and constrained domain. It remains to be seen if transcendence can be generalized to other, more complex tasks. 2. Assumptions: The theoretical framework relies on several simplifying assumptions, such as uniform sampling of experts and the nature of reward functions, which may not hold in real-world scenarios. 3. Practical Implications: While the concept of transcendence is theoretically and experimentally validated, the practical implications and real-world applications of this phenomenon are not thoroughly explored. 4. Computational Resources: The training of large transformer models, particularly with extensive datasets and multiple temperature settings, requires significant computational resources, potentially limiting reproducibility for researchers with fewer resources.

Questions

1. How do the authors plan to test the concept of transcendence in domains beyond chess, particularly in more complex and less structured tasks? 2. What are the potential impacts of relaxing some of the theoretical assumptions made in this study? For example, how would a non-uniform sampling of experts affect the results? 3. What ethical implications might arise from models that can transcend their training data, particularly in sensitive domains like healthcare or finance? 4. How can dataset diversity be ensured or measured effectively in other domains to facilitate transcendence? 5. Can the authors provide specific examples or case studies where transcendence could lead to significant real-world benefits?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

1. Domain Specificity: The study is currently limited to chess, and its applicability to other domains is speculative at this point. Further research is needed to confirm the generalizability of the findings. 2. Simplifying Assumptions: The theoretical results are based on several simplifying assumptions that may not hold in practice. Future work should aim to relax these assumptions and test the robustness of the findings. 3. Resource Intensity: The models and experiments require substantial computational resources, which may not be accessible to all researchers, potentially hindering reproducibility and further investigation. 4. Ethical and Practical Implications: The paper briefly touches on the broader impact but does not delve deeply into the ethical and practical implications of deploying transcendent models in real-world applications. 5. Future Work: While the paper lays a strong theoretical foundation, it leaves many avenues for future research, including exploring other domains, addressing ethical concerns, and developing practical applications.

Thank you!

Thank you for the additional results and analyses provided. After also studying the other reviews and responses, I will raise my score accordingly. While I think the insights aren't of very surprising nature, the paper provides clear evidence and theory for a interesting question/phenomenon.

Good!

Thank you for your response. I have gained a deeper understanding of your work through your replies and interactions with the reviewers. I will consider your comments and adjust my score accordingly.

Thanks for the rebuttal

Thank you for your answers and the additional results and insights provided. After reading also the other reviews and replies, I will raise my score accordingly.