Neural scaling laws are observed in a range of domains, to date with no universal understanding of why they occur. Recent theories suggest that loss power laws arise from Zipf's law, a power law observed in domains like natural language. One theory suggests that language scaling laws emerge when Zipf-distributed task quanta are learned in descending order of frequency. In this paper we examine power-law scaling in AlphaZero, a reinforcement learning algorithm, using a model of language-model scaling. We find that game states in training and inference data scale with Zipf's law, which is known to arise from the tree structure of the environment, and examine the correlation between scaling-law and Zipf's-law exponents. In agreement with the quanta scaling model, we find that agents optimize state loss in descending order of frequency, even though this order scales inversely with modelling complexity. We also find that inverse scaling, the failure of models to improve with size, is correlated with unusual Zipf curves where end-game states are among the most frequent states. We show evidence that larger models shift their focus to these less-important states, sacrificing their understanding of important early-game states.
Paper
Similar papers
Peer review
Summary
The main goal of this paper is to demonstrate the connection between performance (ELO) scaling and Zipf's law (power-law distribution of states) in reinforcement learning (RL) agents for board games. The study examines how temperature in policy sampling affects scaling coefficients and also reports the inverse scaling phenomenon, where performance declines above a certain model size.
Strengths
1. The paper clearly describes the procedures and results of the experiments. 2. The paper presents a clear demonstration of Zipf’s law in game states and examines how this is influenced by game rules and policy sampling temperature. 3. The paper establishes a link between Zipf's law in game states and performance (ELO) scaling. 4. The paper reports the interesting observation of inverse scaling in RL agents for board games and attempts to provide an explanation for this phenomenon.
Weaknesses
1. The paper relies on a vague assumption that treats individual game states as "quanta" in a quantization model, but lacks verification or discussion of this assumption. 2. Related to point 1, the paper does not demonstrate that the model learns different game states orthogonally, raising questions about the validity of this assumption. 3. Also related to point 1, the paper does not show that the frequency of each game state is linked to performance gains from learning that state, further questioning the validity of the assumption.
Questions
Could you address the weaknesses outlined above? For example, two types of analysis could provide a more rigorous justification for treating game states as quanta, which would strengthen the soundness of the paper. (1) Can you provide an analysis correlating state frequency with performance improvements? For instance, you could analyze how changes in value prediction accuracy or policy quality for specific states relate to their frequency in the training data. (2) Can you show how the loss of specific states decreases over training time and whether the learning of each state is independent of others? These would help to validate the concept of states as quanta.
Rating
8
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Rebuttal response
Thank you for your review. We are glad you found our paper clear and interesting. We collected here the topics brought up in your review, and provide an answer to each one: - **Difference between states and quanta:** You are indeed correct in pointing out that the board states we track are not independent task quanta, in the sense that they are not orthogonal to each other. We state this clearly throughout the paper. We also dedicate appendix F to address this issue, where we discuss the difference between states and pure quanta and show the problem of using board states as proxies for quanta through an example (figure 15). In the Discussion section, we address the difficulty of finding true independent quanta, which is why we use board positions as proxies. To improve clarity, we add another mention of the discrepancy between states and pure quanta in sections 6. - **Connection between frequency and performance gain:** You are correct in pointing out that we do not present a full mechanism connecting data frequency to Elo score, as we state clearly in the Discussion section. We would argue though that we do show a substantial (even if incomplete) connection between frequency and performance. Section 6 shows that value loss is strongly correlated with state frequency, and value accuracy directly determines performance by guiding the MCTS step. The missing step is quantifying the exact amount by which performance improves, measured in Elo points. - **Experiment suggestions:** We thank you for the suggestions, these experiments can indeed strengthen the paper. Regarding your second suggestion, we added a new experiment (figure 8) where we track how the loss of individual states changes during training. Similar to the results of section 6, state loss improves gradually, unlike how pure task quanta should evolve, but they do improve in order of frequency like pure quanta. Regarding your first suggestion, figure 4 shows how frequency relates to prediction accuracy, since we measure the MSE loss $L_{\rm value} = \left(z - v\right)^2$ which directly defines accuracy. Measuring policy quality would be challenging, since the policy quality on each state depends (through MCTS) on the accuracy of all future states stemming from it. It is not clear how one can decouple the effect of accuracy on future states from the accuracy of the current state.
I thank the authors for improving the clarity of work. Thanks for answering the question related to difference between states and quanta, and performing the additional analysis on frequency and performance gain and loss evolution of each states. The authors' response clarifies all my previous questions. My further question (and suggestion) is whether there might be other ways, beyond individual states, to define quanta in this scenario—such as chunks of states that evolve together or specific behaviors in the game. This could be an interesting direction for future work.
Thank you for your response, we are glad we could answer all your questions. Regarding other ways to define quanta, we agree that grouping states together in a way that creates more orthogonal task quanta is a good direction for future work. Other than using game-specific domain knowledge, which will not transfer between environments, one possible approach is the clustering method used by Michaud et al. They computed the angular similarity between the gradients of LLM inputs, using the smallest model of the Pythia suite for gradient calculation. It is possible to do something similar with AlphaZero, calculating only the gradient from the value head and computing loss using ground-truth labels. It might be though that different AlphaZero models will produce different angles between gradients.
Summary
This paper studies scaling laws for for AlphaZero, and RL algorithm, and aims to connect it to insights from a previous work, Michaud et al. '23, that proposes that the capabilities of a neural network are composed of discrete “quanta” that each help reduce loss on a subset of training examples, and that are learned in approximately decreasing order of frequency. If the quanta are present in a Zipfian (power-law) frequency distribution in the training data, and each quantum is approximately equally valuable in the examples where it’s present, such a theory would provide a mechanistic explanation for power law scaling of loss with respect to model and dataset size. This paper is empirical, and exhibits the Zipf-law (or approximations/variations of it) of game states for 4 games: Connect4, Pentago, Oware and Checkers. It shows that Zipf's law arises in self-play in AlphaZero style algorithms (which was already known in other settings/games). The paper then studies (empirically) the scaling laws that arise, the loss distribution over game states as a function of their rank/frequency. It also shows some inverse scaling for two of the games and relates it to anomalies in the Zipf distribution.
Strengths
A novel contribution is the empirical observation that self-played games also lead to Zipf law distributed data, not just human games. Another nice insight is a loss-analysis as a function of state rank showing that loss increases for less frequent state. While this is neither surprising nor counterintuitive it is nice to see this clear picture. The paper is very well written and contains a good number of experiments and empirical analyses. I like the deep dive into the reasons for inverse scaling, even though they seem to stem from idiosyncracies of certain games (and not all games). It has a genuine and immediate acknowledgement of limitations after each section. A good, critical and honest review of related work is given.
Weaknesses
A crucial weakness of this paper is the lack of convincing connection to the quanta theory or even the necessaity to make this connection, not to mention the even more far fetched claimed connection to LLMs. The paper establishes that game states obey a Zipf law, and then invokes the quanta theory to say that this implies scaling laws and emergence. This seems to ignore the rather vast literature of other theory papers that derives scaling laws by starting from a power law distribution on the data. The simplest such example is the infinite-memory model of Hutter [Hutter'21] which starts with a power law on the data distribution as an assumption to derive scaling laws. But there are many other such papers ([Cui et al][Cabannes et al'23][Maloney et al'22], the work of Pehlevan's group etc etc. Another poignant exposition on the role of the tails of the power-law distributed data can be found in [Dohmatob et al ICML'24]. All of these paper should be cited - and discussed). I thus view this paper mostly as a curious empirical paper, which in itself is not a weakness. But then several insights strongly depend on the idiosyncracies of each game (in particular the late turn difference between Oware/Checkers versus Connect4/Pentago). In my view, a theoretical analysis is lacking (which is fine if it wasn't for the prominent overpromising of explainability from the quantization method. Again: I do not see how this method is different from other methods that assume a Zipf law on the data and then prove a scaling law.) To relate to the theory of quanta (if you must), another weakness is the identification of game states with quanta. Surely, there must be equivalence classes of games that could more closely match this notion. I also do not see how this paper "relates RL to language". That Zipf laws exists in states of game play was known (as the authors also point out). Minor: 1) Line 99: what is the definition of \chi? 2) Figure 3B is not referenced anywhere in the text, it seems. To which paragraph does it belong? Cabannes, V., Dohmatob, E., and Bietti, A. Scaling laws for associative memories, 2023. Cui, H., Loureiro, B., Krzakala, F., and Zdeborova, L. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 2021. Cui, H., Loureiro, B., Krzakala, F., and Zdeborova, L. Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime. Journal of Statistical Mechanics: Theory and Experiment, 2022(11):114004, nov 2022. Cui, H., Loureiro, B., Krzakala, F., and Zdeborova, L. Error scaling laws for kernel classification under source and capacity conditions. Machine Learning: Science and Technology, 4(3):035033, August 2023. ISSN 2632-2153. doi: 10.1088/2632-2153/acf041. Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, Julia Kempe: A Tale of Tails: Model Collapse as a Change of Scaling Laws, ICML 2024 Maloney, A., Roberts, D. A., and Sully, J. A solvable model of neural scaling laws, 2022.
Questions
0) I am unclear on what exactly you call a game state when you establish the power law histograms. Is it a configuration of the game on the board, or would you call two configurations that look exactly the same *different*, if they have been reached through two distinct series of moves? I infer from the context that you probably refer to the latter (as is also done in the physics paper on chess), but please clarify directly when you define them. 1) I find the connection to LLMs (line 292 and following) way too speculative and unsubstantiated. Can you elaborate? 2) LIne 313: "The fact that the quantization model predicts this unintuitive behavior supports the claim that it could explain AlphaZero scaling." - How so? 3) Why do large models overfit, lines 457 and following: Here is where you could have explained more, expecially if the quanta hypothesis would hold. The connection to distribution shift is not clear to me - do you have any evidence there?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Rebuttal response
Thank you for the detailed review, it is clear that you spent time and effort reading our paper. We also thank you for the constructive feedback, which helped us significantly regarding comparison to other works. We will address the points you raised in the order they were presented: - **Whether the loss scaling is intuitive or not:** we want to clarify our statement that section 6's loss scaling is surprising and unintuitive. We specifically refer to figure 4B when we make that statement, where we show that ground-truth loss increases with state rank. Note that this is not the label-fitting loss, but the loss against the real state value, which should be exponentially harder to fit the higher the frequency of a state is (see figure 4C). While we agree that normal loss should decrease with frequency, we expected ground-truth loss to increase with frequency due to the correlation with combinatorial complexity. Even if an agent perfectly fits the label of a high-frequency state, the probability that this label is close to the actual true label decreases rapidly with state frequency. Consider the following example that illustrates our claim: the opening state of Connect Four has a ground-truth value of 1, since it is a first-player-wins game. The only way a model can learn this is by improving its knowledge on many states occurring later in the game, slowly approaching perfect play, and then generating enough training samples to unlearn the value it learned in early training (0.5, the value for random/low skill play). It is clear that improving performance on late-game states is a prerequisite to improving on early-game states. We have changed the text to reflect that it is figure 4B that we find surprising, and indeed not 4A. - **Connection to LLMs and quanta theory:** Thank you for the list of citations. We chose to compare our results to Michaud et al. since their model recovers data, parameters and compute scaling, assumes a Zipf distribution and contains real examples of how task loss behaves in the language domain. We definitely agree that this paper should mention other similar literature, since other models also contain mixtures of the elements listed above. The comparisons we make to quanta theory also hold for other models, and we now state this in a new subsection in section 2, where we discuss other models of scaling that use Zipf's law. Regarding the claims on connections to LLMs, we have significantly toned them down in the revised version to match the contribution of this paper (especially in the abstract and introduction). Indeed, as you claim, this is an empirical paper, where we present and discuss AlphaZero scaling phenomena to help understand why these RL scaling laws exist. - **Identification of states with quanta:** As you point out, plain board states are not independent task quanta, and we point this out several times in the paper. Your suggestion of searching for equivalence classes of games where states are more separated from each other is intriguing, and indeed we discussed it ourselves while writing this paper. We would argue though that it is infeasible to ask for such experiments, not only because it requires a large survey of games (and analyzing their scaling), but because such games probably don't exist. A board game where states are independent, meaning the player cannot transfer skills learned from one state to another, will not be popular among humans. Human players like games where correlations can be drawn between states, and learning one board position pays off in other positions. The only games we can think of that would have a clear task separation are artificial variants on problems like Michaud's multitask sparse parity problem. Such games don't exist "in the wild" simply because they are (by definition) not fun to play. Minor comment responses: 1) $\zeta()$ is the Riemann zeta function, we now state this clearly in the text. 2) We added the missing reference to figure 3B in the caption and main text. Question responses: 0) The definition we used for a game state does not include the history of moves, unlike the chess analysis of Blasius & Tönjes. Thank you for pointing out this ambiguity, we have now clarified this in the last paragraph of section 4. We use the observation (the input tensor) as the definition of a game state, since any information excluded from it is not modeled by the neural net. The OpenSpiel implementation does not include a full move history in the observation, even for games like chess where history is significant due to repetition rules (adding history would make the observation infeasibly large). We continue in the next comment...
1) Could you elaborate on what you found speculative in the *'Comparison to LLMs'* paragraph? This paragraph mostly contrasts our results with quanta theory, so we assume you meant the explanation on why the loss curves are smooth. This explanation directly relies on results from Michaud et al., specifically the distinction between Monogenic/Polygenic samples, illustrated in their figure 4. If board states were indeed pure task 'quanta', our figure 4A should have looked like a collection of step functions (according to Michaud et al.). The smooth, gradual decrease of loss as model size goes down could be explained if states were polygenic samples. However, since the purpose of this paragraph is to show differences to quanta theory rather than guess their origin, we removed this speculation from the text. 2) We hope we answered this question in our above reply on whether our results are surprising or not. Even so, we removed this sentence since we do not think it is a crucial part of the text. 3) Connection to distribution shift: We added a section (section 7.3) elaborating on the non-stationarity argument (distribution shift was a misnomer, see comment below). We present new evidence supporting this hypothesis, showing that large models continuously improve performance on end-game states during training, while performance on early-game states first improves, then degrades. In short, we argue that the mechanism causing inverse scaling is likely a combination of dormant neurons (see Sokar et al.) caused by non-stationary targets (=state values), coupled with the fact that late-game states have more stable targets than early-game states. * A better term than distribution shift is non-stationarity, since we talk about labels shifting during training, while distribution shift formally refers to the difference between train and test distributions only. Citations: Michaud et al., The quantization model of neural scaling. NeurIPS, 2023. Blasius and Tönjes, Zipf’s law in the popularity distribution of chess openings. Physical Review Letters, 2009. Sokar et al., The dormant neuron phenomenon in deep reinforcement learning. ICML, 2023.
Thank you for your response and effort
I thank the authors for their effort to improve the paper and their thorough responses. (On a side note, it would have been nice if changes in the manuscript had been marked in some color, as it is difficult to remember how exactly the paper looked before, not to mention having to skim through everything again.) I appreciate the added references and better positioning and clarification of scope. I accordingly raise my score to 6, though I need to point out that it's a 6 rather pointing towards a 5 than to a 7.
Thank you for your feedback and score change. We apologize for not using color marking for the revision changes, we misunderstood OpenReview's revision comparison tool and assumed it would allow comparison with the original manuscript.
Summary
The paper explores the relationship between neural scaling laws for the AlphaZero-type RL algorithms and Zipf's law, and the authors observe that the frequency of game states encountered by AlphaZero during training and testing adheres to Zipf's law. This pattern arises from the tree-like structure of board games, where certain states are more frequently visited than others.
Strengths
The application of Zipf’s Law in RL contexts is innovative, and the paper has conducted substantial empirical analyses to support its core hypothesis: AlphaZero’s experience with game states follows Zipf’s distribution and correlates with neural scaling behavior. The paper also finds that inverse scaling, the failure of models to improve with size, is correlated with unusual Zipf curves where end-game states are among the most frequent states.
Weaknesses
While the paper explores an intriguing connection between scaling laws and Zipf's law within the context of AlphaZero, a key concern is the practical utility of the discovered scaling law for advancing RL algorithms in meaningful ways --- the paper does not clarify how the observed relationship between state rank and frequency could directly inform the development of improved RL algorithms or strategies for board games or other RL tasks. For instance, if this relationship could guide the allocation of computational resources or suggest new exploration strategies, it would strengthen the practical relevance of the findings. Additionally, the work lacks sufficient intuitive explanations for the scaling law it reports. The authors could, for example, discuss why certain states might consistently follow Zipf’s law in terms of frequency and how this frequency pattern impacts AlphaZero’s learning efficiency. Some illustrative figures/plots would be helpful. Without these insights, it is challenging to interpret the implications of the scaling law or understand how it might generalize to other RL environments. Providing concrete examples or scenarios where this scaling law could impact AlphaZero's decision-making process or resource allocation would make the findings more actionable and relevant for the RL community.
Questions
1. What's the practical usage of the discovered scaling law? Can the observed relationship between state rank and frequency improve the design of RL algorithms? 2. Can the discovered scaling law generalize to RL environments beyond board games, particularly those with continuous state and action spaces? If so, what would be the theoretical basis for this? 3. The inverse scaling phenomenon is an intriguing observation, but its causes are not fully explained. Could the authors elaborate on potential factors contributing to this behavior? 4. What's the connection between the discovered RL scaling law for AlphaZero and LLMs?
Rating
5
Confidence
2
Soundness
2
Presentation
2
Contribution
2
Rebuttal response
Thank you for your review. We are glad you found the paper innovative, with substantial evidence supporting its claims. We list the points raised in your review and address each of them: - **Explaining AlphaZero scaling laws and their practical significance:** We would like to clarify that AlphaZero power laws were not discovered by us, but rather by Jones (2021) and Neumann & Gros (2022). These works extensively discuss the importance of scaling laws for RL and how to utilize them. As you suggest, they explain how an optimal allocation of compute (both in training and inference) can be found using the power law exponents. They also discuss the generalization of these laws to other RL settings, as do other works on RL scaling laws like Hilton et al. (2023). - **Can the results of this paper improve RL algorithms?** That is a good question. One practical use that comes to mind is directly manipulating the state distribution, to avoid the inverse scaling we show in the paper. Ruoss et al. do something similar when they use supervised learning to fit a chess dataset, as we discuss in the Related work section. We believe this is a good research direction, but out of scope for this paper, which is an empirical analysis paper and does not have enough space for a complete algorithm. However, we do like the idea of using knowledge on state frequency for an informed selection of training states (similar to what was done by Ruoss et al.) and added a mention of this future research direction in the Discussion. - **Discussing why states follow Zipf's law:** The paper contains a section dedicated to answering this question (section 5.1), where we expand on the discussion provided by Blasius & Tönjes and add intuitive explanations and illustrations. Questions section: 1) Answered above. 2) As we mention in the introduction section, scaling laws were discovered in other RL environments (though not many). Some prominent examples are Hilton et al. (Atari-like), Tuyls et al. (NetHack imitation) and Springenberg et al. (offline learning, continuous control tasks). If you meant to ask whether our state-frequency analysis could apply to other environments, that is a good question. Board games are an easy test bed because board positions are easy to group together, but this is not true for most RL problems. It could be possible to perform a similar analysis if one uses a smart way to group similar states together, but this would depend on the individual environment. 3) We are glad you found the inverse scaling section interesting. Seeing your and other reviewers' interest in the cause for inverse scaling pushed us to add a new subsection (7.3). In this subsection we present new results supporting our hypothesis for the mechanism behind inverse scaling, which we now lay out in detail. We invite you to take a look at the revised version. 4) Could you specify what aspects of this connection you are asking for? Since the main subject of this paper is to discuss a connection between AlphaZero scaling and a scaling theory for LLMS, we are not sure what the scope of this question is. Citations: Jones, Scaling scaling laws with board games. arXiv preprint, 2021. Neumann and Gros, Scaling laws for a multi-agent reinforcement learning model. ICLR, 2022. Hilton et al., Scaling laws for single-agent reinforcement learning. arXiv preprint, 2023. Ruoss et al., Grandmaster-level chess without search. arXiv preprint, 2024. Blasius and Tönjes, Zipf’s law in the popularity distribution of chess openings. Physical Review Letters, 2009. Tuyls et al., Scaling laws for imitation learning in nethack. arXiv preprint, 2023. Springenberg et al., Offline actor-critic reinforcement learning scales to large models. ICML, 2024.
Dear reviewer QW8y, thank you again for your review. As the discussion phase is coming to a close, we would greatly appreciate your feedback on our responses and the revision we made following your suggestions. We would be glad to answer any questions you may have.
Thank you for the explanations! It helps a lot. I do still have a few comments if you could kindly address them: 1. Can the authors provide some intuitions behind Zipf's law? Specifically, intuitions on how these intuitions can help with choosing an optimal allocation of compute in training, inference, and RL would be super helpful. 2. Thank you for adding section 7.3! I am still a little bit confused by Figures 7 and 8, I wonder what the implications for higher state rank are. For example, do they mean more difficult game states? How to measure it systematically? 3. Clarification on connection: I wonder how Zipf's law in RL differs from the scaling laws of LLMs from a high-level perspective. Many thanks!
Thank you for your questions. We are glad to hear you found our response helpful. 1) The paper's focus is on analyzing RL scaling rather than building a method to improve RL training, but as we mentioned in our response (_'Can the results of this paper improve RL algorithms?'_) and in the revised version, our results imply that model performance could be improved through selective training. Our results suggest that inverse scaling occurs in our experiments due to overfitting on a subset of states, which includes board states close to the last move of the game and that have relatively fixed values regardless of player skill. As we explain in section 7.3 and appendix F, we can only identify part of this subset, but not all of it. A promising research direction is to overcome inverse scaling by identifying this subset and under-sampling it during optimization (or even discarding it). As we mention in the paper, our attempts to identify the correct subset of states to discard from training did not succeed in eliminating inverse scaling, likely because the overfitting-subset is not trivial to define. 2) Thank you for the positive feedback! In our case, higher-rank states are states that have low frequency, and are less likely to occur in a game. This definition would hold for other RL problems, given any definition of a state. In the case of board games, higher rank also correlates with easier states, because the number of potential future moves to explore is smaller. We illustrate this in figure 4C. This correlation might break in other RL environments, but our definition of state rank is easy to generalize. Since you mention figures 7 and 8, we assume you might mean the set of late-game states when you refer to high-rank states. We define them as states occurring $>N$ time steps into the game ($N=40$ to be specific, see figure 6). This definition can also be used for other RL environments. 3) We are still not sure what you mean, but if your question is how RL scaling laws differ from LLM scaling laws, we are happy to answer that. LLM scaling laws scale test loss as a power of compute/parameters/data, resulting in a power law of the optimal model size with compute. In AlphaZero, Bradley-Terry playing strength $\gamma$ scales as a power of compute/parameters, but although a different performance metric is scaled, this leads to an optimal-size power law just like LLMs. This power law is used for choosing mode size in expensive, large-compute training runs.
Summary
The current paper explores scaling laws in an RL setting---specifically, in board games. The main goal (at least as I understand) is to borrow prior theory on explaining LLM scaling laws to investigate scaling laws in an RL setting. To this end, the authors first show that the assumptions of prior theory (zipfian distribution of tasks) is met in this setting (since board games have a tree-like data-generating process). The authors' investigations also leads to the demonstration of an inverse scaling effect, which they argue is explained by larger models memorizing relatively endgame states, which are relatively less important, at the cost of reducing loss on earlier game states, which are important for developing solid opening strategies. Overall, this is a good model system work, wherein a synthetic abstraction (a model system) is used to study some phenomenon of interest (scaling laws), but my apprehension lies in lack of a coherent question-driven narrative.
Strengths
Though scaling laws have been explored in RL before (see Jones 2021), the current paper is broader in its scope. It often centers itself in a prior theoretical framework by Michaud et al. (2023), with several experiments primarily verifying the validity of that framework. I'm a big fan of such synthetic studies, and hence really like the experiments in the paper. For example, these experiments allow authors to develop a concrete demonstration of inverse scaling. The paper is also well written (though there are certain parts that can be made more precise, e.g., simply specifying what a "state" means).
Weaknesses
- **Unsure what the point of the paper is.** My biggest apprehension with the paper is that I am unsure what the guiding question is. I currently see the paper as replication of scaling laws in RL / board games (which we know is possible because of Jones 2021) and a validation of Michaud et al. (2023)'s theory of how scaling laws arise. Given the latter paper never makes an assumption about modality, but merely relies on the existence of a multi-task setting with a Zipfian prior, any setting that satisfies these conditions should yield a power law scaling with increase in model / data size. I note that it's certainly good to confirm theories in different settings, but I'm apprehensive of accepting a paper for merely that reason. Overall, then, I argue results in the paper are either known or relatively expected corollaries of prior results. If I'm missing something, please let me know what questions the paper's contributions help address, and I'd be happy to reevaluate my opinions accordingly. - **How does the proposed inverse scaling hypothesis connect with prior explanations of the concept?** A possibly novel result may be the exploration of inverse scaling in this work, but I first need to clarify how the proposed hypothesis is different from prior work on the topic. For example, in the original inverse scaling paper by McKenzie et al., the main connecting thread between several tasks is that there are multiple viable ways of responding to a query, but the model starts to prefer using an undesired way (e.g., by relying on some distractor features) as it is scaled. This leads to a reduction in performance as size is scaled. Is the proposed hypothesis by authors, i.e., inverse scaling is driven by memorization of a set of endgame states (which will lead to faster loss reduction), inline with prior distractor features hypothesis? If not, how is the proposed hypothesis different? - **Related work.** Related to both points above, I think authors did not properly situate their work with respect to existing work: are you trying to develop a new hypothesis for a given phenomenon, are you trying to identify a new phenomenon, or are you trying to simply verify existing theories? Generally, the relevant literature on all these questions should be cited and discussed, but especially if a new hypothesis is being proposed, at least a contrast should be made between the new and previous hypotheses to help clarify how the current one is different (e.g., prior work on inverse scaling should be cited https://proceedings.mlr.press/v238/ildiz24a/ildiz24a.pdf.
Questions
See weaknesses.
Rating
6
Confidence
4
Soundness
4
Presentation
3
Contribution
2
Rebuttal response
Thank you for your review. We are glad you liked the experiments in the paper and that you find it well written. We also appreciate your advice regarding distractor tasks - we found it very useful when writing the revised version. We address the points you raised in the order they were presented. - **Specifying what 'state' means:** thank you for bringing this to our attention, we changed the text to make it clear what we mean by 'state' (i.e. the observation associated with an MDP state). A full definition is now included in section 4. - **Regarding the goal of the paper and relation to Michaud et al.:** The paper's goal is to reveal and interpret parts of the mechanism behind RL scaling laws, specifically AlphaZero. We respectfully disagree that our results are expected corollaries of Michaud et al., or that the reasons for scaling laws are well understood in our setting. Indeed, Michaud et al. don't restrict themselves to any modality, but they work under the assumption of supervised-learning training on a static dataset, showing a connection to loss scaling (not Elo). To our knowledge, no model has been proposed to explain scaling laws in RL, where data and label distributions constantly change during training, with strong correlation to changes in the policy. This work is not a replication or validation of previous works, since no existing model can currently explain these scaling laws. Even given our results, we point out in the paper that a full model of RL/MARL scaling cannot be built without an extra missing part (the exact connection between value loss and Elo). We do believe though that our work provides a significant step towards understanding the full scaling mechanism. We understand that the goal of the paper should have been stated more clearly, and have changed the introduction text to fix that. - **Inverse-scaling explanation and connection to prior work:** Thank you for pointing out the connection to distractor tasks. We believe the connection to Oware's late-game states is more nuanced, since unlike distractor tasks, there seems to be no similarity between late- and early-game states. We updated the manuscript to include a detailed explanation of our hypothesis regarding the cause of AlphaZero's inverse scaling, backed with new results (section 7.3). In short, we argue that the mechanism causing inverse scaling is likely a combination of dormant neurons (see Sokar et al.) caused by non-stationary targets (=state values), coupled with the fact that late-game states have more stable targets than early-game states. We added a paragraph to this section discussing the similarity to distractor tasks (*'Comparison to inverse scaling in supervised learning'*). - **Related work:** To answer your question: this paper aims to present previously unexplored/unknown phenomena (Zipf's law in non-human games, loss scaling of an RL model in reverse order of complexity, AlphaZero inverse scaling) and provide hypotheses explaining them, often by drawing correlations to Michaud et al.'s scaling theory. Following your remarks, we changed the abstract and the introduction summary of our contributions, specifically making clear the separation between previous work and new results regarding Zipf's law in board games. We thank you for the citation and agree we should compare our inverse scaling model with it. We discuss it together with McKenzie et al.'s work in the new paragraph on similarity to distractor tasks (section 7.3). Citations: Sokar et al., The dormant neuron phenomenon in deep reinforcement learning. ICML, 2023. McKenzie et al., Inverse scaling: When bigger isn’t better. TMLR, 2023.
Response
I appreciate the authors' response. The updated draft improves coherence from a narrative standpoint, addressing my main concerns. I'm accordingly happy raising my score to 6.
Revision summary
We thank all reviewers for their time and effort, and for their valuable feedback. We were happy to hear that reviewers found our paper innovative, clear and well written, and that they liked the substantial amount of experiments we present. We have introduced several changes in the revised edition, which we believe strengthen the paper significantly. All reviewers were curious about our hypothesis on the origin of inverse scaling, prompting us to expand it and run a new experiment that supports our theory. Two reviewers (Tkxe and broL) significantly helped us expand our comparison to prior literature, providing us valuable feedback and relevant citations. We list below the additions to the paper: - A new section (7.3) where we explain in detail our theory regarding AlphaZero's inverse scaling, and present new results that support it (figure 8). Our results show large models selectively forget knowledge on early-game states while steadily improving loss on late-game states, supporting the theory that inverse scaling is caused by selective non-stationarity of the data, e.g. through dormant neurons (Sokar et al.). The GitHub repo contains code for the new results. - Section 7.3 also contains a discussion of distractor tasks from McKenzie et al. and their similarity to our case of inverse scaling. - Section 2 now contains a new subsection on scaling law models other than Michaud et al.'s that assume a Zipf distribution, where we discuss a wide variety of models and their relevance to our results. Many of our results that relate to the quanta model apply to these models as well. - We made changes throughout the paper to correctly frame the connection of our analysis to LLM theory, making the scope of our contributions clearer. - The paper's goal is stated more clearly in the introduction. - The Discussion section now contains a suggestion for a future line of research that utilizes our results for improved RL training. - Several other minor changes and additions following reviewer suggestions, improving overall clarity for the reader. We hope the reviewers find these changes satisfactory, and are looking forward to their feedback. Citations: Sokar et al., The dormant neuron phenomenon in deep reinforcement learning. ICML, 2023. McKenzie et al., Inverse scaling: When bigger isn’t better. arXiv preprint, 2023.
Meta-review
In this work, the authors discover power law frequency distributions within the AlphaZero train and test data, and show how agents minimize loss on game states in descending order of frequency (inverse scaling behavior). Reviewers found the paper to be well-written and broad in scope, covering substantial empirical analyses. Prominent concerns focused on the perceived limited practicality of the work, and its nature as simply a curious empirical study, with little justification for the assumptions made in the underlying exposition. In the early part of the discussion period, there seemed to be some confusion amongst reviewers about what the primary contributions of the work really were. Without the material in the discussion, I also struggled with this. The power law frequencies in AlphaZero are stated to have been discovered previously, but this isn't really emphasized in the report. The practical benefit comes from related work, but their discussion in the manuscript is rather limited. I appreciate that the authors have gone to lengths to address these concerns, but it is unclear that these fundamental misunderstandings would not carry through into the exposition of the final revision. I believe further refinement of the exposition, distinguishing the claims made here from the findings in prior work, and another round of reviews are appropriate.
Additional comments on reviewer discussion
Reviewer Tkxe expressed confusion about the point of the paper, and raised concerns about links with other related work. These concerns were addressed by the author rebuttal. Reviewer QW8y questioned the practical utility of the findings, and requested further experiments to better interpret implications to other RL environments. Concerns were partially addressed by the author rebuttal, after which the reviewer expressed some remaining confusion behind the intuitive explanations for the perceived Zipf laws. Reviewer broL was unconvinced by the connection to quanta theory, and requested some discussion on how these results relate to other literature deriving scaling laws from power laws in the data. They raised their score after the author rebuttal, but acknowledged some remaining concerns with the work. Reviewer khcs provided the most positive assessment, while criticizing the reliance on some vague assumptions stemming from the quanta theory. The author response addressed all of these concerns.