Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models

Summary

Builds up on previous works from [Li et al (2023)](https://arxiv.org/pdf/2210.13382) and [Nanda et al. (2023)](https://arxiv.org/pdf/2309.00941) to show that autoregressive transformer decoder models, when trained on chess moves, can learn to capture the board-state in its internal representations (latents). The authors also show that the model is capable of guaging the still of the player that the model is trying to imitate. And it is possible to improve the model's performance in certain settings with a simple intervention by adding a skill vector.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* Provides further evidence for the claim that large transformer models do not just memorize surface-level patterns but also capture the world-model in their latents and use it to make certain decisions. * Well written paper with strong experiments.

Reasons to reject

The paper is incremental in nature, directly building up on previous works. A *task* vector that makes the model perform a certain task, is also discussed in previously in the works ([Li et al (2023)](https://arxiv.org/abs/2306.03341), [Illharco et al (2023)](https://arxiv.org/abs/2212.04089), [Todd et al (2023)](https://arxiv.org/pdf/2310.15213)).

Questions to authors

1. "... To use this objective, we must train separate linear probes for predicting the board state at every white move and every black move ..." - I was a little lost here. Correct me if I am wrong, but this is how I understand it: A probe $P_{i, l, m}$ is trained for a square $i$, at a layer $l$, for a move $m$; where $m$ is either *white turn* or *black turn*? Or $m \in$ {1st white move, 1st black move, 2nd white move, 2nd black move, ...}? Either way (or I am missing the point completely) this should be clarified in the next revision. And if it is the latter, I am not sure how it follows the insights from Nanada et al. (2023). 2. Data for training the probes is not balanced - at any time you would expect there to be more blank or pawns on the board. How did you ensure that class imbalance is not skewing the accuracy? Consider also reporting the class-wise accuracy or F1 scores. Maybe also add a confusion matrix reflecting the quality of all the probes (combined for all configs) trained for some of the (best performing) layers. 3. On board state interventions, Li et al (2023)'s strategy using gradient descent is complicated, but it gives you enough flexibility to make sure that the intervention doesn't unintentionally effect the state of other squares. How does the simple addition/subtraction intervention ensure that? On Figure 5 you show that this intervention indeed has unintended side effects. Maybe gradients are the proper way to do this intervention? What's your take on this? 4. Altough not explicitly discussed in the paper, it seemed to me that the model is trying to imitate a player according to the estimated skill level. This reminded me of [Andreas (2022)](https://arxiv.org/abs/2212.01681). Maybe by adding the skill vector you can fool the model into thinking that it is imitating a stronger player, and vice versa. Is this a reasonable assumption? Some minor questions/comments: - Missing reference in the abstract "Li et al"? - Whenever you draw chessboards the x-axis should always be ticked from $a$ to $h$. This will help following the discussion. - "... do language models only form robust world models when trained on large synthetic datasets?" - I found this question a bit odd. Why the dataset has to be *synthetic*? [Li et al (2023)](https://arxiv.org/pdf/2210.13382) used synthetic data since they didn't have access to enough real data for training. It can be argued that it is the size of the dataset that matters, not whether it is synthetic or not. - Board state interventions, Table 2 => does the *no intevention - Modified Board* mean that you didn't do an intervention yet, but you are evaluating the legality of next moves on the board state you intend to achieve after the intervention? - A glossary of terms/phrases such as; "Elo Rating", "Stockfish" would be helpful. **EDIT**: Rating increased to **7** after rebuttal

Summary

This study revisits the debate about whether language models (LM) extract semantics and a world model from text, or only surface level statistics through learning to play chess games. Although previous work shows that LM can only learn a world model from synthetic data, the authors hypothesize that the root cause is from the small amount of human behavior data being used. The authors empirically prove this hypothesis by gathering a large amount of online chess game records played by humans, and train a LM to demonstrate it. The authors further attest this by conducting linear probing based on the LM’s internal activations to predict chess board states and player skills, which clearly aligns with the hypothesis.

Rating

9

Confidence

5

Ethics flag

1

Reasons to accept

1. The paper is well written, high-quality, and easy to follow. 2. The experiment design and results are well-positioned and convincing, rendering little space to challenge the idea. 3. The paper has a significant contribution to this line of work by clearly demonstrating that what is learned by LM is beyond surface statistics.

Reasons to reject

1. A few analyses can be added to support the claim. Check my questions.

Questions to authors

1. P5: for predicting the Elo, the current experiment uses the first 25-35 turns. Do you have studies on the difference between using less turns and more turns in this experiment? Also, why is the number of turns a range here, rather than a constant number? 2. Figure 3: The hypothesis to explain the layer performance difference is that the model learns the chess board state first and then be able to infer the player Elo and next move. How can we design an experiment to dive deeper into this hypothesis? 3. Figure 3: looking at the 16-layer model result, it seems that the model fully understands the chess board at 12th layer, while has a big performance jump in elo classification at 7th layer. Any potential hypothesis to explain this observation?

Summary

The authors study the ability of a chess playing language model to encode board state and player skill level. The paper includes layer-wise linear probing experiments, finding that both 8- and 16-layer models can predict board state very accurately at an intermediate layer and player skill classification with increasing accuracy in each layer. The paper also includes linear interventions, finding that the quality of play can be increased/decreased by adding/subtracting "skill vectors". This work extends earlier analyses of the capabilities of chess language models, and also adds to the literature on world modeling in language models (at least in highly structured "worlds") more broadly. Although Toshniwal et al. 2022 showed some world state tracking abilities of chess LMs (without probing the models layers), and Li et al. 2023 and Nanda et al. 2023 performed layer-wise probing experiments on Othello-playing LMs, I believe this paper provides the first analyses of layer-wise behavior of chess models. I find the results about player skill to be particularly interesting; the ability to judge and adjust player skill is less intuitive than board state tracking, which had already been studied previously. The paper is generally well written. The contribution is somewhat limited since only a single model (of two sizes) was studied, and since the chess task itself is naturally limited.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

+ Interesting results + Generally well written

Reasons to reject

- Limited setting - Unclear implications for other models

Questions to authors

- I didn't entirely follow the board state intervention experiment. I wasn't quite sure how the board is modified, or what is the difference between the board intervention (original vs. modified board) and the model intervention. I can imagine how it may have been done, but could you provide more details? - In the case of board state intervention, it seems that the modified board state could be an impossible state. Is that correct? If so, can you comment on how this might impact the results? - A more expansive discussion/conclusion would be helpful, describing how the authors think their work could impact a broader range of models/settings. Minor wording issues: - I don't quite follow "the model had a linear representation that could be recovered using linear probes." Could you clarify? - "In 2017, researchers trained a long short-term memory ..." -- this sentence doesn't quite parse. Maybe it's missing "and" after "reviews"?

Summary

### Summary of Paper This paper shows that a model trained on human chess gameplay learns to track position information and models player skill level (ELO). Causal experiments suggest that these found representations play a part in how the model makes its decisions. ### Summary of Review Overall, I think this work requires 1) additional experiments, 2) further depth for existing sections, and 3) some clarifications of current experiments and what the results imply. For example, they find that causally intervening on a skill vector can improve performance, but its unclear how that intervention is actually doing this or qualitatively what that change in behavior amounts to. As is, I think its hard to know what to conclude.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

* The ELO intervention finding is interesting. * With some clarification, the fact that interventions are all relatively simpl/linear (but still effective) is telling. * This work supports a growing body of research, replicating findings observed for othello

Reasons to reject

### Overview * Most of the papers' results would be the assumption given prior work (mcgrath and other probing over game papers like tomlin/forde et al + othello gpt, showing that models track game state.) Some readers would be unsure what to takeaway from reading this paper. * The paper doesn't uncover more on how such models are working under the hood; The exception here is that they do raise the possibility that the model would try to track multiple possible actions and then use the predicted player skill level to choose an appropriate move. Further testing this hypothesis would make this work stronger. * There are times in the paper where the setup is unclear. The hypothesis/alternatives are not always clearly defined nor how to interpret the results/metrics. ### What to conclude? It is hard to know what to conclude from reading this paper. W.r.t. to the positions of the pieces, I think the field understood that a model would track this information. Table 2 (for me) doesn't adjudicate on whether the found representations capture something meaningful. Spelling out the significance of the experiments and their conclusions would help. I think that the results say that 85.4% of the time the ablated piece is successfully removed and ignored simply by a linear intervention. Is that the case? I'm not perfectly confident that is how legal move rates should be interpreted here. W.r.t. to ELO, I think this is the most exciting part of the paper. Pushing this forward with more study and analysis would make this paper much stronger. (One could almost entirely for-go the position probing, or push it to the appendix.) An in-depth qualitative study of one (a few?) games and how the elo intervention changes the model's specific choices would be illuminating. How does the ranking of moves change with measured ELO? How does the ranking of moves change with intervened ELO? ### Reviewing Dimensions of the Work Okay ``Empiricism, Data, and Evaluation`` Okay ``Understanding Depth, Principled Approach`` * I think this paper requires more depth; more diving into the details and further evaluation. Ultimately, this approach is empirical and I think empirically evaluating and understanding what is going on would be helpful. See some of my questions/notes for specifics. Okay ``Clarity, Honesty, and Trust`` * Though I found some parts unclear, I think they positioned their findings with honesty/trust-worthiness. Limited ``Ambition, Vision, Forward-outlook`` Limited ``Technological Impact`` * Overall, this paper studied previously explored domains (chess board representations) without making a large contribution to further understanding that domain (chess board representations); I think their work does sketch some interesting hypotheses, but require further follow-up experiments.

Questions to authors

### Questions * How do the moves change when you increase the "elo" for the standard-board games? A qualitative or expert or in-depth analysis of a game on a standard board (with the model boosted to perform even better, getting +3% to +5% win rate, and comparing the moves before and after might (while potentially misleading), would be exciting. * How do move probabilities and rankings change given the ELO/intervened-ELO? * How good at predicting the lichess games did the models get? Did the error correlate with game ELO? I guess at some point low-ELO games are harder to predict than mid/high level games. For the first, the moves are more random; for the second, its "harder" to find the good moves the players find. * How do you remove piece (pawn at c2?) and have a valid PGN/board-state? * In table 2, bottom-right quadrant: This result says that giving the modified board to the model w/o intervention has a high error rate? What makes the model error here? * In table 2, top-left quadrant: does this mean that 85.4% of the time it picks a new non-ablated piece to move? Does it ever try to move the ablated piece? What happens in that remaining 10%? * Does the ELO probe work for ranges that it wasn't trained on? * Table 3: If you subtract random noise of equal magnitude to the elo-vector does the model reach a similar win-rate (11.9)? * Do you have results for probing for "my pieces"/"your pieces" instead of black/white? [Following https://arxiv.org/pdf/2309.0094] ### Notes * I find the move-board-state intervention logic confusing; the point is that you modify the state and then board and check that the model's move post the modified board and intervention state are the same? Writing more explicitly what you expect and what the performances mean would be helpful. * (suggestion) Adding board positions or labels to the axes of Figure 5 would help the reader track what C2 means. You could also add additional manual highlighting to the squares. * (nitpick) "it is recommended" in the first paragraph of "4 Model Interventions" sounds funny. I'd say, "Following Belnikov, 2022, we..." or directly argue for your case.