Evaluating the World Model Implicit in a Generative Model

Thank you for your rebuttal and the updated results. My responses are below. ## Summary of existing work ``` Your review states that we ignore the probe test of Li et. al [1]. ``` No, it doesn't. My criticism that *"The authors inaccurately summarize existing work and consequently address a non-existent flaw"* is directly related to the specific statement I referenced from your manuscript: *"Toshniwal et al. and Li et al. assess world models by measuring…"*. Given that your proposed metrics are motivated by the shortcomings of existing work, it is essential to faithfully represent those work and to identify actual, rather than imagined, flaws. Nonetheless, I consider this to be a relatively minor issue, one that can be addressed through some clarification, particularly when weighed against the validity of the proposed metrics. ## Validity of the metrics ``` We validated our metrics in the original submission by showing 1) they capture behavior other metrics don't and 2) they correlate with detour performance for the taxi exercise (Table 2). We now include additional validation for Othello using detours. ``` It's good to have multiple metrics, but it's also crucial that each one holds validity. I find it somewhat unclear how detour performance serves as a valid indicator in this context. Specifically, how can we be certain that the performance is solely influenced by the world model? For instance, could you clarify why $\text{Synthetic}$ appears to possess a near-perfect world model, whereas $\text{Championship}$ doesn't? To me, it's a typical phenomenon of exposure bias rather than a specific issue of implicit world model. $\text{Championship}$ is trained on expert data, where the prefix trajectories fall within a rather limited distribution. In contrast, $\text{Synthetic}$ is trained on randomly generated legal trajectories, which are distributed more evenly. As a result, the disparity between training and testing is significantly smaller for $\text{Synthetic}$ than for $\text{Championship}$. ``` Common decoding strategies do provide mechanisms for determining sequence acceptance, like top-k, top-p, and threshold-based sampling (i.e. epsilon sampling) ``` I respectfully disagree. It seems there might be some conflation of concepts here. DFA are designed to accept or reject sequences through checking transition rules. In contrast, auto-regressive generative models don't have such mechanisms. All they have are token-level distributions and they are not trained to accept all "legal" sequences. Although it's possible to implement post-processing functions to pull an acceptance label out of it, such functions can influence metric scores, potentially weakening the soundness of the evaluation. Particularly, it's well known that the commonly used sampling-based decoding methods, e.g. top-p sampling, tend to select sequences that diverge significantly from the generative models' modes. I'm looking at Table 2 (top-k sampling) in the new pdf. With k=1, the scores for $\text{Random walks}$ are 0.40, 0.69, and 0.30, respectively. However, when k=4, these values rise to 0.64, 0.98, and 0.51. The disparity is quite pronounced to me. When it comes to Table 3 (top-p sampling), the compression precision for $\text{Random walks}$ is 0.16 when p=0.9 but 0.73 when p=0.99. The results somehow confirm my concerns.

Thank you for your follow-up response. It helped me sort out some disagreements between us. I believe the root cause is this: "our metrics assess whether a model has a world model by assessing the outputs of the model. " While you state that you're evaluating the "implicit world model" of generative models, it seems to me that the outputs—and therefore the metrics—are influenced by various factors (such as the entropy of the learned distribution, decoding hyper-parameters, and algorithms) beyond just the implicit world model. ``` The detours exercise assesses the outputs of a model. A model whose output is consistent with the true world model will perform well on detours ``` I agree. But I'm not sure the model performs poorly on detours/proposed metrics if and only if it's inconsistent with the true world model. Even if there is some inconsistency, it might be exaggerated—or maybe even minimized; I'm not sure. The new results you presented earlier clearly confirm this concern. ``` And our results across thresholding parameters and sampling mechanisms point to the same final conclusions. ``` To be frank, I'm not sure I fully understand your point. If you're suggesting that the relative ranking of Random-walks > Shortest-paths > Noisy-shortest-paths is consistent across different decoding methods/hyperparameters, that's accurate. However, since you're proposing evaluation metrics, they should be applicable to more realistic models, not just those trained artificially. What happens if two models are trained on the same dataset, and their performance gap is much smaller? How can we ensure their relative ranking will still hold? This level of hyperparameter sensitivity raises concerns about the reliability of these metrics. On a related note, I have another question: why does Shortest-paths outperform Noisy-shortest-paths? If I understood correctly, Noisy-shortest-paths should involve more random exploration during training compared to Shortest-paths. ``` Our hypothesis as to why the Championship Othello doesn't have the correct world model is similar to your intuition -- that the way it's trained prevents it from differentiating between invalid moves and very bad moves ``` Indeed the gap between Championship and Othello in terms of differentiating between invalid moves and very bad moves are captured by the probe tests in Li et al. (2022). As shown in their Table 2, the error rate with Synthetic is significantly lower than with Championship. ``` To implement these metrics, we need to define what it means for a model to accept or reject a sequence. This isn't unique to our setting; one way that Li et al [1] and Toshniwal et al. [2] test world models is by looking at whether the top-ranked prediction of a model is legal. ``` Your setting is indeed unique. Sequence generation models are typically trained to maximize the probability of ground-truth sequences, making the selection of the top-ranked prediction—essentially MAP estimation—consistent with their training. While it's possible to use heuristics to sample multiple sequences from the distribution and argue that these sequences are "accepted" by the distribution (though what it means for a sample to be "accepted" by a distribution is unclear to me), this approach of inference differs from how the models are trained. This inconsistency arises due to the evaluation method, so it's questionable to blame the models if they don't perform well under these conditions. Moreover, if the top-1 prediction is typically used for reasoning tasks that rely on underlying world models, why should we be concerned with the consistency of top-$k$ predictions? ``` The dependence on a thresholding parameter is not a weakness; it's crucial for decomposing error into precision and recall and measuring AUC. ``` I'm not quite sure how those decoding hyperparameters function similarly to a classification threshold. With classification thresholds, it's easy to predict how adjusting them will affect precision and recall, making it a useful tool for balancing the precision-recall tradeoff. But when it comes to those hyperparameters and decoding algorithms, how exactly do they influence the metric values? For example, why does top-p decoding (p=0.999) result in a higher metric value for Random-walks compared to top-k decoding (k=4), yet performs worse for Shortest-path?

Thanks for continuing to engage with our paper. We appreciate your feedback and will clarify our paper. > _I believe the root cause is this: "our metrics assess whether a model has a world model by assessing the outputs of the model."_ We think a helpful comparison for our metrics is the next-token test performed by Li et al [1] and Toshniwal et al. [2]. Like our metrics, this test seeks to evaluate world models by the _behavior of their outputs_. Once we've established that a model behaves like it has the true world model, we can ask questions about the mechanism for why it behaves like that (e.g. via probing). For example, only after showing that OthelloGPT performs well on the next-token test do Li et al. [1] perform probing tests. The goal of our metrics is to test whether a model, when using common decoding schemes (we evaluate three different schemes), behaves like it has the true world model. For both our metrics and the next-token test, if a model's generations are influenced by factors like entropy, it would show up when practitioners decode from the model, and is therefore crucial for incorporating into metrics that assess world model behavior. At a high level, we believe that metrics which evaluate a model in a black-box manner, just based on observing their generative behavior, are vitally important and qualitatively distinct from those which rely on a certain implementation and/or internal parameters (we agree these are also useful). Several of the benefits of black-box metrics are the following - They directly measure the quality of the actual thing we care about: the sequences generated by the model. - They can be applied to any language generator, regardless of implementation. This also means that humans can be evaluated by the same measurements to get a baseline to compare models against. - They provide a definition of what it means to have recovered a world model which is generic. We believe that the question “does a language model have an implicit world model?” can and should be evaluated by metrics that look only at the outputs of the model. Prior works [1, 2] introduced the ingenious idea of testing this question on rules-based systems (i.e. games and other types of DFAs) where we know the true world model. Our interpretation of having a world model corresponds to recovering the state space of the system. Automata theory tells us that this is the correct way of understanding these systems. **Recovering the true states of the world corresponds to an intuitive notion of capturing the world model.** The key idea is using this connection to automata theory to understand how well states are captured just by looking at sequences. We believe this connection as well as the extensive evaluation on new and standard tasks for interrogating world models contributes an important contribution to this line of work. >_I agree. But I'm not sure the model performs poorly on detours/proposed metrics if and only if it's inconsistent with the true world model. Even if there is some inconsistency, it might be exaggerated—or maybe even minimized_ We're glad you agree that models whose outputs are consistent with the true world model will perform well on detours. Then we have the following: - If a model performs poorly on detours it doesn't behave like the true world model - Models like Synthetic Othello perform poorly on detours - This means they don't behave like the true world models - Previously proposed metrics suggest that all models we consider (including Synthetic Othello) do behave like correct world models - Our metrics correctly measure that they don't. This is a contribution of our work: we not only show that previously proposed metrics can lead to incorrect conclusions about world models, we also propose and validate new ones. We'd be happy to discuss any aspects of this further if you have questions or would like additional clarification. We agree that while poor performance on detours implies an incorrect world model, an incorrect world model can still perform well on detours. This is why it's important to have metrics that fully measure world model structure, like the ones we propose. Our detours exercise serves as one-way validation, which is the direction of disagreement between our proposed metrics and existing ones.

> _However, since you're proposing evaluation metrics, they should be applicable to more realistic models, not just those trained artificially. What happens if two models are trained on the same dataset, and their performance gap is much smaller?_ Like other papers about world model metrics [1, 2], we validate our metrics on models trained on artificial data. It is theoretically possible for model performance to depend on the decoding mechanism. It could imply that the models are so close in the amount of world model structure they capture that their performance depends on the decoding mechanism. You're correct that noisy shortest paths would involve more random exploration during training compared to shortest paths. It's a good question why it performs worse. One possibility is that it reaches a middle ground between random and constrained exploration that means it can't reap the benefits of either extreme. While we focus on world model metrics, it's important for future work to assess _why_ some models have world models and others do not > _Indeed the gap between Championship and Othello in terms of differentiating between invalid moves and very bad moves are captured by the probe tests in Li et al. (2022). As shown in their Table 2, the error rate with Synthetic is significantly lower than with Championship._ The best probe accuracy for the Championship model from Li et al. [1] is indeed lower than the best accuracy for the Synthetic model (90.6% vs 98.3%). However a challenge of probing tests is the difficulty of distinguishing between two high scores [3, 4, 5] (Li et al [1] make no difference in their conclusions for each model's world modeling capabilities). In contrast, our metrics clearly differentiate the two Othello models (e.g. 0.98 compression precision for Synthetic and 0.00 for Championship) which is validated on detours (e.g. 99% accuracy for Synthetic with 0.50 detours compared to 1% accuracy for Championship). > _Sequence generation models are typically trained to maximize the probability of ground-truth sequences, making the selection of the top-ranked prediction—essentially MAP estimation—consistent with their training_ Note that this is defining an acceptance criterion: top-k with k=1. Our metrics apply to this setting and we included experiments based on them. We also tested multiple ablations and two other sampling mechanisms to capture the diversity of ways in which it is possible to decode from LLMs. We believe it is important to consider various settings instead of constrain ourselves to a single one. > _Moreover, if the top-1 prediction is typically used for reasoning tasks that rely on underlying world models, why should we be concerned with the consistency of top-k predictions?_ We performed multiple top-k ablations because of your suggestion to consider top-k sampling. This was a great suggestion: Even if top-1 predictions are typically used for reasoning, it certainly isn't the only sampling mechanism, and our experiments show the robustness to sampling mechanisms. > _But when it comes to those hyperparameters and decoding algorithms, how exactly do they influence the metric values? For example, why does top-p decoding (p=0.999) result in a higher metric value for Random-walks compared to top-k decoding (k=4), yet performs worse for Shortest-path?_ There are patterns for how performance varies with threshold. For example, compression precision will typically increase as more sequences are accepted. This is because as more sequences are accepted, it is more likely that a model will accept a suffix for two prefixes that lead to the same state. While it's not an exact 1-1 mapping, we find this to be true for almost all models we consider. While we included top-k with the k=4 ablation for completeness, it is not an especially useful decoding mechanism for navigation. This is because there are only 8 possible cardinal directions, not all of which will be legal moves, and so decoding with top-k (k=4) can force the model to select sequences that aren't legal. [1] https://arxiv.org/abs/2210.13382 [2] https://arxiv.org/abs/2102.13249 [3] https://arxiv.org/abs/2102.12452 [4] https://arxiv.org/abs/1805.01070 [5] https://arxiv.org/abs/2004.03061

Dear author, I have some quick questions. > *The best probe accuracy for the Championship model from Li et al. [1] is indeed lower than the best accuracy for the Synthetic model (90.6% vs 98.3%).* Could you direct me to the specific paragraph or section where this result is reported? > *While we included top-k with the k=4 ablation for completeness, it is not an especially useful decoding mechanism for navigation.* But nearly all models achieve their best metric scores in this setting. No? I will reply to other responses later.

Thank you for engaging with our paper. We're glad our rebuttal addressed some of your concerns. Here we attempt to answer your remaining questions: - We want to clarify a potential confusion: while it's correct that any metric less than 100% implies there is no world model, the **metrics can also be directly compared to one another.** This is again similar to supervised learning: while there's a tradeoff between false positive rate and false negative rate, if one model always has a better false positive rate and false negative rate, it is doing better than another model, even if it's imperfect. Our metrics are not arbitrary scores; they are grounded in a theoretically-based formulation of what it means to learn the structure of a world model. We also validate this empirically: for each model (across maps and Othello), a model's ranking on our metric is exactly its ranking on the detours exercise (for all three metrics and both kinds of detour exercises). - Our results would still be interesting even if our metrics only captured whether there is a world model or not. This is because **prior metrics would lead us to conclude that all models we test across maps and Othello do have world models.** Our metrics are capturing something new, and this is not only motivated theoretically but also validated empirically on 1) map reconstruction and 2) detour exercises on Othello and maps.