Thanks, I would still prefer some more interpretable comparisons
Thanks for continuing to follow up. I think this most recent response has helped me to achieve a bit more clarity out of what I believe would improve the paper.
I think fundamentally the issue is that performance on these tasks is hard to interpret without comparison to some baseline that is a "known quantity" that the reader will understand, and that supports the argument the authors are trying to make. That is, just as we measure distance with respect to some baseline unit, it is useful to contextualize performance on a task with respect to some well-understood comparison. While chance level is one such comparison (and RL research in AI often includes random baselines, even for multistep tasks), there are other approaches such as comparing to another well-known system or model. I will point to several approaches to this below.
1) The Tolman studies indeed use no-exposure as a baseline condition. This is a reasonably interpretable no-prior-experience baseline; and we interpret the rats as learning because they outperform it given prior experience. That is, this comparison supports Tolman's argument that latent learning occurs. I would be satisfied with such a comparison in this work; it would effectively fulfill the goal of what the chance level baseline was supposed to do, which is to provide a comparison of whether some sort of meaningful learning occurs. However, to my understanding the authors did not perform a no-exposure comparison (please correct me if I have misunderstood however).
2) many more recent computational studies make explicit comparisons to well-understood models. For example, if I recall correctly Momennejad et al 2017 compared human performance to a model-free baseline, a model-based planner, and SR. By making these comparisons to explicit computational models (in addition to the task condition comparisons), the authors more clearly highlighted what features humans seemed to use to solve their tasks. Again, these models provide benchmarks the reader can understand when assessing the performance of the system in question. Likewise, I think such baselines would be helpful for the corresponding tasks in this work; and more generally, comparing to well-understood models would help to provide a clearer benchmark for model performance.
3) alternatively, as I suggested originally, explicitly comparing to human performance can provide such a "known-quantity" baseline in some sense; while we don't understand human cognition in all its detail, it at least provides a familiar yardstick by which a system can be measured, and is a common standard in ML.
By contrast, comparing to another LM or even another task condition is harder to interpret; there are many factors that could affect performance in each condition, so it's not entirely clear what the comparison means, other than to show that one model is better or one graph is harder. This issue is exacerbated because the comparisons are not clearly presented, at least in the original draft of the paper. For example, if I were interested in transition revaluation in Table 2, which other condition would be relevant to compare it to in order to isolate this ability? And even if this were clarified, the core point remains: these comparisons don't seem to communicate how the reader should understand the performance of the system overall.
The authors note that "artificially imposing nontrivial chance metrics might unfairly bias the interpretation to make it seem like LLMs do better," and while I can understand this concern, I'm not sure that I agree it would bias the interpretation any more than leaving it out does. As highlighted in our discussion above, there are many versions of randomly acting baselines that could be used to compare different types of random behavior. What, precisely, is it that the authors think the models are doing that is better than these but doesn't count as some kind of real-but-imperfect planning? The authors could of course contextualize such results through many of the methods discussed above, such as comparisons to a well understood baseline model (such as a model-based planner, which presumably would be much better in many conditions), condition comparisons (e.g. "models only perform above random transition sampling on paths that have property X" or "as graphs get more complex, models get closer to chance"), or other metrics like quantifying % hallucinations (which I agree would be useful).
In summary, I feel the current results are a bit hard to interpret without comparison to systems that can be more readily understood.
The authors are, of course, also welcome to ignore these comments; I will re-emphasize that I find the experiments useful, and I have already increased my score to an accept (if a borderline one) in response to the reframing and improvements in the original response. Given that several other reviewers have given higher scores the paper may well be accepted regardless.