Thank you for the response and the feedback.
> I am still not convinced that the hypotheses and the empirical results go beyond anything currently not known about the different types of planning agents, particularly the ones considered in the paper. Errors in policy learning in 'explicit' planning agents based on an incorrectly learned world model is already well understood. Similarly, the results for 'implicit' planning agents are not surprising either. A more objective analysis, even for a single category of approaches, could make the paper much stronger.
We respectfully disagree with the assertion that the hypothesis and empirical results are already known. To the best of our knowledge, there has not been any study on the predictability of different RL agents. We would appreciate any references to relevant papers if they exist. Our experimental results partially confirm our hypothesis, while some findings are relatively more surprising (as stated in the global response). It’s important to note that the error in policy learning is distinct from what we are considering here—we are investigating the predictability of RL agents, not the process of learning a policy.
> I do not agree with the authors' definition of planning.
We did not propose any new definitions for planning or a new taxonomy of RL agents in the paper. Our definition of planning is consistent with previous work, such as [1] and [2]. There are many proposed definitions of planning. We do not believe there is a universally agreed-upon definition.
Recent works [3, 4] suggest that agents can also learn to perform planning with the correct architecture. Therefore, we believe that further distinguishing between explicit and implicit planning agents is beneficial for discussing the results, as labeling them as planning-free contradicts the findings of these works. This distinguishment between explicit and implicit planning agents is not novel, as it has already been addressed in the literature—see, for example, Table 5 in [5], which distinguishes between explicit and implicit planning. Developing a rigorous mathematical definition for explicit and implicit planning agents is far from trivial (and was also not attempted in [5]), as it is difficult to determine whether an agent learns to plan within its neural network. This challenge is well beyond the scope of our paper.
In summary, our classification of agents follows existing work, and we have not introduced any new definitions or made any claims to contribute to the taxonomy of existing RL agents. We will reference [5] in the revised paper when discussing the agent types to clarify that this is not a new taxonomy. We included the taxonomy solely to aid in discussing the methods and results.
[1] Chung, S., Anokhin, I., & Krueger, D. (2024). Thinker: learning to plan and act. Advances in Neural Information Processing Systems, 36.
[2] Hamrick, J. B., Friesen, A. L., Behbahani, F., Guez, A., Viola, F., Witherspoon, S., ... & Weber, T. On the role of planning in model-based deep reinforcement learning. In International Conference on Learning Representations 2021.
[3] Guez, A., Mirza, M., Gregor, K., Kabra, R., Racanière, S., Weber, T., ... & Lillicrap, T. (2019, May). An investigation of model-free planning. In International Conference on Machine Learning (pp. 2464-2473). PMLR.
[4] Garriga-Alonso, A., Taufeeque, M., & Gleave, A. (2024). Planning behavior in a recurrent neural network that plays Sokoban. In ICML 2024 Workshop on Mechanistic Interpretability.
[5] Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023). Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1), 1-118.
> It is hard to interpret the proposed taxonomy as 'Methods' in the current version of the work. A proposed method is a formal framework and/or approach, which is not a contribution in the current version of this work.
Section 4 proposes using inner-state and simulated rollouts to aid in predicting future actions and events. We introduced new types of additional information that can enhance prediction and experimentally demonstrated that they indeed improve prediction accuracy. A taxonomy typically involves a simple categorization of existing methods, but we have emphasized that both of these methods are novel. Simultaneously proposing two methods in a new problem setting does not constitute creating a taxonomy. We strongly disagree with the reviewer’s assessment that our two proposed methods should be dismissed as merely a taxonomy.