Predicting Future Actions of Reinforcement Learning Agents

As reinforcement learning agents become increasingly deployed in real-world scenarios, predicting future agent actions and events during deployment is important for facilitating better human-agent interaction and preventing catastrophic outcomes. This paper experimentally evaluates and compares the effectiveness of future action and event prediction for three types of RL agents: explicitly planning, implicitly planning, and non-planning. We employ two approaches: the inner state approach, which involves predicting based on the inner computations of the agents (e.g., plans or neuron activations), and a simulation-based approach, which involves unrolling the agent in a learned world model. Our results show that the plans of explicitly planning agents are significantly more informative for prediction than the neuron activations of the other types. Furthermore, using internal plans proves more robust to model quality compared to simulation-based approaches when predicting actions, while the results for event prediction are more mixed. These findings highlight the benefits of leveraging inner states and simulations to predict future agent actions and events, thereby improving interaction and safety in real-world deployments.

Paper

References (28)

Scroll for more · 16 remaining

Similar papers

Peer review

Reviewer tXgP5/10 · confidence 4/52024-07-12

Summary

This paper investigates methods for predicting future actions and events of reinforcement learning agents during deployment. It compares the predictability of explicitly planning, implicitly planning, and non-planning agents using two approaches: an inner state approach that leverages the agent's internal computations, and a simulation-based approach that uses a learned world model. The result from the Sokoban environment shows that plans from explicitly planning agents are more informative for prediction than other types of inner states.

Strengths

- This paper addresses an important and practical problem in RL deployment - predicting future agent behavior. - The proposed method is well-motivated, with a clear analysis of its idea. - The structure and presentation of the paper are clear and well-organized.

Weaknesses

See the questions below.

Questions

- What's the rationality behind the choices in different methods? - For MuZero, why only use the most visited rollout as auxiliary information? Wouldn't using top-K rollouts provide more information? - For DRC, have you experimented with using only the final tick or a subset of ticks? - For IMPALA, have you tried using multiple layers or earlier layers? - For simulation-based approach, how do you handle the potential divergence between simulated trajectories and real trajectories over time? - How are the training/testing samples generated? Are they from trajectories with fixed length?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

None

Reviewer W8D87/10 · confidence 4/52024-07-12

Summary

This paper focuses on predicting future actions and events using policies learned by different types of planning agents. Namely, the authors distinguish between explicit and implicit planning agents, where explicit planning agents learn a model of the environment dynamics/transitions. Given this distinction between agents, the authors investigate how well these agents can reliably predict future actions or reaching a certain states within a finite time frame. They study the agents empirically in the domain of Sokoban and show the findings of their study.

Strengths

**Originality:** This is an investigative study on how the type of agent being trained influences its ability to predict future actions and events. While the prediction of future actions is not a new problem, the novelty arises from the study over different types of agents. **Significance:** I believe that the problem is interesting and understanding this topic can go along way in how users of AI agents can decide on which agents to train/deploy --- e.g., "does one learn an environment model or are the gains not worth the increase in complexity?", etc. **Clarity:** The paper was clear. **Quality:** The main motivation behind the paper is simple and well-defined. The problem being studied is an interesting one. The chosen experimental design is reasonable for what the authors are trying to show.

Weaknesses

- The results are plotting standard deviation instead of standard errors to show confidence intervals. - Predicting the event of reaching the blue tile seems like it may not be a good evaluation criteria. If the blue tile can be placed, with uniform probability, on any empty tile in the puzzle, then if the puzzle is large, the agent is very unlikely to reach the tile. This prediction task seems very trivial.

Questions

See my last point in the previous section.

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes, but these seem like things the authors should do to make the paper stronger.

Reviewer SSgd4/10 · confidence 4/52024-07-13

Summary

The paper presents a comparative analysis of future action and event predictions in Reinforcement Learning settings by comparing inner-step approaches (that predict via inner computations like by generating a plan or via neuron activations) and simulation-based approaches (that simulate a rollout in a learned world model). The results indicate that while the latter are superior in terms of plans generated, the former methods are more robust in future action predictions.

Strengths

1) The problem statement is cleanly defined, and is easy to understand the rest of the paper. 2) The paper provides a neat overview of each of the approaches compared via empirical results.

Weaknesses

1) The authors mention that simulation-based approaches are superior in terms of plans generated, but inner-state approaches are more robust in future action prediction. Why is this result surprising? What was the hypothesis to begin with? Can the contribution of the paper be re-stated to empirically showing an expected result? There is no issue if that is the case, but the fact that there is no mention of an initial hypothesis to draw comparisons between the two types of approaches is misleading. 2) What is the intuition behind the categorization of planning agents into explicit, implicit and non-planning? What definition of planning do authors have in mind such that these categories are important to talk about? 3) Line 176: Why are simulation-based approaches expected to predict the same state space as the input to the agent? Works such as Dreamer and Planet, as far as I understand, learn to predict the latent states in the world model. What is the reason if this specific class of approaches are being considered? 4) Why is Section 5 Related Works in the middle of the paper? Moreover, the bigger issue I find is none of the topics of Safe RL, Opponent Modelling in MA settings, and Predictability for Human-AI interactions are relevant as "related works". Indeed, these are important applications to the approaches being compared in this work, but I am not sure how they can be counted as related works. 5) What are the final takeaways of this work? Which approaches need to be considered for which respective problem settings? Minor comments: 1) Why is section 4 titled Methods? There are no method contributions in this work. Please rephrase it as Comparative Study or something like it. 2) Shift line 159-160 before line 145.

Questions

Please see the weaknesses section for question.

Rating

4

Confidence

4

Soundness

2

Presentation

1

Contribution

2

Limitations

1) Why do authors write that environments with diverse states do not exist? There are several RL benchmarks used for evaluating diversity in exploration. Is there any other type of 'diversity' that is being talked about here for which designing new environments will be required? 2) What is meant by 'false alarms' in line 350? Please rephrase the broader impact statement for better clarity.

Authorsrebuttal2024-08-12

Thank you for the response and the feedback. > I am still not convinced that the hypotheses and the empirical results go beyond anything currently not known about the different types of planning agents, particularly the ones considered in the paper. Errors in policy learning in 'explicit' planning agents based on an incorrectly learned world model is already well understood. Similarly, the results for 'implicit' planning agents are not surprising either. A more objective analysis, even for a single category of approaches, could make the paper much stronger. We respectfully disagree with the assertion that the hypothesis and empirical results are already known. To the best of our knowledge, there has not been any study on the predictability of different RL agents. We would appreciate any references to relevant papers if they exist. Our experimental results partially confirm our hypothesis, while some findings are relatively more surprising (as stated in the global response). It’s important to note that the error in policy learning is distinct from what we are considering here—we are investigating the predictability of RL agents, not the process of learning a policy. > I do not agree with the authors' definition of planning. We did not propose any new definitions for planning or a new taxonomy of RL agents in the paper. Our definition of planning is consistent with previous work, such as [1] and [2]. There are many proposed definitions of planning. We do not believe there is a universally agreed-upon definition. Recent works [3, 4] suggest that agents can also learn to perform planning with the correct architecture. Therefore, we believe that further distinguishing between explicit and implicit planning agents is beneficial for discussing the results, as labeling them as planning-free contradicts the findings of these works. This distinguishment between explicit and implicit planning agents is not novel, as it has already been addressed in the literature—see, for example, Table 5 in [5], which distinguishes between explicit and implicit planning. Developing a rigorous mathematical definition for explicit and implicit planning agents is far from trivial (and was also not attempted in [5]), as it is difficult to determine whether an agent learns to plan within its neural network. This challenge is well beyond the scope of our paper. In summary, our classification of agents follows existing work, and we have not introduced any new definitions or made any claims to contribute to the taxonomy of existing RL agents. We will reference [5] in the revised paper when discussing the agent types to clarify that this is not a new taxonomy. We included the taxonomy solely to aid in discussing the methods and results. [1] Chung, S., Anokhin, I., & Krueger, D. (2024). Thinker: learning to plan and act. Advances in Neural Information Processing Systems, 36. [2] Hamrick, J. B., Friesen, A. L., Behbahani, F., Guez, A., Viola, F., Witherspoon, S., ... & Weber, T. On the role of planning in model-based deep reinforcement learning. In International Conference on Learning Representations 2021. [3] Guez, A., Mirza, M., Gregor, K., Kabra, R., Racanière, S., Weber, T., ... & Lillicrap, T. (2019, May). An investigation of model-free planning. In International Conference on Machine Learning (pp. 2464-2473). PMLR. [4] Garriga-Alonso, A., Taufeeque, M., & Gleave, A. (2024). Planning behavior in a recurrent neural network that plays Sokoban. In ICML 2024 Workshop on Mechanistic Interpretability. [5] Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M. (2023). Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1), 1-118. > It is hard to interpret the proposed taxonomy as 'Methods' in the current version of the work. A proposed method is a formal framework and/or approach, which is not a contribution in the current version of this work. Section 4 proposes using inner-state and simulated rollouts to aid in predicting future actions and events. We introduced new types of additional information that can enhance prediction and experimentally demonstrated that they indeed improve prediction accuracy. A taxonomy typically involves a simple categorization of existing methods, but we have emphasized that both of these methods are novel. Simultaneously proposing two methods in a new problem setting does not constitute creating a taxonomy. We strongly disagree with the reviewer’s assessment that our two proposed methods should be dismissed as merely a taxonomy.

Reviewer SSgd2024-08-12

Response to Author Rebuttal

Thanks once again for your responses and respectful disagreements! I very well understand the authors’ perspective, and this discussion has surely been helpful to me to get a better understanding of the idea behind the work. I’ll maintain my current assessment and my incremented score.

Reviewer 7yyB6/10 · confidence 3/52024-07-14

Summary

The authors present two approaches for predicting future states and actions in an RL setting. This is done in a supervised setting, where the inputs are the state and action and the target is either the future action sequence or the time of occurrence of some event. The first approach leverages using the inner-state as additional information to the state action input. The second approach uses a model trained to simulate the environment to generate a rollout to serve as the additional information.

Strengths

While look-up tables of reward-action have been utilized before, the reviewer is not aware of any prior work that includes entire rollouts, or leverages external hidden states as predictors of future events or actions. The authors’ method is clearly explained. The experiments clearly demonstrate the authors’ claims and both metrics illustrate the capability of their approach in predicting future actions and events. The problem statement, prediction of future actions and the events that may be a result, is undoubtably a significant issue in the larger ML community as a whole.

Weaknesses

The reviewer’s primary concern is with data leakage that could occur with the rollouts used as additional information when training the model, especially with the ‘Thinker’ agent. If the action space is rolled out and appended to the input, would this not result in the potential future actions already being included with the input? The reviewer is also unsure as to the benefits of the method over, for example, training a world model, rolling out several series of actions and choosing the branch that accumulates the highest predicted reward to also serve as a predictor of future actions.

Questions

The reviewer invites the authors to kindly clarify if they have misunderstood any parts of the paper leading to the weaknesses outlined above. Beyond those related to the weaknesses, the reviewer does not have any other questions.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors acknowledge the limited range of RL agents and environments tested.

Reviewer SSgd2024-08-11

Response to Author Rebuttal

Thanks for your response! I have updated my assessment after going through the responses and the rest of the author-reviewer discussions.

Authorsrebuttal2024-08-12

Thank you for the updated assessment. Could you please share the specific reasons for rejecting the paper? It appears to us that the questions raised in the weaknesses section have been addressed adequately. We would appreciate any additional feedback on how we can improve the paper.

Reviewer SSgd2024-08-12

Response to Author Rebuttal

#### **[W1]** I am still not convinced that the hypotheses and the empirical results go beyond anything currently not known about the different types of planning agents, particularly the ones considered in the paper. Errors in policy learning in 'explicit' planning agents based on an incorrectly learned world model is already well understood. Similarly, the results for 'implicit' planning agents are not surprising either. A more objective analysis, even for a single category of approaches, could make the paper much stronger. #### **[W2]** I do not agree with the authors' definition of planning. Planning in its crude form can be understood as search, and what distinguishes planning is the fact that there are dependencies between states. The proposed taxonomy is simply making the distinction if policy learning is either based off on first learning an explicit world model, or learning latent states, or none of the above. The proposed definitions in the paper (lines 75-96) primarily are interpretations of different methods in literature. The paper does not provide any mathematical or objective definitions to distinguish between the different methods. It is not clear why this taxonomy was needed at the first place - the empirical questions of action prediction are event prediction are not sufficient to motivate this. Even if the current taxonomy is taken into consideration, it would have been much more useful to understand the reasons behind advantages/disadvantages of learning latent representations and/or a world model. #### **Section 4** It is hard to interpret the proposed taxonomy as 'Methods' in the current version of the work. A proposed method is a formal framework and/or approach, which is not a contribution in the current version of this work. The paper provides a taxonomy, which I believe is still not objectively defined, over existing approaches and performs a comparative evaluation among them. In summary, the paper's claims, writing, presentation, and structure, all require significant changes to make it a stronger contribution.

Reviewer tXgP2024-08-12

I appreciate the author's response, most of my concerns have been resolved. For now, I'll keep my score and continue to pay attention to other reviews and ongoing discussions.

Reviewer W8D82024-08-12

Response to rebuttal

I would like to thank the authors for their response to my concerns. As they have addressed the statistical concern and explained the other I will increase my score.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC