We propose the Thinker algorithm, a novel approach that enables reinforcement learning agents to autonomously interact with and utilize a learned world model. The Thinker algorithm wraps the environment with a world model and introduces new actions designed for interacting with the world model. These model-interaction actions enable agents to perform planning by proposing alternative plans to the world model before selecting a final action to execute in the environment. This approach eliminates the need for handcrafted planning algorithms by enabling the agent to learn how to plan autonomously and allows for easy interpretation of the agent's plan with visualization. We demonstrate the algorithm's effectiveness through experimental results in the game of Sokoban and the Atari 2600 benchmark, where the Thinker algorithm achieves state-of-the-art performance and competitive results, respectively. Visualizations of agents trained with the Thinker algorithm demonstrate that they have learned to plan effectively with the world model to select better actions. Thinker is the first work showing that an RL agent can learn to plan with a learned world model in complex environments.
Paper
Similar papers
Peer review
Summary
This paper presents *Thinker*, a method for augmenting an MDP with a learned model so that model-free algorithms can perform “planning” on the augmented MDP. Specifically, Thinker creates a cross-product MDP between the “real world” and a learned world model. This augmented MDP has a union action set that enables an RL agent to either take an action in the real world at a given step, or take an action in the world model. Decision-making is broken down into K-1 steps of acting in the world model (“planning”), followed by a single step of acting in the real model, potentially exploiting information from the K-1 steps of simulation. The authors argue that this allows model-free algorithms to learn how to execute various common planning algorithms. Evaluation is performed on Sokoban and 57 ALE games. The authors find that IMPALA applied to the Thinker-augmented MDP achieves a higher solve rate for a fixed number of world steps than IMPALA applied to the base MDP, as well as better results than one pre-existing baseline. Analogous results hold on the ALE benchmarks.
Strengths
- **[S1]** The fact that this method merely augments the MDP with some extra transitions (and a learned world model) means that it is both conceptually simple and broadly applicable to many existing reinforcement learning algorithms. - **[S2]** Experimental evaluation is thorough. Sokoban is a genuinely difficult benchmark for planning (it’s PSPACE hard in general), and the paper also evaluates across 57 Atari games, which greatly reduces the probability that results were cherry-picked. The ablations in the appendix were also quite thorough. - **[S3]** The paper is tackling a problem of significance for the community (planning in reinforcement learning) and was clearly written.
Weaknesses
- **[W1]** It’s not clear how much the empirical benefits of Thinker come from actually doing search as opposed to just giving the model a more expressive feature space (since the policy can depend on the rollout history at each node), or something else of that nature. The fact that the algorithm’s benefits seem to plateau after K=10-20 planning steps is particularly surprising: traditional planning algorithms might take tens of thousands of node expansions to find a plan of tens to hundreds of steps. The ablations go some way towards resolving this issue, particularly the ablations that decrease K and the ablation that removes some of the redundant features from the state space. However, more analysis of what the algorithm is actually doing during the planning phase would be helpful. For instance, the paper could include a qualitative analysis of Sokoban rollouts that tries to explain what kind of planning the algorithm is doing. It could also include an extra ablation in which actions are randomly chosen during the planning phase, rather than being chosen by Impala—does learning the planner with Impala actually help? - **[W2]** As I understand it, the main contribution of the paper is a way of augmenting MDPs, as opposed to a specific algorithm. Indeed, the paper uses this as a justification for the fact that it only has one third-party baseline for Sokoban and one for Atari. However, the paper only applies the Thinker augmentation to one RL algorithm (Impala). This is not a fatal problem for the paper, but it would be valuable to see the Thinker augmentation applied to other model-free RL algorithms to verify that it actually does work well for algorithms other than Impala. - **[W3]** The paper appeals to the “generality” and “universality” of the Thinker augmentation, applying that it can express any (?) planning algorithm. I think this is over-claiming: the Gym-like interface available during planning steps makes it hard to implement even a simple algorithm like BFS, and I’m not sure how Thinker would express more complex algorithms like regression (backwards search from the goal) or partial-order planning, not to mention planning algorithms that require complicated heuristics. It would be helpful if the paper was more precise about what planning algorithms Thinker can and can’t express. Note that W1 is my biggest objection, since I feel it cuts against the core claim of the paper. I would really like to see this addressed in the rebuttal. W3 is a more minor objection but is pretty easy to address, so it would be good to see the paper updated to reflect that (or to see an argument in the rebuttal for why it can’t be done). W2 is a big request for more experiments & so I understand if the authors don’t have time or compute to do this; it’s more of a nice-to-have than something I consider essential to publication. I’m marking this a weak accept for now due to the issues listed above, but I’m overall pretty happy with the paper, and expect to upgrade my rating later so long as the rebuttal is reasonable & no other reviewers identify major flaws in the experiments or conceptual basis of the idea.
Questions
1. Please explain in more detail why you believe the benefits of this algorithm come from it doing planning (systematic search), as opposed to some other benefit of the Thinker augmentation (i.e. address W1 above). 2. Is there a crisp way of separating the class of planning algorithms that Thinker can emulate from those that it can’t? (W3)
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
I felt the limitations were pretty thorough, although I would have liked it if this section listed the expressiveness concern (W3) as well, or at least made more precise claims about the expressiveness of Thinker-augmented model-free RL algorithms.
Summary
The authors propose Thinker, a model-based RL algorithm that transforms a MDP into one where at each step, the agent is required to generate plans in imagination before selecting a real world action. Then through the RL gradient, the agent learns to plan in order to act. The authors get good results in Atari and Sokoban benchmarks.
Strengths
- An interesting way of combining planning and learning by integrating planning into the MDP itself. By viewing planning as part of the MDP itself, an RL agent maximizing this MDP will learn how to plan rather than relying on a given planning algorithm to maximize reward. - Good results on Sokoban, a hard planning task, and analysis (some issues, see below)
Weaknesses
There are two major concerns I have, one in experimentation and one in writing. **Experimentation issues**. - **Lack of baselines** In Sokoban, the work currently compares to one baseline, DRC in Sokoban. DRC is a good choice for a baseline, since it also is a "learning to plan" baseline. - However, it is worth comparing the learning to plan strategy against hand-crafted planning and other ways of using the world model for RL. For example, Dyna and model-based value expansion are two alternative ways to leverage a world model for RL. I would like to see competitive baselines from these families of MBRL agents (e.g. DreamerV3 and STEVE). - The authors claim to omit common baselines since they want to investigate the benefit of the Augmented MDP. If so, I would like to see the Augmented MDP be applied to a few other RL algorithms (Rainbow, DreamerV3, etc.). Then the focus isn't on raw performance, but rather how much Augmented MDP improves over the vanilla MDP. - **Low number of seeds (3)** reduces my confidence in the results. Please run at least 5 seeds before reporting metrics, and even better, use IQM and 95% bootstrapped confidence intervals. - **Dual network ablation:** There is no ablation on the dual network itself. What happens if we just use 1 big RNN with the same amount of parameters to predict all quantities? - **Parameters for Thinker vs. baselines:** Is it possible the gains from Thinker are just from the increase in world model parameters compared to baselines? **Poor Writing and Presentation.** This paper unfortunately suffers heavily from writing and presentation issues, which I believe will make it inaccessible to readers. The authors tend to write dense paragraphs about the important ideas and mechanisms, rather than summarizing points succinctly and relying on visual figures. It seems like the paper is easy to understand for those deep in the subfields of "learning to plan" and MuZero style of MBRL. However, readers not in this particular area will find it hard to grasp. L88 - The paragraph on explaining the planning stage is verbose and hard to understand. The provided Figure 2 hides a lot of the details in the planning stage. I was initially a bit confused between K and L. Thought that K was the max length of 1 rollout. But instead, it seems like K is the # nodes of a planning tree, which is composed of several rollouts all starting from the same root node. Would have been nice to just have a figure of the planning tree to avoid this confusion, perhaps Figure 2 can be updated to have these details? Some more explanation of planning reward is needed. How does this avoid penalizing searching for paths with low rollout? Is this because we are using Max instead of mean return?
Questions
I think there is a lot of potential in this paper. It needs a good amount of polishing on the writing side to make the idea digestible to a general reader. Next, the experiment section can be improved in a variety of ways - more baselines, ablations, and seeds. Some additional questions: Why is the non-stationary network needed? Couldn't we get value and policy distributions directly by evaluating the policy and value functions on the predicted states? Why does K=1 ablation perform poorly? I thought this would be equivalent to the Raw-MDP baseline. What is the agent policy vs. the model policy? Is the agent policy the actor in IMPALA and the model policy the world model's predicted action probabilities?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
1 poor
Contribution
3 good
Limitations
The limitations section is already quite fair. However, there are some missing pieces. 1. Computational Cost. MBRL algorithms can be very slow to run in comparison to model free RL. I would like to see more information about the computational and resource cost of Thinker, and how it compares to the model-free baseline. How many GPUs does it require, how long does it take to run, and what are the main performance bottlenecks? 2. Hyperparameter tuning. There seems to be quite a bit of hyperparameters.
Summary
- This paper introduces a novel algorithm, Thinker, which learns how to plan by interacting with a learned world model using “imaginary actions”—negating the need for hand-designing a planner. The algorithm produces SOTA results on Sokoban, and its efficacy on Atari 2600 is demonstrated as well.
Strengths
- The algorithm design is natural given the motivation of the work. - The introduced algorithm has a number of novel components, including the planning stage used over the augmented MDP, and the augmented reward, and the duel network architecture. - Thinker produces state-of-the-art results on Sokoban (over previous state-of-the-art, DRC), and in Atari 2600, the Thinker-augmented approach outperforms the non-Thinker-augmented baseline. - The writing is clear and relatively concise, the paper is well-structured, the significance of the work is well-communicated and compelling, and the related work section appears to be complete.
Weaknesses
- The authors note “as the goal of the [Atari 2600] experiments is to investigate the benefits of the augmented MDP, we do not include other common baselines here” (line 307). While this point is understood, the results on Atari would have been more compelling if Thinker were to have outperformed comparable learned planning approaches, e.g. those mentioned in related work. - Figure 1 is a bit difficult to read—it is hard to note the difference between each of the successive images. Better visualization to highlight the differentials (and zooming in on the level, maybe) could have been employed.
Questions
- In line 234, various approaches using gradient updates to plan within a planning stage are cited. These approaches do not appear to be used in evaluation as baselines/ablations—why not? - Could some of the improved results be due to the novel duel network architecture, rather than the new algorithm? I don’t see an ablation for this in the main body or supplementary material. - How were the output statistics chosen? Could alternatives have been used? - I am confused by the phrase “the real action at the first K-1 steps” (132), since I understand that the fist K-1 actions are imaginary. Is it possible the authors are trying to say that we’re not using real actions for the first K-1 steps? It seems like these real actions are not simply “not used”, but they don’t exist in the first place. The structure of the sentence also makes it unclear to me which “imaginary action” we are talking about (I would suggest maybe ditch the last comma for clarity, since this also corresponds to the last step, K).
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
- The authors note the limitations that (i) the algorithm carries a large computational cost, (ii) only rigid planning is supported, i.e. requiring agent to roll out from the root state and restricts it to planning for fixed number of steps, and (iii) a deterministic environment is assumed. I agree with this authors that these elements are appropriate for future work. - No other major limitations stand out to me.
Summary
This paper presents a model-based RL approach, Thinker, that learns a world model and plans to take actions by generating imaginary rollouts with the learned world model. The paper claims that Thinker is a general method for any RL algorithms and does not rely on any hand-crafted planning algorithm. Experiments were conducted on Sokoban and the Atari 2600 benchmark, showing better performance over model-free RL methods.
Strengths
1. The basic idea of learning world models and training model-based RL policies with the learned models is technically solid. Experiments indeed show better performance against model-free baselines 2. The method description is clear and it is good to see performance in different domains.
Weaknesses
1. I am very surprised that Thinker is not compared against model-based RL baselines. 2. The contribution is unclear to me: a) I do not understand the claim about not having a hand-crafted planning algorithm. Based on the method described in the paper, you did introduce an MCTS-type of planning algorithm. This claim needs to be explained and justified. b) What is the difference between your method and other model-based RL methods? Particularly, it looks similar to MuZero, except that you have a different network architecture for learning the world model.
Questions
Please address my questions in the review.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
1 poor
Presentation
2 fair
Contribution
1 poor
Limitations
It is hard for me to judge whether limitations have been sufficiently addressed since I do not even understand what new technical advances have been proposed here.
Summary
This work presents Thinker, a new model-based reinforcement learning algorithm that achieves state-of-the-art performance on the puzzle video game, Skoban. The main contribution made by the authors is using a dual network architecture which addresses the sample inefficiency problem that other algorithms such as MuZero have. This dual-network setup consists of two sub-networks, namely a stationary and non-stationary network. The stationary network takes as input the current state and a sequence of raw actions and predicts future states, rewards and episode termination probabilities. The non-stationary network takes as input both the stationary network’s inputs and its predicted next states. In addition to the supervised training loss over the four predicted quantities, the authors proposed another L2 loss that encourages the encoded representation of the stationary network’s predicted state to match the encoded representation of the true state observed. The encoder helps the non-stationary network focus on only encoding task-specific information. The static network encodes static policy-independent information, while the non-static network encodes information that changes as the policy changes. Lastly, the encoder is only updated using the loss of the non-static network. The static network uses this encoder, without updating it, and updates its future state predictions to minimise the L2 squared distances in latent space.
Strengths
The authors show that their proposed system achieves a new state-of-the-art result on Sokoban. Furthermore, they proposed using an L2 distance in a latent representation instead of using the full state. This makes the entire system much more scalable. They further provide a detailed breakdown of the algorithm in the appendices and open source their code.
Weaknesses
The main weakness I see is that the use of these dual networks is not well motivated. Along with a better motivation an additional experiment or two is needed to demonstrate that they outperform a single network setup. Furthermore, it would be helpful to directly compare against open-source versions of models such as DreamerV3 and MuZero. This can help motivate in what ways their approach is superior.
Questions
1. On line 245 you state, “ In contrast to these methods, our proposed dual network fits the next-state prediction with a feature loss that enables visualization and prioritizes the learning of task-relevant features.” How does a feature loss enable visualisation? Especially if the agent only needs to remember task-relevant information. When the stationary network predicts the next state, would it not be able to ignore parts of the state that the encoder does not use? Therefore the latent space distance is still close to zero, but the predicted state might have some missing information. 2. On line 210 you state, “It is important to note that g is not being optimized when minimizing this loss, and ŝt+l does not receive gradients from the non-stationary network, as the two sub-networks are separately optimized. Further details regarding the model’s training can be found in Appendix B.” If it uses the encoder from the non-stationary network would that not make the stationary network’s updates non-stationary as well? As the policy changes the encoder (g)’s parameters might change. Which in turn changes the next state representation that gets generated. 3. On line 192 you state, “Nonetheless, this approach suffers from sample inefficiency since it discards information from the future state, which carries a rich supervised learning signal.” In what way does the dual network approach address the inefficiency problem that you state other methods such as MuZero and DreamerV3 have?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
This proposed solution currently only works for environments with deterministic dynamics. Furthermore, the planning component is quite computationally expensive, which might make it too expensive to use in many scenarios. The main limitation I see is that there has not been enough evidence provided that the dual network setup performs better than alternatives. Especially since the stationary network actually uses components from the non-stationary setup. It would be of great benefit to have additional experiments that show the superiority of this architecture when compared to alternatives (such as predicting every value using a single network).
Summary
This paper presents the Thinker algorithm, a method that augments an MDP such that an agent takes 'imaginary actions' in a learned model prior to executing actions in the environment. This allows the agent to incorporate planning strategies into a policy learned with RL. State-of-the-art performance is achieved on Sokoban and competitive results are shown on Atari.
Strengths
- The paper is well-written and polished. - The appendix is thorough and documents implementation details and architecture details. Source code is provided for reproducibility. - The proposed algorithm for augmenting an MDP to induce planning behavior in an agent seems to be original, as is the architecture of the dual network for learning the model. The results on Sokoban and certain Atari tasks also appear to be quite strong. - An analysis of the learned behavior of agents (e.g., when resetting happens during planning, what types of imaginary and real actions are taken) is provided in the Appendix. This is useful for characterizing the behaviors of the learned policies.
Weaknesses
- Comparison is given on Sokoban to DRC and to Rainbow on Atari. Most significantly, I think this work would benefit from comparison to model-based RL baselines to help contextualize how well the Thinker algorithm performs. - The method has multiple moving parts and several involved design choices (e.g., what elements to include in the augmented state, non-stationary planning reward, dual network architecture), though ablations are provided for some of these choices in Appendix F. - Related to the above point, regarding the dual network. The paper notes that "one could adopt the same architecture and training methodology used by MuZero" but that "[MuZero] suffers from sample inefficiency since it discards information from the future state." It would be useful to provide a quantitative comparison/ablation here, where the encoder/unrolling function/prediction function are learned in the manner of MuZero, as described, in order to justify the proposed architecture. - The augmented MDP seems to increase the complexity of the MDP (in the state-action space as well as the time-horizon). An expanded discussion on the computational cost of both training and evaluation, compared to using the raw MDP and also to other model-based methods, would be useful.
Questions
I would appreciate clarification on the items described in weaknesses; some additional questions: - Is it correct that the augmentation increases the cardinality of the action space from |A| to 2|A|^2? Does this adversely affect the difficulty of learning a policy? - Is there an intuition for why the K=1 case does worse than in the raw MDP in Figure 5? - An ablation on the unroll length L is provided in Figure F.3 in the Appendix. Have the authors tested additional values <5 or >10 -- i.e., is there a clear trend here? I am also curious how sensitive the values of L and K are to a particular task.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
- The authors have noted some limitations in Section 6, including computational cost and rigid planning for a fixed number of steps, including determinism of the MDP.
Reply to Rebuttal
Thank you for the response. I think that the additional ablations (including on the dual network, since it is considered a significant contribution) as well as the additional model-based baselines are necessary additions to the paper, and am glad the authors have included these in their rebuttal. I continue to lean towards accepting the paper.
Thank you for the clarifications and new experiments. The additional baselines and evaluations clears up my concerns about experimentation. The new figure does explain the Thinker planning process more clearly. I see the point about K, L, and the fact that the tree can reuse nodes if there are repeat states. Could you also add a figure showing some example trees with varying K and L? That would be very easy for readers to quickly get the idea and not get confused from just reading the text. Updating my score to a 6. --- With my immediate concerns cleared up, I would like to add some more discussion. First, my opinion is that Thinker is tightly coupled with its model-based RL algorithm. However, conceptually, it seems like we can use other RL algorithms with the Thinker-augmented MDP. It would be very interesting to compare commonly used RL algorithms with their Thinker counterparts, like SAC, PPO, etc. Perhaps some RL agents will be better with Thinker, and others will not depending on their design choices. At the very least, the authors should include this in the limitations and pose it as a question for future work. Next, planning with K steps every env step seems costly. Can we come up with a version of Thinker that does variable length planning?
Thank you for your comment and the updated score. In the main paper, we will add a figure showing an example search tree with the corresponding imaginary actions, to better clarify the concepts of planning steps (K), maximum search depth (L), and the reset action. > First, my opinion is that Thinker is tightly coupled with its model-based RL algorithm. However, conceptually, it seems like we can use other RL algorithms with the Thinker-augmented MDP. Throughout our study, we employed IMPALA—a standard, model-free actor-critic RL algorithm—on the Thinker-augmented MDP. The preference for IMPALA over PPO was mainly due to its superior computational efficiency; IMPALA's design facilitates parallel threads for collecting self-play transitions. We postulate that the performance of PPO or SAC on the Thinker-augmented MDP would align closely with IMPALA's since all these are variants of actor-critic algorithms. We agree that it would be very interesting to experiment with how the Thinker-augmented MDP interacts with other model-free RL algorithms, especially value-based ones like Q-Learning. As cited in line 333 of the paper, *"Exploring how other RL algorithms perform within the Thinker-augmented MDP is an additional direction for future work."* > Next, planning with K steps every env step seems costly. Can we come up with a version of Thinker that does variable length planning? We have briefly experimented with introducing an extra action in the augmented MDP, so the RL agent can choose to stop planning and act. We found that the final performance at 5e7 frames is similar, and the number of planning steps is reduced by around half. We did not include this in the paper as (i) the focus of the paper is the performance of Thinker given a limited number of frames instead of wall time, (ii) as Thinker opens up a new way of using a learned model in model-based RL, we believe the algorithm should be as simple as possible so as to allow future work to build on it easily.
I want to thank the authors for their response and for the proposed updates to the paper. I think the updated network names are much better. Furthermore it is now easier to verify the necessity of using dual networks over just a single network. The results also seem to indicate that Thinker significantly outperforms Dreamer-v3 and MuZero on Sokoban when using environment frames as a reference. I have updated my score to 5. One question I still have is how Tinker compares against the other algorithms when considering wall clock time. Do the other algorithms perform better using this metric? If they perform better here, what is the main advantage of Tinker? Would it maybe perform better in environments with computationally intensive/slow environment steps?
Thank you for your comment and the updated score. Thinker does indeed outperform both Dreamer-v3 and MuZero on Sokoban when using environment frames as a reference. While Dreamer-v3 isn't specifically tailored for the planning domain, its performance understandably lags behind RL algorithms designed for planning. Regarding MuZero, its model's sample inefficiency means it needs considerably more frames than Thinker (it requires 20 billion frames for Atari). Moreover, substituting MCTS with an RL agent introduces greater flexibility in both planning and acting. For example, if planning proves non-essential, the RL agent can bypass the search and behave like a typical plan-free RL agent; or, as Appendix G demonstrates, an RL agent might choose to stick to an existing plan unless encountering an unfavorable leaf node, which is not possible with MCTS. When we take wall time as a reference, the same conclusion holds - Thinker outperforms both Dreamer-v3 and MuZero on Sokoban when using wall time as a reference. As we mentioned in our response to Reviewer WUsk, Dreamer-v3 and MuZero require similar or longer training times. Wall time is not the primary focus of our paper, and various modifications and implementation details can enhance Thinker's wall time. For instance, substituting the RNN in the actor-critic with an MLP and reducing the planning steps from 20 to 10 could decrease the training time by over 70% while maintaining similar performance levels. Our interest mainly lies in evaluating the performance of an RL algorithm based on a fixed number of frames, a more typical reference point as seen in the Atari-200m benchmark.
Thanks for your response
After reading the responses, I understand better about the contribution. I also appreciate the additional results. I have increased my rating accordingly.
The authors have addressed my questions adequately and I maintain my positive opinion of the paper.
Good rebuttal
### Overall take * The authors have addressed all concerns that I thought needed to be addressed before publication. * I'm in favor of acceptance and have updated my score. * I've read the other reviews and they didn't give me reason to change my score much either way given the author rebuttal. ### (R1) Algorithm’s benefits seem to plateau after K=10-20 planning steps is particularly surprising > Note that if the better performance of Thinker stems merely from the expanded feature spaces of the model, ... then one would expect the rollouts in runs without planning rewards to be completely random [because] entropy regularization would push all non-useful actions towards a uniform distribution. This is a good point that I hadn't considered. > The new experimental results in Fig A3 provide more direct evidence of the benefits of the learned imaginary actions. Thank you! A3 resolves my main concern here. ### (R2) Thinker augmentation applied to other model-free RL algorithms > the substantial computational cost associated with the algorithm prohibits us from doing so Fair enough. The existing experiments look pretty brutal, computationally. Well done for getting through all of them! ### (R3) Planning capacity under the augmented MDP > We will revise the paper (e.g., replace line 30 any planning algorithm with common forward-based planning algorithms) and include the following discussions in the appendix: ... Thanks a lot, this is very precise and I think strengthens the paper.
Decision
Accept (poster)