Agents Explore the Environment Beyond Good Actions to Improve Their Model for Better Decisions

Summary

Model-based planning agents such as MuZero leverages a learned model of the environment dynamics to learn a policy to follow in the actual environment. By planning using the learned model dynamics, the agent may generate a stronger policy than when limited to only experience from the real environment. The drawback to this approach lies in the agent being forced into decision states it has not seen before. The agent's model will have incorrect value predictions for these states and the planning stage will result in weaker or sub-optmal policies in these states. This paper proposes an exploration scheme that ensures the agent explores sub-optimal regions of the search space to build better value estimates of the states within.

Strengths

**Originality:** The ideas introduced in this paper have not been published before as far as I can tell. **Significance:** The algorithm proposed by the authors is nominally significant. It is a naive implementation of the explore-first-then-exploit type of algorithms. **Clarity:** The writing itself of the paper was fair but could use considerable improvement. **Quality:** It is hard to assess the strengths in the quality of this paper.

Weaknesses

First off, I think it needs to be said that this is not a paper that can be improved upon with changes in the writing or presentation. The algorithm introduced is only a minor change to an already existing algorithm and it provides very little new insight. That being said, I do want to encourage the authors to continue learning and working in the field. For the authors, I would recommend attending workshop or course on technical writing and building a stronger background in RL and planning. For the submission specifically, I have a number of critiques. However, I will touch on only a few because as I've said this is not a paper that I think can be reworked and resubmitted: 1) The introduction of the paper goes well into page 3. A strong problem statement and motivation section can go along way in emphasizing the importance of the work but the paper needs to focus more on the actual contributions. 2) A similar note on the background and related works section. A lot of it deals with the history of the body of the work and not their relation to this particular paper. Furthermore, the background section does not provide the actual background needed for the paper. A good background section typically formalizes the problem state; clarifies any assumptions being made; defines notation and how the problem is being modelled; and provides a primer on the literature specifically needed to understand the rest of the paper. This submisison fails in those regards. 3) It is difficult to understand the algorithm introduced because nothing has been defined. 4) The experiments are limited a single small domain and does not give any indication of how the approach scales to larger ones. 5) The experiments do not provide comparisons to other baselines, that is, to other algorithms that take the explore-first approach which are, in my opinion, a direct competitor to proposed algorithm.

Questions

1) In equation 4, should this be the definition of $P_{normal}$ or are you defining $P_{exploring}$ twice? 2) On line 40, you initially restrict $T> 1$ yet $T$ is set to values outside of this range. What is going on? 3) Re Lines 45 - 49: Do agents not already make decisions about next actions? 4) What is Gumbel MuZero's policy or how is it learned?

Rating

1: Very Strong Reject: For instance, a paper with trivial results or unaddressed ethical considerations

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

1 poor

Contribution

1 poor

Limitations

The authors provide a list of limitations of their work.

Summary

The paper presents a method to improve the exploration of the MuZero agent in games. The authors propose an hybrid policy that mixes an exploratory policy and the optimized policy. The exploratory policy is meant to reduce brittleness of optimal policies. The new method is demonstrated on Tic-Tac-Toe.

Strengths

The paper addresses maybe the most important issue in decision making. The authors' approach to encouraging exploration in the decision tree addresses an important problem that could have significant implications if successful. Adversarial examples in games and self-play is certainly an interesting domain to investigate this. The motivation is good, and the authors also evaluate their approach on a small game. The idea is simple and makes sense. The appendix is also nice, and the source code is included. Prior work is also well researched. I like that the authors analyzed an example gameplay in Section 6.2

Weaknesses

Section 2 Recent Historical Background - For relevant work on adversarial policies, I like that you included “Adversarial policies beat professional-level 330 go ais. arXiv preprint arXiv:2211.00241, 2022.”. Another paper that investigates this issue and I think should also be included is then “Timbers, Finbarr, et al. "Approximate exploitability: learning a best response." Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 2022.” - When it comes down to AlphaZero extensions, a good addition in this section would be "Player of games." arXiv preprint arXiv:2112.03178 (2021).”, as it extends AlphaZero to imperfect information games Paper sometimes does not read that well and individual sections sometimes feel bit disconnected (i.e. little connection to other sections in terms of semantics and flow). E.g. the word curiosity is only really used at the beginning, and my understanding is that authors really just mean exploration (and indeed exploration is used in the rest of the paper). Also note that "curiosity" clashes with prior work on Intrinsic motivation (Oudeyer et al., 2007; Schmidhuber, 1991, ...). The idea is very simple, adding exploration at the beginning, which AlphaZero already does using different methods. I am not sure if the main evaluation in Section 7, namely Figure 5 is entirely fair (I might be wrong). The figure compares exploration-on vs exploration-off, but I don’t think that exploration-off in this case collapses to the “baseline” case? (e.g. MuZero or AlphaZero, as both do also force exploration at the beginning) This issue connects to the second one, which is that the experiments are not that well explained. It is fine that the authors moved many details to the appendix (which contains a lot of interesting information), but the main body of the paper still should include enough for people to understand what is presented. But my biggest issue is that the main challenge in balancing exploration in learning in games is when the approximator can’t really memorize the full game. If it could, than any exploration just works as we get to see all the states and memorize them (i.e. no generalization is needed). The game presented now is too small to see the effect, I suggest the authors run it on e.g. connect four. I believe the current experiments are simply not sufficient. Finally, I think the measure of whether the exploration helps or not should ultimately be exploitability rather than just a uniform measure of bad decision over all the game states.

Questions

Can you elaborate why you chose MuZero rather than AlphaZero to investigate the exploration issue? What’s the point of learning the model in your case? Can you also measure exploitability for exploration on/off? I believe that is the ultimate metric in your case, and I also believe the suggested exploration would indeed help there. How would the proposed method perform in more complex environments where the representational power of the estimator is limited with respect to the complexity of the full environment?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Addressed

Summary

Inspired by the failure of KataGo against an amateur-level agent, the author proposes to use an additional randomization scheme to encourage the agent to explore the less experienced part of the decision tree. Such a randomization scheme allows the agent to randomly deviate from the planned policy, and then switch back to learn the correct value function. Empirically, they evaluate their method on Tic-Tac-Toe and justify the effectiveness of their method.

Strengths

1. The problem of exploring the less experienced part of the decision tree is interesting. 2. The paper provides detailed related work.

Weaknesses

The main concern of this paper is the contribution is marginal. The proposed method is a minor modification (a randomization scheme) to the existing method. This is not to say the investigated problem itself is not important. But with such a minor modification, the paper needs to provide sufficient evidence to prove its effectiveness, either by theoretical analysis or a large body of experiments. However, I am afraid the paper provides neither. For the evaluation, the paper only evaluates the proposed method in a very simple scenario, Tic-Tac-Toe, which is way too easy. More experiments on more complicated scenarios are definitely needed. The writing is unprofessional and should be improved substantially. For example, references should not be included in the abstract.

Questions

n/a

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Limitations

Yes

Summary

To improve prediction accuracy, this paper divides the training phase's policy into exploration and normal policies. The proposed algorithm is tested in the game of Tic-Tac-Toe, and different noise strategies are introduced for experimentation.

Strengths

The paper introduces a novel method for increasing policy exploration and experiments with different noise strategies for exploration.

Weaknesses

1. The experiments lack persuasiveness as there is no comparison or theoretical analysis with advanced exploration algorithms. 2. The discrepancy between the mentioned "poor" predictions and the proposed exploration method is not fully addressed. 3. The setting of random time $t_{startNormal}$ is not clear.

Questions

See the weakness part.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The application of the proposed algorithm in more complex tasks and its effectiveness compared to algorithms with other exploration techniques remains to be studied.

Thank your for your rebuttal

I thank the authors for their rebuttal and answering the few questions I had. However, as they do not address the main issues that cause the paper to suffer, my opinion remains the same and I will not be changing my score.