Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning

Summary

This paper considers goal selection in goal-conditioned reinforcement learning. The core idea of this paper is to group states with small temporal distance into clusters, and then select goals that are on the clutter boundaries. In addition, the method gives priority to goal states that are accessible to the agent. Once the agent reaches the goal, a go-explore style exploration can be performed to potentially add new visited states to the clusters. The authors test their system across 5 different domains, and show improved performance over previous methods.

Strengths

- This paper is well written and easy to understand. - The idea of learning state representations that can help group state and identify easy-to-reach states is novel and exciting. - The experiment is thorough and the results are promising.

Weaknesses

- It is worth pointing out that the method is only evaluated in relatively simple domains, with the maximum state dimensions being 29. - See the question section.

Questions

- Figure 9 seems confusing. It seems to imply that CE2 would bias the exploration direction towards task completion, which shouldn’t be the case considering that the training phase of CE2 is task agnostic. - For previous methods like MEGA or PEG, couldn’t you still use the temporal distance to select “next goal to explore”? Why is the clustering necessary here? - Is there a more rigorous way of understanding why “less explored regions are naturally adjacent to these boundaries”? This seems intuitive in the original state space but I’m not sure if it carries over to the latent state space.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation is briefly discussed in the conclusion section.

Summary

The paper introduces a cluster edge exploration (CE2) algorithm, which is implementing the “Go-Explore” principle in a – to the best of my knowledge – novel manner. Key idea is to use clustering of the state space latents – go to one of these clusters and then explore from there. As a main result, exploration improves and, as a consequence, success rates rise.

Strengths

The idea builds on prior work based on go-explore, is charmingly simple, and yields positive effects as expected. The model outperforms previous methods on standard tasks.

Weaknesses

The evaluations are rather restricted to some standard artificial benchmark tasks. The temporal distance network seems tedious to train additionally. The algorithm in the end just introduces yet another method to explore the edge of the search space. The clustering algorithm seems detached from the latent learning algorithm – it does not structure the latent state in any way.

Questions

Eq. 2 is not quite clear – particularly I wonder if here is not a unit problem as the first distance measure in latent state is representational while the second one depends on the movement distance estimates. Isn’t this severely restricting? Eq. 4 could you elaborate slightly? How are p and q densities represented / determined? Where does the learned value exploration function come from? What about environment, where sub-areas are hard to get to but do not offer themselves suitably for any further exploration.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

An integration / automatic abstraction via shielded / gated latents would be much more appealing. The evaluations are restricted to a standard test suite that does not really require deep reward propagation.

Summary

This paper develops an approach for frontier exploration in the context of model based reinforcement learning. The key idea of the paper (inspired by prior work Go-Explore) is to cluster a group reachable states in the latent space and keep track of the current frontier, such that new goals can be sampled close to the frontier. Experiments in simulated control benchmarks show that the proposed approach ism ore effective in solving hard exploration tasks compared to a few related baselines.

Strengths

The paper is well motivated and targets an important problem in reinforcement learning that has been studied for many years - that of effective exploration. Under the goal-conditioned RL setting, the proposed modification to a prior work Go-Explore is novel to the best of my knowledge, and is intuitively sound in terms of choosing goals that are near the frontier and yet "accessible" with a high probability. The paper is well presented, with sufficient background on prior works, and appropriate details on the algorithm and simulation environments used for experiments. The experiments show better results than baselines on some challenging exploration tasks like in-hand manipulation and point maze navigation. The ablation experiments are good and provide into different stages of the approach like goal sampling, reaching the farthest reachable state, and exploration heuristic beyond that.

Weaknesses

One of the main weaknesses of the paper is that the delta in terms of core algorithmic contribution beyond go-explore is limited. The paper pitches the core contribution as an exploration strategy but couples it with model-based RL in the instantiation, but the exploration algorithm on its own is a small modification to Go-Explore. A small change to an existing approach (Go-Explore) leading to massive improvements in tasks would be interesting, but the paper is not convincing in showing this. One reason for that is Go-Explore was shown to perform well in some very challenging scenarios (like Montezuma's Revenge) and the proposed approach doesn't perform direct comparisons on those environments. Combining model-based RL with exploration is a bit confusing. In many environments, a hard exploration problem needs to be solved (because of sparse rewards and long horizon tasks but learning a model of the world is an even harder task than exploring the environment to learn a policy - it seems the proposed approach will be very limiting in such scenarios)

Questions

Please refer to the weaknesses above. - The paper pitches the core contribution as an exploration strategy but couples it with model-based RL in the instantiation, but the exploration algorithm on its own is a small modification to Go-Explore. Is there a reason why this instantiation is necessary? - Can the authors comment on the choice of completely orthogonal environments to those in Go-Explore (which is a direct baseline) ? - How feasible is the approach for deployment in real world control systems where building a model of the world may be much harder than learning a policy to solve a task?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Please refer to the weaknesses and questions above, and address limitations of the approach.

Summary

This paper presents a method called "Cluster Edge Exploration" (CE2) to perform goal selection in goal-conditioned reinforcement learning and enable efficient exploration. Concretely, the method builds on the Go-Explore principle, which learns separate policies for exploration and goal-reaching. The main idea is to learn a latent embedding space parameterized by Gaussian Mixture Models (GMMs) in the Dreamer framework. Selecting a goal is a two-step process. First, the method samples points from the GMM and only keeps a low probability collection. These points approximate the boundary of the existing explored states. Next, the algorithm uses the goal-conditioned policy to perform an imaginary rollout towards each goal candidate. That goal is selected if the rollout ends at a state with the highest exploration value. In addition, when the test goal distribution is known, the CE2-G variant learns the GMM only with trajectory data from test goals. This further provides an inductive bias to sample goals near the test distribution. The paper presents extensive empirical studies in navigation, locomotion, and manipulation. Quantitatively, both CE2 and CE2-G outperform various baselines in their respective settings. Through visualizations of the sampled goals, CE2 bears the advantage that it samples points that are both near the frontier of the existing data and feasible to reach.

Strengths

- The method's main idea is sound: sampling goals that are feasible but underexplored makes sense. - The authors did a good job explaining the algorithm and training objectives. They also made the distinction between EC2 and EC2-G clear. - The experiments are done in a good span of different domains. This shows that the exploration strategy is not tailored only for a certain application, such as navigation. - Visualizations of the goals picked by the CE2 and relevant baselines are insightful.

Weaknesses

- It seems that some of the tasks haven't been trained to convergence. This makes it harder to draw definitive conclusions on the method's sample efficiency or final performance. - The method experiments appear to be done in state-based environments only. Since Dreamer is a strong algorithm for vision-based control, I'm curious how this approach performs with image observations. Would additional challenges arise for feature learning?

Questions

- Equation 7 performs imaginary rollouts toward the goal candidates. How is $T$ chosen here? - In the ablation studies, the relative performance of CE2 and CE2-noPEG seems stochastic. Why would the inclusion of exploration value estimate hurt performance? Does this mean that the exploration value function is under-trained?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, there's a dedicated section on page 15.

Summary

The authors propose a method for model-based exploration based on exploring reachable trajectories. The expand upon the Dreamer algorithm by optimizing the encoder to encode a notion of distance between state and optimizing for a likely reachable goal when training in the learned world model. The method is evaluated on a wide range of tasks, including dexterous manipulation tasks, outperforming other baselines in final performance.

Strengths

The paper is clear and the algorithm well-motivated. The strong experimental performance in the complex tasks is promising and they provide nice visualizations of their algorithm versus others and over time.

Weaknesses

One thing I would have like to see is some analysis of the additional computation time required to optimize for the goal states versus other methods. If it is prohibitively expensive, then the wall time could still be much longer than other methods.

Questions

Why do you think the method is not able to achieve 100% on pen rotation?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, on computational costs and if it can be applied in model-free settings

Rebuttal Response

I thank the authors for taking the time to answer my questions!

thanks for the response. my main concerns remain and so will not fight for acceptance

Dear authors, Thanks for the rebuttal response. The explanations to my questions are helpful, and I would definitely encourage providing the additional context regarding goal selection being the key proposed novel contribution in the revised paper. However my main concerns regarding comparison with Go-Explore more directly, limited algorithmic contributions, and empirical evidence for applicability to model-free scenarios remain. In particular, I think a more direct comparison to Go-Explore is necessary because "the strength of our goal selection method" is something subjective in the larger context of the exploration strategy i.e. it may be the case that Go-Explore's 'heuristic' goal-selection strategy is good enough for tough exploration problems and the proposed approach is actually not scalable to these challenging scenarios - without empirical comparisons, we simple don't know! (I understand that experimental comparisons might be beyond the scope of the short rebuttal window) Apart from my concerns, I do not see any major flaws in the algorithm/experiments, and as such I am not recommending reject, but will not fight for acceptance if any other reviewers have major concerns.

Discussion

Dear Reviewer WfSq, We kindly inquire whether the information provided in our rebuttal adequately addresses your concerns. If you have any further questions or issues, we would be grateful for the opportunity to address them. Thank you once again for your valuable feedback and constructive comments! Best regards, The Authors