Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning

Summary

This paper proposes a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. The mutual information objective design is simple yet effective. Intensive empirical results show the superiority of the proposed method.

Strengths

The algorithm design, based on factored MDP, is well motivated. The empirical study shows that DUSDi consistently outperform other unsupervised skill discovery and unsupervised RL methods.

Weaknesses

(a) The algorithmic contribution is not significant compared with other MI-based unsupervised skill discovery methods. (b) The key objective design lacks theoretical support. As the author mentioned, Eq. (4) does not constitute a lower bound of the real objective. (c) A qualitative study to show if the learned skill embedding is semantically detangled would be beneficial. (d) The submitted code folder should include a readme file on how to reproduce the paper results.

Questions

Please see the weakness part.

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Provided.

Summary

The paper presents a method (DUSDi) for learning reusable skills through unsupervised interactions. Unlike existing methods that produce entangled skills, DUSDi focuses on disentangling skills into components that each affect only one factor of the state space. This enables efficient chaining of skills via hierarchical reinforcement learning. A mutual-information-based objective ensures skill disentanglement, and value factorization optimizes the process.

Strengths

1. The paper is well-written, and the empirical results show consistent advantages over baselines 2. The idea is simple and effective on the tested environments.

Weaknesses

1. The paper assumes that the skill space is discrete; there is little information on how the formulations generalize to continuous skill space. 2. There is a lack of visualization of the learned skill embeddings, and how different skills affect specific factors of the state space, highlighting the disentangled nature, which makes it hard to determine the contribution of the proposed method to the overall performance gain over various baselines.

Questions

Please address my concerns in the Weakness section

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

NA

Summary

Grounding on previous work on mutual information for skill learning, DUSDi proposed an algorithm for disentangled skill learning, which introduces several advances. First, DUSDi proposes to map factorized state components to factorized skill components. Then, it proposes the adoption of Q decomposition and Causal Policy Gradient to further enhance stability and efficiency.

Strengths

The work presents a stable and efficient approach for unsupervised skill discovery, which presents the following strengths: * **Simplicity**: the idea is straightforward and doesn't introduce complex components. Actually, through Q decomposition, DUSDi simplifies the learning of multiple skills. * **Empirical study**: the empirical study is extensive, it includes modern baselines, such as CSD and METRA and it highlights well the strengths of the approach. * **Presentation**: the presentation of the work is clear and of good quality. The additional visualizations on the website are interesting and further demonstrate the approach works as expected.

Weaknesses

* **State space factorization**: the approach works well thanks to the assumption that the state space is well-factorized. However, for POMDPs or high-dimensional environments, the algorithm may not work as expected. The authors show that object-centric approaches can obviate such weakness. This works well for simple environments where colours can easily help distinguish objects, e.g. the multiparticle env, but it may not work as well in more complex environments.

Questions

* the authors explicitly state that their objective is not optimizing a lower bound (line 150). Why not trying to actually keep a bound? e.g. following [1] * since the authors discuss the unsupervised RL benchmark, and they use a similar setup, why not presenting results on the popular benchmark? I understand their point about the fact that these environments mostly require manipulating a single variable, but being a more standard benchmark, it would have been useful to see complete results on it (not just in the Walker env) * what happens if you don't omit proprioceptive states from the MI optimization? (line 264-265) [1] On Variational Bounds of Mutual Information, B. Poole et al,

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The limitations of the approach are addressed in the paper.

Summary

This paper proposed DUSDi, which learned disentangled skills in an unsupervised manner. Specifically, DUSDi learns each skill component only affects one factor of the state space. DUSDi is evaluated in a specialized tasks comparing with skill discovery methods.

Strengths

- The proposed concept of disentanglement is novel - the paper is easy to follow.

Weaknesses

- It is not convincing how the disentanglement is beneficial in real-world scenarios. - The empirical evaluation is limited; standard benchmarks (e.g., D4RL) is not used, and it is questionable whether it can be used in more complex environments like pixel-based environments

Questions

See weaknesses.

Rating

4

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

See weaknesses.