Summary
This paper introduces Curriculum RL with Quantized World Model (CQM), a novel approach that leverages a VQ-VAE to create a discretized goal space and constructs a graph structure over it. CQM further proposes a curriculum strategy based on uncertainty and temporal distance to guide the learning process. The authors evaluate the effectiveness of CQM through experiments conducted on variants of PointMazes and AntMazes, which serve as benchmarks for Hierarchical Reinforcement Learning.
Strengths
- This paper is well-written and easy to follow up
- Extensive experiments
Weaknesses
One significant weakness of this paper is the lack of clarity regarding why using the representation from VQ-VAE is suitable for graph-building. It is crucial to consider the temporal distances between nodes to adequately cover the visited state space, especially given the limited number of nodes. The proposed goal representation learning scheme by VQ-VAE does not appear to take into account these temporal distances, raising questions about the efficacy of the proposed representation in creating a meaningful semantic goal space. It would greatly benefit the paper to provide a more thorough explanation and justification for the use of VQ-VAE in graph construction and its ability to capture temporal relationships.
Furthermore, the proposed goal representation learning scheme should be compared to prior work on representation learning, such as NORL [1] or LESSON [2]. A comprehensive comparison would help establish the novelty and effectiveness of the proposed approach in relation to existing methods. Additionally, it remains unclear how the proposed goal representation learning scheme outperforms or differs from the simple approach of utilizing farthest point sampling from the replay buffer, which warrants further investigation and comparison.
Moreover, the paper introduces new hyperparameters (\alpha, \beta, and \kappa) for curriculum goal generation. However, it is unclear how these hyperparameters were determined and how the performance varies when these hyperparameters are varied. Additionally, it would be valuable to compare the costs associated with hyperparameter search between CQM and the baseline methods.
[1] Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine, “Near-Optimal Representation Learning for Hierarchical Reinforcement Learning”, ICLR 2019.
[2] Siyuan Li, Lulu Zheng, Jianhao Wang, Chongjie Zhang, “Learning Subgoal Representations with Slow Dynamics”, ICLR 2021
Questions
- How does performance vary when \alpha and \beta are changed? Could you provide insights into the selection process for these hyperparameters? Is it possible to set them automatically?
- Regarding planning over the graph, what happens if an agent is unable to directly reach a specific node w_{i}? Could the system handle a scenario where, after TemporalDist(\phi(w_{i-1}), \phi(w_{i})), the agent conditions its behavior on reaching w_{i+1} instead?
- Could you please explain how the landmarks were sampled from the replay buffer for the "CQM Landmark from Replay Buff" depicted in Figure 7? What criteria or process were used to select the landmarks from the replay buffer?
Minor:
- It appears that there are two different symbols used for \beta, one in Equation 2 and another in \alpha = 1 / \max (\beta + \kappa D_{\text{KL}} (p_{g}^{f} || p_{ag}), 1). I would recommend using distinct symbols to avoid confusion.
- In Figure 7, the y-label representing the distance is partially hidden.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors adequately addressed the limitations and potential negative societal impact.