Summary
The paper introduces HIQL, a hierarchical algorithm for offline goal-conditioned RL. HIQL utilizes an action-free version of IQL to learn the value function and subsequently derives both high-level and low-level policies from this shared value function using AWR. The paper asserts that the hierarchical structure offers increased robustness to value function noise, leading to enhanced performance in achieving long-horizon goals. The effectiveness of the proposed method is validated through experiments conducted in both state-based and pixel-based tasks.
Strengths
* Overall, the paper is well written and well organized.
* The proposed method HIQL incorporates IQL into a hierarchical framework, similar to prior offline RL work POR. However, HIQL incorporates additional designs from hierarchical RL, including predicting latent waypoints and k-step waypoints. Consequently, it is a new approach from both directions, offering valuable insights for tackling offline GCRL with hierarchical RL.
* The examples in Section 4.1 and Section 4.3 are helpful for readers to understand.
* The experiments are thorough, showing improved performance on both state-based and image-based tasks.
Weaknesses
* The given examples show that the hierarchical structure is more robust to value noises in discrete tasks. Generally, in continuous tasks, the value function tends to be smoother. This raises the question of whether the performance improvement observed in HIQL is solely attributed to the hierarchical structure. If the claim in the paper is right, a method using smoothed value function (e.g., [1]) without hierarchical structure may achieve comparable performance with HIQL. Additional explanation and comparison are encouraged.
* More information to reproduce the Figure 2 need to be provided.
* In the appendix, the authors concat [s,g] before feeding into $\phi$. Therefore, it is recommended that Figure 1 be revised to address this matter.
* In the experiments, all the baselines considered are related to (weighted) imitation learning, making the inclusion of additional baselines utilizing temporal difference learning [2][3] necessary for a more comprehensive evaluation. In addition, it is notable that HIQL leverages the advantage of goal relabeling, while baselines such as IQL do not utilize this technique. Despite this, IQL has already demonstrated strong performance on the pixel-based Procgen Maze benchmark. Therefore, it remains unknown if HIQL can achieve comparable performance when compared to IQL+HER.
* For the experiments, I encourage the authors to report the performance of the low-level policy without the high-level policy. This can effectively highlight the true usefulness of the hierarchical structure.
* There is an error in Eq (7) in which the variable g is mistakenly used to train the low-level policy, while $s_{t+1}$ in the expectation is not used.
[1] Hong Z W, Yang G, Agrawal P. Bilinear value networks[J]. arXiv preprint arXiv:2204.13695, 2022.
[2] Chebotar Y, Hausman K, Lu Y, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills[J]. arXiv preprint arXiv:2104.07749, 2021.
[3] Li J, Tang C, Tomizuka M, et al. Hierarchical planning through goal-conditioned offline reinforcement learning[J]. IEEE Robotics and Automation Letters, 2022, 7(4): 10216-10223.
Questions
* Additional explanation is needed to clarify how the hierarchical structure can be applicable in continuous situations, as well as a comparison with smoothed Q methods
* More information should be provided to enable the reproduction of Figure 2.
* Additional baselines for temporal difference learning and goal relabeling should be included in order to facilitate a more comprehensive and fair comparison.
* Authors are encouraged to report the performance of the low-level policy in order to emphasize the necessity of a hierarchical structure.
* Is it possible to expand the HIQL framework to include more than two levels? Considering that the parameter k is manually determined, the top level could be assigned a large k value, while a medium level would utilize a moderate k value.
* A fix in Figure 1 and Eq (7) may be needed.
## post rebuttal
I am delighted to see that the authors have addressed all of my concerns, and I am pleased to raise my score for this paper.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
One limitation mentioned in the paper is the necessity for deterministic dynamics. It would be beneficial for the authors to further elaborate on why disentangling the controllable aspects from the uncontrollable elements in the environment is advantageous.