Summary
This paper investigates the issue of mitigating estimation artifacts in the value function of off-policy RL. The authors adopt an unsupervised scheme to collect exploratory data, which is subsequently used to train both off-policy and offline RL algorithms. In their analysis, the authors identify a geometric characteristic known as $\mathcal{T}$-local optima, which represents inaccurate value landscapes. To address this issue, the authors propose a combination of model-based planning and graph-based aggregation, resulting in enhanced performance on long-horizon goal-conditioned tasks.
Weaknesses
* I have concern regarding the performance of the proposed method, as it fails to attain comparable results with previously published works, such as [1][2] on Maze2D, [3][4] on FetchPush, and [5][6] on Kitchen.
* Furthermore, the planning method incurs higher training and testing costs than prior offline goal-conditioned works [4][6], yet its performance does not meet the same standards. I recommend that the authors check this issue by conducting a comprehensive comparison, and elucidate the advantages of the proposed method over other offline goal-conditioned RL baselines and model-based planning baselines.
* The literature review in this work is inadequate as it fails to cite and discuss important prior offline goal-conditioned works [4][6][7][8]. Notably, the work by [8], which involves the creation of a graph from offline dataset, is particularly relevant. Furthermore, this paper overlooks prior goal-conditioned methods [7][9][10] that employ model-based planning.
* The evaluation benchmark appears to be limited, especially two environments are simple point-based tasks. To conduct a comprehensive study, it is essential to incorporate additional benchmarks such as AntMaze in D4RL, as well as the environments FetchSlide, FetchPick, and FetchHandReach used in [4].
* The paper would benefit from an improvement in its presentation and organization. A significant portion of the paper is devoted to discussing value learning, leaving only Section 6.3 for the main experimental results. To address this, I recommend compressing or relocating part of Section 2, Section 3, Section 4, and Section 5, perhaps to the Appendix. Additionally, it would be helpful to move some of the interesting evidence from Appendix C to the main paper.
[1] Chen X, Ghadirzadeh A, Yu T, et al. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 36902-36913.
[2] Janner M, Du Y, Tenenbaum J B, et al. Planning with diffusion for flexible behavior synthesis[J]. arXiv preprint arXiv:2205.09991, 2022.
[3] Charlesworth H, Montana G. Plangan: Model-based planning with sparse rewards and multiple goals[J]. Advances in Neural Information Processing Systems, 2020, 33: 8532-8542.
[4] Yang R, Lu Y, Li W, et al. Rethinking goal-conditioned supervised learning and its connection to offline rl[J]. arXiv preprint arXiv:2202.04478, 2022.
[5] Kostrikov I, Nair A, Levine S. Offline reinforcement learning with implicit q-learning[J]. arXiv preprint arXiv:2110.06169, 2021.
[6] Emmons S, Eysenbach B, Kostrikov I, et al. RvS: What is Essential for Offline RL via Supervised Learning?[J]. arXiv preprint arXiv:2112.10751, 2021.
[7] Li J, Tang C, Tomizuka M, et al. Hierarchical planning through goal-conditioned offline reinforcement learning[J]. IEEE Robotics and Automation Letters, 2022, 7(4): 10216-10223.
[8] Mezghani L, Sukhbaatar S, Bojanowski P, et al. Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping[J]. arXiv preprint arXiv:2301.02099, 2023.
[9] Nasiriany S, Pong V, Lin S, et al. Planning with goal-conditioned policies[J]. Advances in Neural Information Processing Systems, 2019, 32.
[10] Charlesworth H, Montana G. Plangan: Model-based planning with sparse rewards and multiple goals[J]. Advances in Neural Information Processing Systems, 2020, 33: 8532-8542.