Summary
This paper proposes a novel approach named Constrained Policy optimization with Explicit Behavior density (CPED) to mitigate the existing policy control methods, e.g., overly conservative or failing to identify OOD areas accurately. CPED use a flow-GAN model to explicitly estimate behavior policy density. The strength of CPED lies in its ability to accurately identify safe regions for explorations, which in turn, leads to less conservative learning policies.
Strengths
1. Well-motivated example in Fig. 1 and overall good writing.
2. Sufficient theoretical evidence to show that CPED has a fast convergence rate and is able to find the optimal Q-function value.
Weaknesses
1. The contribution is limited as it seems to directly borrow the idea of flow-gan into offline RL domains.
2. Besides the motivation example in Fig. 1, there is no faithful evidence to show that "accurately identify the feasible region, which includes both observed and unobserved but safe points" both in theoretical analysis and experiments.
3. Lack of survey and comparison with the most similar work [1], which also learns an expressive generative behavior model but by the diffusion technique.
4. Though there is an obvious edge of CPED against SPOT in experiments, more explanations are lacking to suppose it.
Questions
1. Could the authors provide more comparisons between CPED and SPOT and explain why CPED is more feasible?
2. Regarding policy learning (Eq. 7), why not transfer it to a Lagrange function like SPOT? To the best of my knowledge, explicitly excluding bad transitions has a little edge against other weighting or penalty methods in terms of generalization, which is a key contribution mentioned by the authors.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Lack of related work: the in-sample learning paradigm [2-8] is now one of the trends in offline reinforcement learning, those new methods are worth being well mentioned in related work.
---
I decided to change my score from 4 to 5 after the first round discussion with the authors.
---
[1] Chen H, Lu C, Ying C, et al. Offline reinforcement learning via high-fidelity generative behavior modeling[J]. arXiv preprint arXiv:2209.14548, 2022.
[2] Kostrikov I, Nair A, Levine S. Offline reinforcement learning with implicit q-learning[J]. arXiv preprint arXiv:2110.06169, 2021.
[3] Xu H, Jiang L, Jianxiong L, et al. A policy-guided imitation approach for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 4085-4098.
[4] Garg D, Hejna J, Geist M, et al. Extreme q-learning: Maxent RL without entropy[J]. arXiv preprint arXiv:2301.02328, 2023.
[5] Xiao C, Wang H, Pan Y, et al. The in-sample softmax for offline reinforcement learning[J]. arXiv preprint arXiv:2302.14372, 2023.
[6] Zhang H, Mao Y, Wang B, et al. In-sample Actor Critic for Offline Reinforcement Learning[C]//The Eleventh International Conference on Learning Representations. 2022.
[7] Xu H, Jiang L, Li J, et al. Offline rl with no ood actions: In-sample learning via implicit value regularization[J]. arXiv preprint arXiv:2303.15810, 2023.
[8] Hansen-Estruch P, Kostrikov I, Janner M, et al. Idql: Implicit q-learning as an actor-critic method with diffusion policies[J]. arXiv preprint arXiv:2304.10573, 2023.