Summary
The paper proposes the VP-learning algorithm to solve offline goal-conditioned RL in the context of general function approximations. The algorithm is based on a previous empirically successful algorithm proposed by [1], and the author proves the finite sample complexity for VP-learning under mild assumptions. The proposed algorithm avoids minimax learning by using a duality formulation first identified by [1], which makes the proposed VP-learning algorithm friendly to implementation while also enjoying theoretical guarantees.
**References:**
[1] Ma J Y, Yan J, Jayaraman D, et al. Offline goal-conditioned reinforcement learning via $ f $-advantage regression[J]. Advances in Neural Information Processing Systems, 2022, 35: 310-323.
Strengths
- The VP-learning algorithm proposed in the paper enjoys finite sample guarantees under a mild single policy concentrability assumption. Compared with previous works on offline RL (single-task) algorithms with general function approximations, VP-learning does not involve minimax optimization, potentially making the algorithm more easy and stable to implement.
Weaknesses
- This theoretical paper aims to study the problem of offline goal-conditioned RL. But it seems that the whole theory does not essentially rely on the setting of goal-conditioned RL. The existing theoretical works this paper compares with are also for standard offline RL, which makes the motivation of the paper confusing. It's true that the standard offline RL can be interpreted as a special case of goal conditioned RL, but I hope to see more intuitions and messages on the problem of goal-conditioned RL itself.
- The key path through which the VP-learning algorithm can avoid solving minimax optimization problems is based on [1], which makes the theoretical contributions and insights of the paper weakened.
Questions
- Regarding Assumption 4 (single policy realizability), what is the $\alpha$ in the equation $\pi_{\alpha}^{\star}\in\Pi$? Since in Theorem 2 and 3 this assumption is imposed before the choice of $\alpha$, the $\alpha$ in the assumption seems undetermined. I think this needs clarification since the paper assumes that the policy class $\Pi$ is finite, which necessitates a specific choice of $\pi_{\alpha}^{\star}$.
- Regarding Assumption 5 (lower bound of policy), this seems a quite strong assumption which is not needed by previous single task offline RL algorithms with general function approximations, e.g., [2, 3, 4]. The author argues that the parameter $\tau$ can be very small, but a small $\tau$ will also increase the upper bound of the suboptimality of VP-learning due to Theorem 2&3. Is this assumption actually necessary for offline goal-conditioned RL? or is it only required by the specific analysis for VP-learning algorithm?
**References:**
[2] Zhan W, Huang B, Huang A, et al. Offline reinforcement learning with realizability and single-policy concentrability[C]//Conference on Learning Theory. PMLR, 2022: 2730-2775.
[3] Uehara M, Sun W. Pessimistic model-based offline reinforcement learning under partial coverage[J]. arXiv preprint arXiv:2107.06226, 2021.
[4] Xie T, Cheng C A, Jiang N, et al. Bellman-consistent pessimism for offline reinforcement learning[J]. Advances in neural information processing systems, 2021, 34: 6683-6694.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The above questions are potential limitations.