Summary
This submission studies offline RL with function approximation. Under the relatively mild assumption of partial coverage and realizability of the function approximations, the paper proposes the algorithm M(S)QP that approximates the (soft-)Q-function by solving a minimax optimization problem, and establishes guarantees of $L^2$-convergence of the Q-function estimator and the sub-optimality gap of the learned policy.
Strengths
A novelty of this work is that it studies offline RL based on soft-Q-function, while previous works mostly focus on the standard Q-function or the LP formulation. There are also several technical improvements compared to prior works, including $L^2$-convergence of the estimators and slightly milder technical assumptions.
Weaknesses
(1) The proposed schemes are based on minimax optimization and hence are mainly of theoretical interest, and the dual function $l$ is hard to interpret.
(2) The bounds on PAC sample complexity are either worse than the results in related literatures, or require an extra margin condition.
Questions
(1) In Remark 1, the authors remark that the minimax optimization is computationally feasible when $\mathcal{L}$ is chosen to be RKHS or linear function classes. I would like to know if there is a concrete class of offline RL problems where such a choice of $\mathcal{L}$ satisfies realizability, or otherwise this remark is vacuous.
(2) The bounds presented in this paper seem to be far from optimal (e.g. $O(n^{-1/4})$ convergence or $O(\epsilon^{-8})$ rate), and they are also worse than the results in previous works (e.g. the case $\alpha=0$ requires an extra assumption). I wonder if this is an artifact of the analysis or a price of milder assumptions.
(3) Line 303: "In the following, we demonstrate that MQP, which is a special version of MSQP when $\alpha \rightarrow 0$, can achieve a faster rate of $O\left(1 / \epsilon^2\right)$."
In my opinion, this remark is slightly misleading, because the faster $O\left(1 / \epsilon^2\right)$ rate is achieved under a stronger assumption of "soft margin". For example, in tabular offline RL with gap $\Delta>0$ and single policy concentrability $C^*$, the minimax optimal sample complexity is of order $\frac{SC^*}{\Delta\epsilon}$ (Wang et al., ignoring the horizon factor), which is better than the gap-independent sample complexity $\frac{SC^*}{\epsilon^2}$ as $\epsilon\to 0$.
Xinqi Wang, Qiwen Cui, and Simon S. Du. "On Gap-dependent Bounds for Offline Reinforcement Learning."
(4) Writing:
(a) The notation $\left\|d\_{\pi\_\alpha^{\star}, P\_b} / P\_b\right\|\_{\infty}$ is ambiguous, because the measure $d\_{\pi\_\alpha^{\star}, P\_b}$ is defined on both the space $\mathcal{S}$ and the space $\mathcal{S}\times\mathcal{A}$, and so does the measure $P\_b$.
(b) Line 187 & 188 (typos): $d\_{\pi\_{\alpha^*, P\_b}}$ -> $d\_{\pi\_\alpha^*, P\_b} $
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.