Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

Summary

This paper introduces A-Crab, an offline RL algorithm derived from ATAC, which incorporates a modified loss function for the Q-function. Instead of employing the square Bellman error used in ATAC, A-Crab utilizes the importance-weighted Bellman error. With this modified loss function, A-Crab effectively addresses some disadvantages of ATAC. These include alleviating the visitation coverage assumption, enhancing the suboptimality rate, and simplifying the minimax optimization process. Additionally, A-Crab inherits several benefits from ATAC. Theoretical findings presented in this paper demonstrate the superior advantages of A-Crab in comparison to ATAC.

Strengths

1. This paper is well-organized, presenting a table that compares provable offline RL algorithms, outlining the key distinctions of the proposed algorithm, A-Crab, in contrast to ATAC, and highlighting the advantages brought about by these modifications. 2. By simply incorporating the importance sampling ratio from the loss function of ATAC, A-Crab exhibits significantly improved characteristics, namely: 1) relaxation of the visitation coverage assumption, 2) enhancement of the suboptimal rate, and 3) simplification of the minimax problem into a maximization problem. These advancements result from non-trivial steps and can be accomplished by making only a few adjustments to the terms in the loss functions of ATAC.

Weaknesses

1. Despite the noteworthy novelty of this paper in terms of presenting theoretical advantages of A-Crab, the absence of empirical results diminishes its impact. I acknowledge that it is common to solely provide theoretical results in the realm of provable offline RL. However, considering that A-Crab is built upon ATAC, which offers both theoretical and numerical results, it would be feasible to compare the performance of A-Crab with that of ATAC. I believe that including such results would significantly enhance the paper. 2. The importance sampling ratio in offline RL can become excessively large due to the limited coverage of the state-action space in the offline dataset and substantial differences between the current policy and the dataset policy. Consequently, the A-Crab algorithm may encounter instability when attempting to learn a robust policy in practical scenarios. However, if this paper addresses this issue, it would substantially strengthen its credibility.

Questions

1. [Regarding Weakness 1] Would it be possible for you to present empirical results on D4RL datasets that compare the performance of A-Crab with that of ATAC?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The only limitation of this paper is the absence of empirical results. However, the implementation of A-Crab appears to be relatively straightforward, as it can be achieve by replacing the squared Bellman error in ATAC's implementation with the importance-weighted averaged Bellman error. Given that the implementation of ATAC is already available on GitHub, there is an opportunity to enhance this paper by presenting empirical results that compare the performance of A-Crab with that of ATAC.

Summary

The paper proposes an offline reinforcement learning algorithm called Actor-Critic Regularized by Average Bellman error (A-Crab). A-Crab modifies the pessimistic offline RL framework by replacing the usual squared TD error with an importance sampled TD error. Due to the linearity of the importance sampled TD error, overestimation does not occur, allowing the removal of the correction term. A-Crab achieves the optimal suboptimality rate with weaker assumptions and is computationally efficient. An improvement over the behavior policy is also guaranteed under a wide range of hyperparameters.

Strengths

### Originality The authors introduced a novel technique using the importance sampled TD error instead of the squared TD error. Due to its linearity, the correction for overestimation is unnecessary, allowing better suboptimality bounds and weaker assumptions. The analysis also becomes very simple with the help of Cauchy–Schwarz inequality. ### Quality I could not find any technical flaws in the arguments presented in the paper. ### Clarity The paper is overall well-written and easy to understand. ### Significance A-Crab achieves the optimal statistical rate of $1/\sqrt{N}$ with assumptions that are weaker compared to other provable offline RL algorithms. Also, using importance sampling instead of squaring can be useful in areas other than offline RL theory.

Weaknesses

1. The authors unrealistically assume the action space, the policy space, the importance sampling weight function space, and the value function space to be finite. 2. Experimental verification of the performance bounds on a simple toy example would be interesting.

Questions

Line 70 states that the A-Crab enjoys an optimal statistical rate of $1/\sqrt{N}$. Does this mean the optimality of $O(1/\sqrt{N})$ is theoretically proved? Or does this just mean that it is the best suboptimality rate to be discovered?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors do not address the limitations of their work. Assumptions on the finiteness of the action space, the policy space, the importance sampling weight function space, and the value function space can be viewed as limitations of this work. I believe this work does not have a potential negative societal impact.

Summary

The paper introduces A-Crab, which combines marginalized importance sampling with the actor-critic paradigm to achieve optimal statistical rate in offline RL. Fm theoretical analysis, this algorithm is also more computationally efficient and relies on a weaker average notion of policy coverage compared to prior work.

Strengths

1. The paper is well-written, effectively summarizing prior work and building upon it to propose the new algorithm. The notations are clear and consistent. 2. The algorithm itself exhibits strong theoretical properties, such as optimal statistical rate and efficiency, making it promising for offline RL problems.

Weaknesses

1. The empirical evaluation of the algorithm is a major concern. Since the theory only requires general function approximators, the gap between theory and practice may not be substantial. Experimental results help demonstrate the algorithm's effectiveness on computation cost and learning speed.

Questions

1. In Line 224, where does the overestimation come from? Please provide clarification. 2. In Line 248, why does ATAC need to optimize two functions, f and g? The previous mention in Line 214 only refers to f. 3. In Line 280, how does the theorem also demonstrate robustness against model mismatch? Please elaborate on this point.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the weaknesses section.

Summary

This paper proposes a novel algorithm called A-Crab (Actor-Critic Regularized by Average Bellman Error) for offline reinforcement learning (RL) in complex environments with insufficient data coverage. The algorithm combines the marginalized importance sampling framework with the actor-critic paradigm and addresses the challenges of handling high-dimensional observations and minimal data coverage. The paper presents sufficient theoretical analysis to demonstrate the advantages of the proposed algorithm over existing methods.

Strengths

- The paper proposes a novel algorithm that combines various techniques to address the challenges of offline RL in complex environments. The use of marginalized importance sampling and the average Bellman error regularization in the critic's objective are innovative and practical. - The theoretical analysis provides insights into the statistical properties of the proposed algorithm, with a focus on achieving the optimal statistical rate in converging to the best policy covered in the offline dataset. - The paper provides a comparison with existing provable offline RL algorithms, highlighting the strengths and advantages of the proposed A-Crab algorithm.

Weaknesses

- The paper provides no empirical evaluations or demonstrations of the proposed algorithm. Neither does it shed light on the design of practical algorithms. - There remain some issues unsolved in the paper. See the questions for details. - Some related works [1,2] use a similar optimization problem as Eq. (2) and also focus on computing an prioritization weight $w$ for Bellman error. They should be cited and discussed.

Questions

- There seems to be a lack of comparison between $C_{\text{bellman}}$ and $C_{l_2}$ : How to show $C_{l_2}$ is a weaker assumption than $C_{\text{bellman}}$? - How can the A-Crab algorithm shed light on the design of practical algorithms?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

References [1] Kumar, Aviral, Abhishek Gupta, and Sergey Levine. "Discor: Corrective feedback in reinforcement learning via distribution correction." Advances in Neural Information Processing Systems 33 (2020): 18560-18572. [2] Liu, Xu-Hui, et al. "Regret minimization experience replay in off-policy reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 17604-17615.

Response

I acknowledge the authors' rebuttal and I remain my rating towards acceptance.