Iteratively Refined Behavior Regularization for Offline Reinforcement Learning

Response to Reviewer ByMR

## Q: The performance improvement seems to primarily stem from comprehensive parameter tuning, rather than the proposed iterative refinement of policy constraints. A: In Section 5.1, we conduct an evaluation of CPI within two GridWorld environments. We consider two baseline algorithms: InAC, a method that guarantees to find the in-sample softmax, and a method that employs policy iteration with behavior regularization (BR), which could be viewed as an extension of TD3+BC for the discrete action setting. Both CPI and InAC converge to the oracle across various $\tau$ and environments settings. In contrast, BR underperforms when a larger $\tau$ is applied on 7*7-GridWorld and FourRoom-random. This evaluation ensures no parameter tuning while still demonstrates the effectiveness of CPI. , which proves the performance improvement primarily stem from the proposed iterative refinement of policy constraints. While when function approximation is applied, tuning parameters becomes a common issue of almost all SOTA offline RL algorithms. For example, XQL[1], an in-sample optimization method, is relies on tuning the hyper-parameters according to Figure 7, 10 and 11 in the appendix of the original paper. In practice, we believe it's unlikely that a method could dominate all tasks without tuning parameters. ## Q: TD3+BC might also achieve in-sample optimality and meanwhile prevents OOD issues with a carefully swept conservatism strength $\alpha$. A: The critical hyper-parameter in TD3+BC is $\alpha$ value, which controls the weight of RL learning and behavior cloning process. The $\alpha$ is set 2.5 by default in the original paper in Mujoco tasks, and we indeed find that the value of 2.5 works well only in Mujoco tasks. Thus, as you suggested, we conduct a hyperparameter search in the ranges {0.0001, 0.05, 0.25, 2.5, 25, 36, 50, 100} for Antmaze and Adroit datasets, and report the results in our results tables. It can be found that changing $\alpha$ can indeed improve the performance of TD3+BC. However, the performance of CPI is still significantly better on Antmaze and Adroit, which proves the effectiveness of the mechanism for iterative refinement of policy for behavior regularization in CPI. | | TD3BC (default) | TD3BC (with swept best hyperparameter ) | CPI | CPI-RE | | ---------------------- | --------------- | --------------------------------------- | ----------------- | ----------------- | | antmaze-umaze | 78.6 | 78.6 （$\alpha=2.5$） | **98.8$\pm$1.1** | **99.2$\pm$0.5** | | antmaze-umaze-diverse | 71.4 | 71.4 （$\alpha=2.5$） | **88.6$\pm$5.7** | **92.6$\pm$10.0** | | antmaze-medium-play | 3.0 | 35.7$\pm$11.0 （$\alpha=36$） | **82.4$\pm$5.8** | **84.8$\pm$5.0** | | antmaze-medium-diverse | 10.6 | 17.5$\pm$0.43（$\alpha=25$） | **80.4$\pm$8.9** | **80.6$\pm$11.3** | | antmaze-large-play | 0.0 | 0.0$\pm$0.0 （$\alpha=50$） | **20.6$\pm$16.3** | **33.6$\pm$8.1** | | antmaze-large-diverse | 0.2 | 0.2 （$\alpha=2.5$） | **45.2$\pm$6.9** | **48.0$\pm$6.2** | | pen-cloned | 5.13 | 64.2 $\pm$17.3（$\alpha=0.0001$） | **71.8$\pm$35.2** | **70.7$\pm$15.8** | ## Q: how do CPI and CPI-RE perform using one group of hyperparameters for each types of tasks A: We are still working on the experiments. We will post the results when the experiments finish. ## Q: term $\tau(1-\lambda)$ A: Thanks for pointing this out. We have modify this in the revised paper. ## Q: In proposition 1, the (34) would be better (5). A: This is due to a replicated label used in the same equation within method and appendix. We have revised them. ## Citations [1] Garg, Divyansh, et al. "Extreme Q-Learning: MaxEnt RL without Entropy." *The Eleventh International Conference on Learning Representations*. 2022.

Additional concerns

Dear authors, I appreciate the authors for their detailed responses. Some of my concerns are addressed. However, I still have further concerns. 1. Regarding this response, >In practice, we believe it's unlikely that a method could dominate all tasks without tuning parameters. As the authors proposed a relatively simple method, I believe this kind of simple method should be robust enough for different tasks using the same hyperparameters. Otherwise, the simplicity of the proposed method goes nowhere as we still require a lot of hyper-parameter tuning to obtain a good result. For example, TD3+BC/CQL/IQL/POR/EDAC in the paper all used as little as possible hyper-parameter tuning to ensure good results. Therefore, based on the fact that CPI and CPI-RE thoroughly tuned the hyper-parameters, other baselines should also be well-tuned to ensure a fair comparison. 2. Why does Table 1 bold CPI and CPI-RE for some tasks that are inferior to IQL, such as some antmaze tasks?

Response to Reviewer cTmM

## Q: There's a noticeable discrepancy between the theoretical framework and the practical implementation. A: The theoretical analysis considers the **tabular setting** with finite state and action space. In this setting, CPI converges to the optimal policy that are well-covered by the dataset, commonly referred to as the in-sample optimal policy. However in practice, implementations often rely on a limited number of gradient descent steps for optimizing when function approximation is applied. To address this issue, we introduce another behavior regularization to ensure the stability of the implementation. The mismatch between theory and practice is only due to function approximation and using gradient descent as optimization tools. Such an issue cannot be avoided and also exists in previous algorithms. For example, to learn the in-sample max, IQL requires solving expectile regression [1] (similarly, XQL [2] requires solving gumble regression), but practical implementation of IQL also depends on a few steps of gradient descent. The in-sample softmax policy iteration introduced in InAC also uses samples from the current policy in the policy evaluation step [3]. ## Q: There are two hyperparameters in ISPI, which may be hard to tune in practice. A: We provide the settings of regularization parameter $\tau$ and weighting factor $\lambda$ used in our main experiments in Table 4 of the appendix. The parameters $\lambda$ are settled by sweeping in [0.1, 0.3, 0.5, 0.7, 0.9] and choosing the best performing one. For ease of tunning, we also provide an empirical suggestion in Section 5.3.4 for the potential algorithm practitioners. ## Citations [1] Kostrikov, Ilya, et al. "Offline Reinforcement Learning with Implicit Q-Learning." *The Tenth International Conference on Learning Representations*. 2021. [2] Garg, Divyansh, et al. "Extreme Q-Learning: MaxEnt RL without Entropy." *The Eleventh International Conference on Learning Representations*. 2022. [3] Xiao, Chenjun, et al. "The In-Sample Softmax for Offline Reinforcement Learning." *The Eleventh International Conference on Learning Representations*. 2022.

Response to Reviewer cTmM

Thanks for your reply. It seems that the reviewer judge our contibution only based on terminology: "In-sample algorithms only uses samples from the data. CPI doesn't do this. Therefore in-sample optimality cannot hold for CPI. " In Section 3.1, we prove that (Theorem 1) in the tabular case, CPI converges to the in-sample optimal solution, defined by Eq.6. That means, CPI enjoys the same theoretical guarantees as BRAC, IQL and InAC in the tabular setting case. If the reviewer believes that this is not true, we kindly ask the reviewer to point out which step of our derivation is wrong.

Response to Reviewer JFNQ

## Q: Proposition 1 references an equation found in the appendix A: This is due to a replicated label used in the same equation within method and appendix. We have revised them. ## Q: the algorithm box lacks clarity A: We provide the pseudocode for both CPI and CPI-RE in Algorithm 1. For line 9, it's true that the reference policy is the optimal policy between $w_{1}$ and $w_{2}$. As in CPI-RE, two policies are trained. For example, when $w_{1}$ is updated, we compare the Q-value of actions from $w_{1}$ and $w_{2}$. If the action of $w_{1}$ has higher Q-value, we will choose $w_{1}$ as the current reference policy. In this situation, only the behavior cloning term plays a role, which avoids the better policy $w_{1}$ being dragged down by the inferior policy $w_{2}$. Else, $w_{2}$ will be chose as the current reference policy. We try to express the sequential updating of $w_{1}$ and $w_{2}$ (i.e., update $w_{1}$ first, then update $w_{2}$) using the word cross-updating, we have modified this to avoid misunderstanding. Thanks for your advice. ## Q: the paper's primary contribution is a trick A: While we appreciate the reviewer's comparison with TRPO and PPO, our approach offers a unique contribution. We don't merely employ a target policy network for KL regularization but derive it from the conservative policy updates in the tabular offline RL context. In other words, the iterative refinement of policy constraints is the core idea of our work and adding the behavior cloning term is to help the implementation when function approximation is applied. This is not just a 'trick' but a significant enhancement to offline RL both in the theoretical aspects under the tabular settings. In the practical evaluation, we also outperform existing SOTA offline RL baselines. ## Q: Evaluating only on TD3+BC doesn't provide enough evidence to support the universal benefits of adding conservative policy learning. A: We add conservative policy learning on IQL and test the variant on the following datasets. For each dataset, the weight of conservative policy learning is set to 1. For other parameters, we follow the original settings in IQL. Below are the final average normalized scores across three seeds trained after1M gradient steps. | | IQL | IQL with conservative policy learning | | ------------------------- | ----- | ------------------------------------- | | halfcheetah-medium | 47.4 | 48.3 $\pm$ 0.3 | | halfcheetah-medium-replay | 44.2 | 44.5 $\pm$ 0.5 | | halfcheetah-medium-expert | 86.7 | **94.6 $\pm$ 0.1** | | hopper-medium | 66.3 | 58.6 $\pm$ 6.6 | | hopper-medium-replay | 94.7 | **100.6 $\pm$ 0.4** | | hopper-medium-expert | 91.5 | **112.3 $\pm$ 0.4** | | walker2d-medium | 78.3 | 76.4 $\pm$ 3.4 | | walker2d-medium-replay | 73.9 | **84.3 $\pm$ 3.1** | | walker2d-medium-expert | 109.6 | **111.2 $\pm$ 0.9** | Note that while it's efficient to add conservative policy learning on IQL, it may depart from the original intention of our algorithm design. This is because in CPI and CPI-RE, the behavior cloning term is used to assist the iterative refinement of policy constraints to ensure the the learning policy staying in the dataset distribution. While in the implementation in IQL with conservative policy learning, the iterative refinement of policy constraints becomes an auxiliary loss. ## Q: In Figure 1, does the dashed line represent a BC trained solely on the curated dataset? A: The dashed lines indicate the performance of BC policies trained on filtered sub-datasets consisting of top 5%, median 5%, and bottom 5% trajectories. ## Q: The paper notes that $\pi_{w}$ is initialized as $\pi_{D}$. Could you specify where this initialization is reflected in the algorithm box? A: In the tabular setting, Proposition 1 establishes that the exact solution of Eq.9 remains within the data support when the actor is initialized as $\pi_\omega=\pi_{\mathcal{D}}$. However, practical implementations often rely on a limited number of gradient descent steps for optimizing Eq.9. This leads to policy optimization errors which are further exacerbated iteratively. To overcome this issue, we find it is useful to also add the original behavior regularization. In summary, adding this term is used to help the policy remains within the data support, thus there is no further need to initialize $\pi_\omega=\pi_{\mathcal{D}}$ in the practical algorithm. In our experiments, we also didn't find clear differences of initializing or not initializing $\pi_\omega=\pi_{\mathcal{D}}$. Thus in the practical algorithm box, there is no need to perform the initialization.