Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning

Thank you for bringing up a good point. As we mentioned in our previous response, subgoal conditioning follows the prior work [Ref.7], and our explanation of the effect of the subgoal is summarized below. Let's first reconsider return conditioned supervised learning. In the case of return conditioning, the agent basically learns $a = f_\theta(s, \hat{R})$, where $\hat{R}$ is the return-to-go, i.e., sum of rewards received from the current timestep until the end of each episode. During training, the joint function $f_\theta$ of $(s, \hat{R})$ is learned with various $\hat{R}$ values from good or bad trajectories in the dataset. Basically we learn the function $f_\theta$ on the $(s, \hat{R})$ plane hoping for generalization over a wide region of $(s, \hat{R})$ with various $(s,\hat{R})$ data points. Later during the test phase, we set a high target return-to-go as $\hat{R}$ so that the model will output an action $a$ for each $s$ with high return-to-go. That is, we use $a=f_\theta(s, \hat{R}\_{target})$, a function from $s$ to $a$, where $\hat{R}\_{target}$ is a fixed high target value. Similarly, subgoal conditioning case can be considered to learn a function $a = f_\psi(s, \hat{s}\_{goal})$, where the subgoal $ \hat{s}\_{goal}$ is a randomly-picked state between the next step and the episode end in each episode. If $\hat{s}\_{goal}$ is from a bad episode, the produced action will be bad. If $\hat{s}\_{goal}$ is from a good episode, the produced action will be good. But, again, our goal is to learn an entire function $a = f_\psi(s, \hat{s}\_{goal})$ generalized over a wide joint region of $(s, \hat{s}\_{goal})$, where $\hat{s}\_{goal}$ ranges from good to bad (hoping for interpolation and extrapolation) so that in the later test phase, we use this function $a = f_\psi(s, \hat{s}\_{goal})$ generalized over a wide joint region of $(s, \hat{s}\_{goal})$ by fixing $\hat{s}\_{goal}$ to be the ultimate goal of the task, which is known for each task. We hope our additional response has addressed all your concerns. If your concerns have been resolved or if you have any further questions, please feel free to let us know.   [Ref.7] Emmons, Scott, et al. "RvS: What is Essential for Offline RL via Supervised Learning?." ICLR 2022.

We appreciate the reviewer's additional comments aimed at further clarifying our work.   ### **POR changed their official score after their camera-ready version.** In the POR GitHub, around Nov 2022, questions were raised regarding the reproduction of POR results. Since the POR authors could not access their previous code due to the copyright issue, they re-implemented the work and presented the results anew. While most of the results matched the previous ones, they did not match the hopper-medium-v2 dataset. As a result, they replaced the score with the new results and updated the arxiv version accordingly. Through a comment on GitHub, the author recommended using the new results instead of the conference version, which the reviewer had checked. The exact author's comment is **"We have updated our paper in the arxiv to reflect that and the mismatch results in the hopper-medium-v2 dataset (lower it from 98.2 to 78.6). If you want to compare POR with your work, you can refer to this paper version."**. Due to NeurIPS rebuttal policies, we cannot include external links, but we provide the address of this comment: github.com/ryanxhr/POR/issues/2. Therefore, our POR score reporting is accurate and aligns with the author's intent.   ### **Comparison with QT in Table R9.** Surpassing QT in 13 out of the 15 datasets we compared, with the remaining 2 scores being similar to QT within the range of standard deviation, cannot be a reason for rejection but rather demonstrates the superiority of QCS. (QCS mean 86.3 > QT mean 60.9) We want to emphasize that the offline RL benchmark features a variety of state and action dimensions, as well as different goals, making it a diverse and challenging benchmark where it is difficult for a single algorithm to perform well across all datasets compared to all baselines. For example, both POR and QT have instances where their performance is similar to or even lower than that of previous studies when examined closely at each dataset. For example, POR's performance of 76.6 on walker2d-medium-replay is lower than TD3+BC's 81.8, and QT's reported performance of 59.3 on antmaze-medium-diverse is lower than IQL's 70.0. However, POR and QT are recognized for their sufficiently good results due to their generally strong performance. **Slightly lower or similar scores to baselines on a few datasets cannot be grounds for rejection. We believe that what truly matters is how robustly the performance has improved compared to previous works.**   ### **Comparison of QCS to the Best Score** As mentioned in the previous response, we were unable to confirm the exact evaluation metric for POR. Since we have already compared the scores from the POR paper provided by the POR author in Tables 2 and 3 of our original manuscript, we conduct an additional comparison for QT with the best score. Table R10 shows the results of the comparison using the best score. As can be seen from the table, the QCS score has generally increased compared to the running average score that we previously reported. Compared to QT, it also shows superior results on average in MuJoCo (six datasets are better, two are similar, and one is worse), and especially demonstrates better performance with a large gap in Antmaze. While we knew that reporting the best score would naturally result in better performance, we want to emphasize again, as mentioned in our previous response, that we believe the last running average score is a more appropriate metric for offline settings, which is why we reported using this metric. We have demonstrated better performance than QT, but we would also like to clarify once again that QT should not be a reason for rejection, as it qualifies as concurrent work under NeurIPS policies.   **Table R10. Performance comparison between QCS (ours) and QT. We evaluate each algorithm based on the best score.** | Dataset | QT (best) | QCS (best) | |--------------------|---------------------|--------------------| | halfcheetah-m | 51.4±0.4 | **60.6**±0.3 | | hopper-m | 96.9±3.1 | **99.7**±0.3 | | walker2d-m | 88.8±0.5 | **92.3**±0.4 | | halfcheetah-m-r | 48.9±0.3 | **55.5**±0.5 | | hopper-m-r | 102.0±0.2 | **103.4**±0.5 | | walker2d-m-r | **98.5**±1.1 | **98.6**±0.3 | | halfcheetah-m-e | **96.1**±0.2 | 95.6±0.2 | | hopper-m-e | **113.4**±0.4 | **113.1**±0.6 | | walker2d-m-e | 112.6±0.6 | **118.3**±1.2 | | **average** | **89.8** | **93.0** | | antmaze-u | 96.7±4.7 | **100.0**±0.0 | | antmaze-u-d | 96.7±4.7 | **98.0**±4.0 | | antmaze-m-d | 59.3±0.9 | **100.0**±0.0 | | antmaze-l-d | 53.3±4.7 | **92.0**±7.5 | | **average** | 76.5 | **97.5** |

### **Observation of Q Overgeneralization is one of our contributions.** Starting from the 164th sentence in our manuscript (slightly below the sentence mentioned by the reviewer)—"We present a simple experiment to verify that learning $Q_\theta$ indeed induces over-generalization when trained on optimal trajectories."—this phenomenon is illustrated throughout Fig. 4 and Fig. 5 in Section 3.2.   ### **The topic 'Why Does the Q-Greedy Policy Struggle with Optimal Datasets?' is addressed in Section 3.2 by demonstrating the phenomenon of Q generalization and validating it through various methods across different domains.** Section 3.2 explains that the Q function tends to over-generalize when trained on an optimal dataset, which is why the Q-Greedy Policy struggles with optimal datasets. Throughout the section, we demonstrate that this overgeneralization phenomenon can particularly occur when training with an optimal dataset, using various domains including toy environment, Gym Inverted Double Pendulum, and MuJoCo Hopper, as well as various methods such as Q-value analysis and NTK analysis (please see Figures 4 and 5 in our manuscript). The appendix provides an in-depth analysis that couldn’t be fully covered in the main text due to space constraints, but it is already addressed in the main body.   ### **The results reported in Appendix E (the MuJoCo Hopper results) are additional analyses that connect to the Hopper results shown in Figure 5 of the main paper.** In Section 3.2, Figure 4, we present the results of a toy experiment, and immediately afterward, in Figure 5, we extend this experiment to demonstrate that a similar phenomenon occurs in both the Gym Inverted Double Pendulum and MuJoCo Hopper environments. We would like to note that the main paper includes not only the results of the toy experiment but also the results for Hopper. Therefore, the results in Appendix E are not a sudden comparison involving a change of domain and an expansion of the dimension from the toy example, but rather a more detailed analysis of the results presented for Hopper in Figure 5.