Authors’ response to Reviewer 7X6Q
We appreciate the reviewer’s thoughtful and constructive feedback. We revised multiple text issues that the reviewer pointed out. Due to space limits, tables were deferred to our response to Reviewer HX7o or can be found in the paper.
## Training time for world model and for algorithms with and without COCOA
Learning dynamics is not a significant burden compared to policy training. The time taken to train dynamics models for each algorithm is about 5-10 percent of the policy training time.
For IQL as an example, it takes about 1-2 hours on training dynamics, just a few minutes for anchor seeker training and 13 hours for policy training on TITAN RTX GPU. The training time per epoch increased from 11 seconds to 24 seconds as of now.
We mainly focused on the performance improvement that COCOA can bring to existing offline RL algorithms rather than optimizing the running time of our implementation. The majority of the additional training time by COCOA involves frequent calls to the pretrained anchor seeker policy, the dynamics model and the main policy. We expect that training time with COCOA can be greatly reduced through the operator fusion using a toolbox like TorchInductor.
## Overall performance with COCOA
Upon investigation following the reviewer's suggestion, we present our updated experimental results with the layer normalization for the anchor seeking policy and the transductive predictor, which turned out to be the missing component for stable training of COCOA. The results are shown in Table 1 in the paper. Comparison with reproduced IQL performance on Gym-MuJoCo random tasks are also included in Appendix. Note that we add the layer normalization only to the COCOA-specific components for a fair comparison with the baselines. As a result, COCOA increases the average performance of all base algorithms evaluated on the D4RL Gym-MuJoCo benchmark. The winning task ratio also becomes 63%. This suggests that with a proper training stabilization, adding COCOA can be beneficial across diverse base algorithms.
## Other benchmarks for COCOA
Following the reviewer’s suggestion, we experiment with two more benchmarks - D4RL Adroit and NeoRL [5]. The outcomes of these experiments are summarized in Table 2, 3 in our response to Reviewer HX7o.
This broader analysis reveals that COCOA significantly enhances the performance of IQL and MOBILE in most tasks. For a detailed insight into the experimental setup, please refer to the updated paper.
Our method consistently enhances the performance across six tested D4RL Adroit tasks, showcasing its robustness and adaptability. In the difficult tasks like the door and hammer due to sparse rewards, the baselines work poorly, and our COCOA can still improve them. On the other hand, in the pen task, our method achieves significant performance improvements.
We also verify the effectiveness of our algorithm in the NeoRL benchmark, known for its conservative, less exploratory data generation compared to the D4RL Gym-MuJoCo. Although its state-action space coverage is highly limited, our preliminary experiments, conducted using MOPO, show significant performance improvement, suggesting potential benefits for other algorithms as well, which will be done for the camera-ready.
Due to the time constraint of the rebuttal period, we test COCOA on six D4RL Adroit tasks on top of IQL and MOBILE, with plans to expand to CQL, MOPO, and ReBRAC in the camera-ready, and initially tested only with MOPO for NeoRL, intending to examine more algorithms later.
## Comparison with ReBRAC
Comparing with offline RL methods that use policy regularization would make our results more comprehensive. However, due to the limited time frame, our work on incorporating ReBRAC with COCOA into a single framework is still in progress. We are hoping to include the results in the camera-ready version.
## Hyperparameter sensitivity
Regarding MOPO and MOBILE's hyperparameter sensitivity, due to the time and resource constraints, we could not perform exhaustive analysis like EOP. We found that the `rollout length` and `penalty coefficient` need to be slightly larger than those used in the existing MOPO and MOBILE methods. While MOPO and MOBILE conduct the searches for `rollout length` and `penalty coefficient` with 2 and 4, or 2 and 5 values respectively, we use our search with 3 and 5 values.
We appreciate the reviewer’s time and effort put into improving our paper.
[1] Kumar, Aviral, et al. (2020). Conservative q-learning for offline reinforcement learning. NeurIPS.
[2] Kostrikov, Ilya, et al. (2022). Offline Reinforcement Learning with Implicit Q-Learning. ICLR.
[3] Yu, Tianhe, et al. (2020). Mopo: Model-based offline policy optimization. NeurIPS.
[4] Sun, Yihao, et al. (2023). Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning. ICML.
[5] Rongjun Qin, et al. (2022). NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning. NeurIPS