Summary
The paper proposed a new approach to Offine-RL Online Fine-Tuning called calibrated Q-learning (Cal-QL). This approach aims to learn an effective initialization from offline data that enables fast online fine-tuning capabilities. Cal-QL accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data while ensuring that the learned Q-values are at a reasonable scale. The contributions of this paper include: 1) The paper theoretically analyzes the cumulative regret obtained by online fine-tuning, when the value function pre-training with Cal-QL. 2) Empirically, Cal-QL outperforms previous methods on 9/11 fine-tuning benchmark tasks.
Strengths
1. Writing is clear and understandable.
2. The paper proposes a new definition (calibration) and gives the theoretical analysis of Cal-QL.
3. The method can be implemented on top of conservative Q-learning (CQL) for offline RL with few code changes.
4. The Cal-QL is better than the baseline methods in most tasks.
Weaknesses
1. The assumption of Cal-QL seems a bit too strong, that Cal-QL will be particularly effective in controlling the efficiency of online fine-tuning only when the reference policy μ is close to the narrow expert policy.
2. Cal-QL is also most related to methods that utilize a pessimistic RL algorithm for offline training but incorporate exploration in fine-tuning [1,2]. However, these two important baseline methods are missing from the experimental comparison results (Figure 6, Table 1 and 2).
3. The coefficient (\alpha) of the conservative term (in equation 3.1) is an important parameter that directly affects the experimental results. However, the hyperparameter (\alpha) ablation experiments are missing.
Questions
1. How many random seeds are used for the experimental results of Table 1 and Table 2? Same as Figure 6, six random seeds? Moreover, how many random seeds are used in Figure 7 and 8?
2. In Appendix Table 3, for the visual-manipulation task, the paper utilized separate values for offline (\alpha = 5) and online (\alpha = 0.5) phases for the final results. Is the proposed algorithm sensitive to hyperparameters (\alpha and mixing ratio)? To what extent do the final experimental results of different tasks depend on the hyperparameters?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
1. It would be better if there are more comparative experiments with important baseline methods [1,2].
2. To test the sensitivity of the method to hyperparameters, there should be more experimental comparisons and analyses of hyperparameters \alpha and mixing ratio.
References:
[1] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702–1712. PMLR, 2022.
[2] J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning. arXiv preprint arXiv:2202.06239, 2022.