Apologies for the confusion. We would like to supplement the related explanations about W2 and W3.
**W2: Justification of TD-reward surrogate**
We will use an example to illustrate the effectiveness of the TD-error approximation. Let's assume there are three types of transitions: $(s,a,r_1,s_1),(s,a,r_2,s_2),(s,a,r_3,s_3)$, with the proportions $p$, $1-p$ and $0$ in the dataset, respectively.
For the out-of-distribution transition $(s,a,r_3,s_3)$, it is expected to associate with high uncertainty and a large TD-error, according to previous research [1,2,3] and our empirical results in Figure 14.
For the in-distribution transitions $(s,a,r_1,s_1),(s,a,r_2,s_2)$, the Q-value can be expressed as $Q(s,a)=p\cdot (r_1+V(s_1))+(1-p)\cdot(r_2+V(s_2))$. Thus, $(s,a,r_1,s_1)$ corresponds to a low TD-error $||Q(s,a)-r_1-V(s_1)||^2_2$ when it has a high probability of occurring in the dataset (i.e., when $p$ is close to $1$).
Therefore, we expect to obtain $TD(s,a,r_1,s_1)<TD(s,a,r_2,s_2)<TD(s,a,r_3,s_3)$ when $p$ is high. In this case, TD reward aligns with the estimated dynamics $\hat P(s',r'|s,a)$ in the posterior for in-distribution transitions, where a higher occurrence probability leads to a higher TD reward. For out-of-distribution transitions, while the TD reward might not align with the estimated dynamics, it remains low to mitigate the adverse effects of inaccuracies in dynamics estimation. Similarly, for state distribution, a higher occurrence probability of states corresponds to a more accurate value function estimate and a higher TD reward, whereas out-of-distribution states typically result in a smaller TD reward.
I hope this example provides an intuitive insight into this approximation. Rigorous theoretical derivations would involve the generalization of Q-value neural networks and their relationship to dynamics uncertainty, which are relatively complex and challenging. We would leave further theoretical analysis to our future work. Another potential direction is to adopt alternative approaches, such as explicit dynamic modeling, which could offer more straightforward theoretical results.
[1] Bai, Chenjia, et al. "Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning." ICLR 2021.
[2] Rigter, Marc, Bruno Lacerda, and Nick Hawes. "One risk to rule them all: A risk-sensitive perspective on model-based offline reinforcement learning." NIPS 2024.
[3] Ghosh, D., Ajay, A., Agrawal, P., & Levine, S.. Offline rl policies should be trained to be adaptive. ICML 2022.
**W3: Conservative parameter $\alpha$**
Our statement that "a roughly designed $\alpha$ will not significantly undermine our approach" was based on comparisons with the baselines, where our approach showed relatively fewer violations and a higher correlation between cost and reward (indicating the high policy diversity we expected) than baselines under various $\alpha$. We know this isn't a direct comparation shown in Figure 6 and apologize for any confusion caused by some of our claims in the response.
We acknowledge that the influence of this parameter on the final performance cannot be overlooked, and we appreciate the reviewer for highlighting this. We believe that fine-tuning this parameter using a limited iteration of online interaction and re-adaptation could indeed address this issue. Additionally, the tasks used in the ablation experiments were randomly selected, and we plan to conduct these experiments across a broader set of tasks and report the results in a future version.
Thank you again for the constructive feedback. We are happy to provide further explanations if there are still any confusions.