Response to Additional Comments
Thank you for your valuable comments. We respond to each of your comments as follows.
**Q16**. (1) How does $\tilde{\pi}_{\text{IQL}}$ help to prove the upper bound for TRACER? (2) How do you equate $Z^\pi(s)$ with $D^\pi (s,a)$ in steps 24, 25, and 26 in Theorem A.3? (3) What is $D^\pi (s,a)$ as previously D is defined as D(s, a, r)?
- **A16**.
- For (1), as TRACER directly applies the weighted imitation learning technique from IQL to learn the policy, we can use $\tilde{\pi}_{\text{IQL}}$ as the policy learned by TRACER under data corruptions, akin to RIQL.
In Theorem A.3 of the Appendix, the major difference between TRACER and IQL/RIQL is that TRACER uses the action-value and value distributions rather than the action-value and value functions in IQL/RIQL. Therefore, we further prove an upper bound on the difference in value distributions of TRACER to show its robustness.
- For (2), we apologize for the missing reference in steps 24, 25, and 26 of Theorem A.3 (see Lemma 6.1 of [1]). We follow [1] to provide the detailed derivation below.
- For any $\pi'$ and $\pi$, we have
$$\begin{align}
Z^{\tilde{\pi}}(s) &= \sum_{t=0}^\infty \gamma^t \mathbb{E}\_{(S_t,A_t)\sim P,\tilde{\pi}} \left[R(S_t, A_t) | S_0 = s\right]\\\\
&= \sum_{t=0}^\infty \gamma^t \mathbb{E}_{(S_t,A_t)\sim P,\tilde{\pi}} \left[R(S_t, A_t)+Z^{\pi}(S_t)-Z^{\pi}(S_t) | S_0 = s\right]\\\\
&= \sum\_{t=0}^\infty \gamma^t \mathbb{E}\_{(S\_t,A\_t,S\_{t+1}) \sim P,\tilde{\pi}} \left[R(S\_t, A\_t)+\gamma Z^{\pi}(S\_{t+1})-Z^{\pi}(S\_t) | S\_0 = s\right] + Z^{\pi}(s)\\\\
&= Z^{\pi}(s)+\sum\_{t=0}^\infty \gamma^t \mathbb{E}\_{(S_t,A_t)\sim P,\tilde{\pi}} \left[D^{\pi}(S\_t,A\_t,R\_t)-Z^{\pi}(S\_t) | S\_0 = s\right]\\\\
&= Z^{\pi}(s)+ \frac{1}{1-\gamma} \mathbb{E}\_{(s,a)\sim d^{\tilde{\pi}},\tilde{\pi}} \left[D^{\pi}(s,a,r)-Z^{\pi}(s)\right].
\end{align}$$
Thus, we can derive the step 25 from the step 24.
- For (3), we apologize for any confusion caused by the notations. We will replace the notation $D^\pi (s,a)$ with $D^\pi (s,a,r)$ in Theorem A.3 of our revision.
[1] Approximately optimal approximate reinforcement learning.
**Q17**. (1) What is the value of $c$ we use in experiments? (2) What is the effect on performance for varying $c$?
- **A17**.
- For (1):
- For each experiment with a random seed under individual corruptions in Tables 2 and 3 of our main text, we randomly select $c\\% = 30\\%$ of transitions from the offline dataset. Within these selected transitions, we replace one element per transition with corrupted element.
- In Table 1 with simultaneous corruptions, we also apply $c\\% = 30\\%$ but extend the corruption process across all four elements of the offline dataset. Specifically, we randomly select $30\\%$ of transitions and corrupt one element in each selected transition. Then, we repeat this step four times until all elements are corrupted. Therefore, approximately $76.0\\%$ of data in the offline dataset is corrupted, calculated as $1 - (1 - c)^4$.
In Table G4 of our attached PDF of the global response, we evaluate TRACER using different $c \\%$, including $10\\%, 20\\%, 30\\%, 40\\%,$ and $50\\%$. These rates correspond to approximately $34.4\\%, 59.0\\%, 76.0\\%, 87.0\\%$, and $93.8\\%$ of the data being corrupted.
- For (2):
- Results in Table G4 show that while TRACER is robust to simultaneous corruptions and significantly outperforms RIQL, its performance depends on the extent of corrupted data it encounters, degrading with increased data corruptions.
We hope our responses adequately address your concerns. If you have further concerns, please let us know and we will continue actively responding to your comments, enhancing our submission. We would deeply appreciate it if you could raise your score based on these revisions.