Replying to Reviewer 6EPu
Thank you for your timely response and detailed questions/suggestions! We provide discussions on your concerns as follows.
Q1: **Comparison between SRPO and DARC style of reward augmentation**
A1: We concur with the idea that approximating the ratio of stationary state distributions can be more stable and converge quickly. The reason behind this is that in practice, DARA employs two distinct discriminators that estimates the ratio $\frac{q\left(\text {source} \mid \mathbf{s}, \mathbf{a}, \mathbf{s}'\right)}{q\left(\text{target} \mid \mathbf{s}, \mathbf{a}, \mathbf{s}'\right)}$ and $\frac{q(\text {source} \mid \mathbf{s}, \mathbf{a})}{q(\operatorname{target} \mid \mathbf{s}, \mathbf{a})}$ separately. The discriminators will receive $(s,a,s^\prime)$ and $(s,a)$ as input, respectively. In contrast, SRPO employs a single discriminator that estimates the ratio $\frac{p(\text{Real}|\textbf{s})}{p(\text{Fake}|\textbf{s})}$ . It solely requires the state $s$ as input. The additional inputs of action and next state can **increase the complexity of the discriminator in DARA and lead to a slower convergence**. Also, the joint distributions over $(s,a,s')$ and $(s,a)$, which appear in the denominator of the ratios, are more prone to have values close to zero. This can make the reward augmentation unstable and have high variance. Motivated by the intuition of similar optimal state distribution, SRPO focus on a more fundamental property in environments with dynamics shift and provides a simpler and more efficient approach of reward augmentation. This can explain why SRPO can outperform DARA in most of the offline RL tasks.
To empirically substantiate the aforementioned discussions, we select the task of Walker-medium-expert and show the loss of the different discriminators of SRPO and DARA on the 100th epoch of training. Notably, the discriminator in SRPO demonstrates a smaller loss. The complete loss curve will be added in the revision.
| Discriminator Input | $(s)$ (SRPO) | $(s, a)$ (DARA) | $\left(s, a, s^{\prime}\right)$ (DARA) |
| :--- | :---: | :---: | :---: |
| Loss on the 100th epoch | 0.84 | 1.03 | 1.12 |
Q2: **SRPO doesn't show notably better performance in the initial training steps**
A2: In online experiments, SRPO works as an add-on module to the CaDM algorithm. Apart from reward augmentation, all other modules and hyperparameters remain unchanged. Therefore, in the initial training steps the performance of SRPO+CaDM can be constrained by the exploration setup of the base algorithm CaDM, which is itself a value-based maximum-entropy reinforcement learning algorithm based on SAC. Since the training steps are the total interaction steps with all environments, in the first 2M steps the agent only interacts with each environment for 667k steps when there are 3 different dynamics, and for 400k steps with 5 different dynamics. The online policy can have high entropy and be highly stochastic with such few training steps. As a result, SRPO+CaDM may not show notably better performance in the initial training steps.
Q3: **The theoretical assumptions might not always hold for complex real world cases though it can coverage most of the common ones.**
A3: Potential failure cases will indeed be helpful for further exploration of the similar assumptions. We will add a failure example of maze environments with different obstacle layout, which violates the requirement of homomorphous MDPs (lines 223-256). We will also add an example task of stock market trading with very high transaction fees, which tend to have a high Lipstchitzs coefficient (lines 232-234).
We will add the aforementioned discussions in the revision of our paper. We sincerely appreciate the questions and suggestions you provide in the two responses. We believe that solving the questions and adding the relevant discussions will greatly enhance our paper.