Response to Reviewer epDC
We appreciate the reviewer's valuable feedback. Below are our responses to the raised questions:
**Weakness 1: Eq. (4.1) and (4.2) do not point to the differences between DP and DR. Instead it’s the sentences that follow these equations that describe these differences.**
We will make the following modifications to enhance the clarity:\
Firstly, we'll introduce the unified form of the RP gradient at the beginning of Sec. 4.2:
$\hat{\nabla}\_\theta J(\pi\_{\theta}) = \frac{1}{N}\sum\_{n=1}^N \hat{\nabla}\_\theta V^{\pi\_\theta}(\hat{s}\_{0,n}) = \frac{1}{N}\sum\_{n=1}^N \nabla\_\theta(\sum\_{i=0}^{h-1} \gamma^i r(\hat{s}\_{i,n}, \hat{a}\_{i,n}) +\gamma^h \hat{Q}\_\omega(\hat{s}\_{h,n}, \hat{a}\_{h,n})),$
where $\hat{s}\_{0,n}\sim\mu\_{\pi\_\theta}$, $\hat{a}\_{i,n}=\pi\_\theta(\hat{s}\_{i,n}, \varsigma\_{i,n})$, and $\hat{s}\_{i+1,n} = \hat{f}\_\psi(\hat{s}\_{i,n}, \hat{a}\_{i,n}, \xi\_{i,n})$.\
Next, we'll describe how the DP and DR gradients are estimated by defining the corresponding noises $\varsigma\_{i,n}$ and $\xi\_{i,n}$. We will: (i) substitute Eq. (4.1) with $\varsigma\_{i,n}\sim p(\varsigma), \xi\_{i,n}\sim p(\xi)$ to depict RP-DP, and (ii) describe RP-DR by replacing Eq. (4.2) with " $\varsigma\_{i,n}$ and $\xi\_{i,n}$ are inferred by solving $a\_{i,n} = \pi\_\theta(s\_{i,n}, \varsigma\_{i,n}), s\_{i+1,n} = \hat{f}\_\psi(s\_{i}, a\_{i,n}, \xi\_{i,n})$ to parameterize the real data sample $(s\_{i,n}, a\_{i,n}, s\_{i+1,n})$". We'll also include further specifications: "Take 1d Gaussian models $\hat{s}\_{i+1}\sim\mathcal{N}(\phi(\hat{s}\_{i}, \hat{a}\_{i}), \sigma^2)$ as an example, where the mean is the output of a function $\phi$. Then $\xi\_{i,n} = (s\_{i+1,n} - \phi(s\_{i,n}, a\_{i,n})) / \sigma$".
**Weakness 3: Other ways of inducing transition randomness (e.g., adding randomness to the applied action so that it goes through nonlinearity) or other RL experiments. Besides, I recommend the authors to include the discussion about these stochastic results in the main paper.**
- We'd like to emphasize that the results in this paper are applicable to both stochastic and deterministic transitions. By ''adding randomness to the applied action'', does the reviewer mean stochastic policies such as Gaussian policies employed in our experiments? In these policies, Gaussian noises are added to the action means. Using stochastic rather than deterministic policies introduces randomness into the state transitions in a nonlinear way even for deterministic transitions.
- We appreciate the reviewer's suggestion, and we will incorporate the additional stochastic results with discussions into the main paper.
**Question 3.1: The reward confidence intervals of $h=8$ might be intersecting with other smaller expansion steps, so it’s not clear to me whether it is indeed better.**
- The primary goal of Fig. 2-6 is to demonstrate that applying SN can alleviate the exploding variance issue, enabling longer model unrolls without significant optimization difficulties. *However, a larger $h$ is not necessarily better*. As shown by Proposition 5.7, there is a tradeoff between model error and critic error that determines an optimal $h^*$. This result holds when SN is applied.
- The limited gains when increasing $h$ to $8$ align with this tradeoff. A large $h$ may benefit training when the critic is inaccurate. Meanwhile, increasing $h$ can be detrimental due to the compounding effect of model errors. Determining the optimal $h^*$ is challenging as it depends on the model and critic error.
- As shown in Appendix D.4, learning more accurate models (by incorporating directional derivative error) leads to further performance improvements. This serves as evidence supporting that the constrained improvement partly attributes to inaccurate models.
**Question 3.2: It is also important to consider the sample complexity needed.**
The reported returns were attained by employing an equal number of samples for various $h$ in each task (1e6 for walker2d and 4e5 for hopper). We didn't observe obvious distinctions in the needed samples for different $h$ when SN is applied.
**Limitations: I wonder whether the authors believe having a more accurate model would be inherently contradictory to SN.**
The primary objective of the SN method is to tackle the variance issue and the associated optimization challenges. By employing SN, we can navigate the tradeoff between model error and critic error, encouraging us to learn more accurate models and effectively utilize the model gradient information. This is especially true for non-stiff dynamics with small Lipschitz constants, in which case the models with SN still reside within the function class of the true dynamics. For stiff systems, a balance should be struck between the gradient variance and bias, where applying SN to an accurate (thus non-smooth) model reduces the variance with the cost of increased bias. For a more comprehensive discussion, please refer to Line 50-56 and 242-247.