Unrolled computation graphs are prevalent throughout machine learning but present challenges to automatic differentiation (AD) gradient estimation methods when their loss functions exhibit extreme local sensitivtiy, discontinuity, or blackbox characteristics. In such scenarios, online evolution strategies methods are a more capable alternative, while being more parallelizable than vanilla evolution strategies (ES) by interleaving partial unrolls and gradient updates. In this work, we propose a general class of unbiased online evolution strategies methods. We analytically and empirically characterize the variance of this class of gradient estimators and identify the one with the least variance, which we term Noise-Reuse Evolution Strategies (NRES). Experimentally, we show NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of unroll steps across a variety of applications, including learning dynamical systems, meta-training learned optimizers, and reinforcement learning.
Paper
Similar papers
Peer review
Summary
The study considers numerically estimating gradients for online (reinforcement) learning. The parameters are perturbed and from the performance of the perturbed models the gradient is estimated. In an online setting, the proposed method decouples the number of steps between gradient estimates (which are then used by a first order gradient based optimiser) and the number of steps between changing the perturbation. Two aspects are relevant in the paper: The variance of the gradient estimate and the ability to parallelise the methods.
Strengths
I could not find anything that I would call wrong in the manuscript.
Weaknesses
Allmost all research is to a certain degree incremental. The main problem of the research presented in the paper that is that the increment is rather small, it lacks originality and novel „surprising“ insights. The problem setting is not new. The proposed algorithm, as properly stated in the manuscript, can be viewed as a slight generalisation of „Persistent evolution strategies“ [13]. The canonical way to apply a Monte Carlo method such as ES to non-episodic RL tasks is to split the state-action-reward sequence into „pseudo-episodes“ . This is also done/suggested in other other non-ES RL algorithms, e..g., in Proximal Policy Optimization. This is also sometimes referred to as truncation. The problem of how many (pseudo-)episodes are needed to evaluate a perturbation to get a reliable signal to be exploited by ES for RL has also been studied, for example see Heidrich-Meisner and Igel. Hoeffding and Bernstein Races for Selecting Policies in Evolutionary Direct Policy Search. ICML 2009 Two aspects are relevant in the paper: The variance of the gradient estimate and the ability to parallelise the methods. The first one leads to a non-surprising result, which was simultaneously discovered in another work, properly cited by the authors: [19]. In my evaluation, the patalllization, while being useful in practice, is rather technicality. The discussion the (parallel) runtime is partly not convincing. I understand that 1. FullES takes O(T) time, it sees the states from s^+_0 to s^+_T (and s^-_0 to s^-_T) 2. calling Algorithm 2 takes O(W) time, it sees the states s^+_tau to s^+_tau+W (and s^-_tau to s^-_tau+W) 3. T/W independent calls of Algorithm 2 can be fully parallelised, so on T/W nodes taking the average over T/W independent calls starting from the same state s+ / s- also takes O(W) time (Please correct me if I am wrong). So I think „Under perfect parallelization, the entire NRES gradient estimation would require O(W) time to complete. In contrast, the single FullES gradient estimate has to traverse the UCG from start to finish, thus requiring O(T) time. Hence, NRES is T /W times more parallelizable than FullES with the same compute budget.“ is a bit misleading. Because a single FullES gradient estimate sees all steps from 0 to T, while Algorithm 2 looks at several „parallel“ trajectories starting from the same state running for W steps. I think this is qualitatively different. In the RL setting, you cannot in general not replace information collected over a time period with information collected from an ensemble where the ensemble members start from the same state. Think of an RL task where the rewards are always zero in the first 0.99*T steps. The „Instead, we aim to compare the pure performance of different ES methods assuming perfect parallel implementations. To do this, we measure a method’s performance as a function of the number of sequential environment steps it used. Sequential environment steps are steps that have to happen one after another (e.g., the environment steps within the same truncation window). However, steps that are parallelizable don’t count additionally in the sequential steps.“ This means that global random search, which evaluates 10^50 independent random policies has only a cost of one? This would then be the ultimate baseline method, outperforming all proposed algorithms. The manuscript lacks clarity. For example, some statements about „chaotic loss surfaces“ are confusing. Being chaotic is a well defined mathematical property. It starts to make sense in the experiments. The Lorenz system is chaotic, which inmates the first place refers to perturbations of the system state. The experiments always start from the same initial state. Optimised are the coefficients of the Lorenz system. What is the exact definition of a „chaotic loss surfaces“? Where is the proof that the loss surface is really chaotic when changing r and a (citation missing). A comparison with an evolution strategy that uses the perturbations also for the update, such as (1+1) CMA-ES with restricted covariance matrix) would have been interesting. Hyperparameter tuning seems to be crucial. The description in the appendix is appreciated. The Lorenz training suffers from instabilities. There are countermeasures, like gradient clipping. One could also move to an optimiser that decouple gradient magnitude and update step length, for example Adam. Adam is used later, but for the Lorenz experiments the manuscript sticks to SGD. How are the results with Adam (and corresponding learning rate tuning)? This may be important because the experiments lead to to strong statements „AD methods perform worse than ES methods“ (Appendix) and one has to rule out that the problem is not the optimiser but really the AD as such. Minor: * Instead of talking about a trick and adding a reference, simply state that you use \nabla_theta E_x~p(x|theta) [ f(x) ] = E_x~p [ f(x) \nabla_theta \ln p(x|theta), that is clearer (referring to this as a trick makes me cringe). * The embedding in the existing literature is weak. One may not need no go back to Rechenberg and Schwefel when talking about evolution strategies (although it does not harm), but citing [12] as the only source for evolution strategies for RL is in my evaluation not OK. * The first epsion in line 73should be in bold font. * The hyperparameters are relevant, the main paper should point out to the appendix where the hyperparameters selection is discussed more clearly.
Questions
* What is the exact definition of a „chaotic loss surfaces“? Where is the proof that the Lorenz „loss surface“ is really chaotic when changing r and a? * Under the used „sequential environment steps“ measure, would a global random search algorithm that which evaluates 10^50 independent random policies only have a cost of one? Would this be the ultimate baseline method, outperforming all proposed algorithms?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
OK
Summary
This paper generalizes PES based on noise-reuse, generating a more general class of unbiased online ES gradient estimators. The authors analytically characterize the variance of the estimators and identify the lowest-variance estimator named Noise-Reuse Evolution Strategies (NRES). Experiments on learning dynamical systems, meta-training learned optimizers, and reinforcement learning show that NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of unroll steps.
Strengths
1. The paper proposes a general framework GPES by generalizing the existing PES algorithm based on noise-reuse. 2. The paper gives a theoretical analysis of unbiasedness and variance for GPES, and identifies the lowest-variance gradient estimator under this setting named NRES, which always reuses noise. 3. The paper proves that under some reasonable conditions, NRES has lower variance than FullES.
Weaknesses
1. The proposed GPES is a simple and direct generalization of the existing PES. The novelty is not very strong. 2. The experiments need to be improved, e.g., adding the empirical comparison on longer sequences and higher dimension problems. 3. Why GPES can be better than PES? An intuitive explanation is needed. Minor issue: Step-unlocked is used without explanation.
Questions
1. In Theorem 8, it's said that the required assumption often holds in real-world applications. Can you give more explanation and evidences? 2. In the experiments, the x-axis of figures is measured by wall-clock time. Why not use the number of iterations?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Yes
Summary
This work proposes a method for optimizing unrolled computation graphs (e.g. recurrent networks, etc.). When using ES to optimize a computation graph, the graph must be fully rolled out. Recent methods (PES) have examined using a truncated window to optimize, so that optimization can occur without a full unroll. PES aims to unbias the estimator by accumulating truncation noise during an online rollout, however, it produces high variance. This paper presents a generalization to PES, GPES, that decouples the frequency of noise injection and gradient estimation. A specific instantiation, named NRES, samples the noise only once per episode. This reduces variance as it removes the need for noise accumulation. Experiments are presented on a two-parameter chaotic Lorenz system, where NRES outperforms previous work. An RNN-based meta-learning setup is also evaluated wherein the transition function defines an update over inner parameters, along with a reinforcement learning setup to solve Mujoco tasks.
Strengths
This paper provides an extremely thorough investigation of their proposed framework. The framework includes a generalization of previous work, along with a theoretically-justified instantiation with desirable properties on unbiased estimation and low variance. This is an original idea that is promising. The quality and clarity of the writing is excellent. The work sets up for additional work in investigating the properties of GPES methods, including variations on the proposed NRES.
Weaknesses
The final experiments on meta-learning and reinforcement learning are lacking detail. A more precise experimental setup would strengthen the argument of the paper. In addition, comparisons to non-ES based methods would give more context on how such methods compare to other solutions in the field. The experiments present wall-clock time as the unified axis, but discussion in the paper mentions total computation budget -- a figuring comparing the methods using some consistent measure of computation would strengthen the paper.
Questions
How does NRES compare to FullES in terms of pure computational budget? What are the scaling patterns of additional parallelization? How do non-ES methods perform on the Mujoco tasks? Additionally, ablations on hyperparameters such as the noise variance and # of workers would grant insight into the nuance of the method.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
Limitations are discussed in section 7 -- they largely relate to ES in general.
Summary
In this paper, the author(s) extended the well-known Persistent Evolution Strategies via noise-reuse and proposed an improved version with reduced variance. The main contribution of this paper is to provide detailed mathematical proof to validate their claim.
Strengths
Considering this significant contribution to the evolution strategies and other related ML communities, I personally suggest accepting this high-quality research paper. In this paper, the author(s) also discussed the main limits of their method. These discussions about the possible limitations (hysteresis and complexity) are highly encouraged for academic research because this can better reflect the whole state of their method. Furthermore, the author(s) pointed out that “whether there are better ways to leverage the sequential structure” beyond the isotropic Gaussian distribution. I agree that this is an interesting open question (worth to be investigated).
Weaknesses
Some notes are given in order to further improve this paper: For Section 5.3 Reinforcement Learning, the removal of “additional heuristic tricks” can help us focus on the underlying mechanism, which is very critical for algorithmic understanding, no matter from a practical or theoretical perspective. We often prefer general-purpose design principles, though these “additional heuristic tricks” sometimes work well in some cases. Given that only the linear policy was used in experiments, it is highly expected to include also the more challenging non-linear policy with higher dimensions.
Questions
see above
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
No comments here.
Summary
This paper studies online evaluation strategies for unrolled computation graphs. Especially, the authors 1). propose a general class of unbiased online evolution strategies that generalizes Persistent Evolution Strategies (PES), named Generalized Persistent Evolution Strategies (GPES). The key idea is to share noise across truncation windows instead of sampling every round. 2). characterize the variance and variance reduction properties of their strategies. 3). study a special case of the general class strategies, named Noise Reuse Evolution Strategie (NRES), show the variance advantage of NRES over other estimators. 4). experimentally show the advantages of NRES across a variety of applications, including learning dynamical systems, meta-training learned optimizers, and reinforcement learning.
Strengths
- The idea of reusing noise to reduce variance is interesting and a bit counterintuitive. For first order method, reusing noise always lead to a larger variance (If the noise is reused, averaging gradients cannot reduce the variance of the noise). However, it seems for evolution strategy, such a simple method could significantly reduce variance and improve the performance. - Experimental results are comparable to the previous methods and show a remarkable improvement. - The paper is well writen. The proof in the appendix is well organized and seems correct.
Weaknesses
- No comparision between NRES and first order method. - The authors did not explain the some parameter setting in the experiments. Especially, in Figure 5 and 6, why choose a different N when comparing online ES and FullES? - For online ES, the algorithm update $\theta$ every truncation window, thus the loss will never be in the form of (7). In this case, it seems NRES is still biased.
Questions
Given a time $T$, FullES returns one gradient g=[L([\theta+\epsilon]_{\times T})-L([\theta-\epsilon]_{\times T})]{\epsilon}/{2\sigma^2}, and NRES (Algorithm 2) returns $T/W$ gradients $g_1,\dots, g_{T/W}$, where g_k=[\sum_{t=Wk+1}^{Wk+W}L_t([\theta+\epsilon]_{\times t})-L([\theta-\epsilon]_{\times t})]{\epsilon}/{2\sigma^2 W}. By definition, there is $g=W/T \sum_{k=1}^{T/W} g_k$. In this case, if we choose learning rate $\eta_{FullES} = \eta_{NRES}T/W$, there should be no difference between FullES and NRES. So why is there such a big improvement in the experiments? One minor typo: Algorithm 6 line 10: not $L_t^s$, should be $L_{self_.\tau +1}^s$.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Theoretical work. No limitations.
Acknowledgement of Rebuttal
Thank you for the detailed response and the additional experimental results. It would be great to include these results in a revised version of the paper as well, for future reference. Given that the score already indicates accept, I will maintain this score.
Response to Reviewer's Acknowledgement
Thanks for your prompt reply! We will make sure to incorporate these added results in our revision. If you believe we have answered your questions to the degree that you'd feel comfortable raising your _confidence_ score, we would really appreciate it, but in any case we want to express our gratitude again for your positive feedback and suggestions to improve our work.
Coming back to my questions: Question 1: It is clear and well defined what a chaotic system is; it is clear what the Lorentz system is and for which values of its parameters it is a chaotic system. The chaotic behaviour refers to the there state variables (often denoted x, y, z) of the system. The objective/fitness function is a function of (some of the) the parameters (typically denoted by sigma, rho, and beta) of the Lorentz system. How the properties of the Lorentz system carries over to the objective function is not clear to me. Now, what defines a „chaotic loss surfaces“? Question 2: „Under the used „sequential environment steps“ measure, would a global random search algorithm that which evaluates 10^50 independent random policies only have a cost of one?“ So, the answer to this question is yes?
Further Explanation to the Reviewer's Questions
Thanks for your reply. We believe our rebuttal has answered these questions (see paragraphs on “__What does chaotic mean?__” and “__Measuring the number of sequential steps of global random search__”). However we discuss these questions in more detail below in case our previous response was unclear. __Answer to Question 1__: The reviewer asks how the chaotic property of the Lorenz system carries over to the objective function. We provide a detailed explanation in the paragraph below. However, before doing so, we note that this question is not the focus of our work: the aspect that is relevant to our experiments is that this dynamical system parameter learning problem indeed has _a loss objective function that exhibits extreme sensitivity (oscillations) with respect to small changes in its input learnable parameters_ (__this is what we refer to as a chaotic loss surface__). This terminology is consistent with its usage in prior works on online evolution strategies methods [1,2], and we have empirically shown the Lorenz loss objective function is chaotic as it has high degrees of fluctuation when small changes occur in the parameter space through Figure 4(a) and Figure 7. In terms of why the chaotic property of the Lorenz system carries over to the objective function, we summarize the discussion in [3], which has explained this relationship. Since the Lorenz system is a chaotic dynamical system, the state variable at a later time $s_t$ is highly sensitive to changes in the state variable at a prior time $s_i$, $i < t$. This means that the Jacobian matrix between the two states ($\frac{d s_t}{d s_i}$) has some large singular values. Because the total derivative of the loss function $L_{\textrm{avg}}$ with respect to the Lorenz system parameter $\theta$ is related to the Jacobian matrix through the following relationship (equation (7) and (8) in [3]): $\frac{dL_{\textrm{avg}}}{d\theta} = \frac{1}{T} \sum_{t=1}^T [\frac{\partial L_t}{\partial \theta} + \sum_{i=1}^t \frac{\partial L_t}{\partial s_t} {\color{blue} \frac{d s_t}{d s_i}} \frac{\partial s_i}{\partial \theta} ],$ the magnitude of the loss gradient $\frac{dL_{\textrm{avg}}}{d\theta}$ will also be large due to the existence of the large singular values in the Jacobian matrices $\\{{\color{blue} \frac{d s_t}{d s_i}}\\}$. Having these high magnitude gradients implies that the loss values would have extreme sensitivity to small changes in the system parameters ($\theta$), thus resulting in the chaotic behavior of the loss function. [1] Metz, L., Maheswaranathan, N., Nixon, J., Freeman, D., & Sohl-Dickstein, J. Understanding and correcting pathologies in the training of learned optimizers. ICML, 2019. [2] Vicol, P., Metz, L., & Sohl-Dickstein, J. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. ICML, 2021. [3] Metz, L., Freeman, C. D., Schoenholz, S. S., & Kachman, T. Gradients are not all you need. arXiv:2111.05803, 2021. __Answer to Question 2__: The answer to this question is “no”, as the cost would be significantly larger for global random search in our empirical setup. To see this, we note that we have constrained all the methods in our experiments to use the same number of environment steps per gradient update (line 289) for fair comparison. If the reviewer insists on a comparison to global random search (which is not a gradient-based local update method, as we focus on in this work) in our experiment setup, to appropriately account for costs we should similarly consider __the cost of global random search as an iterative update method__: global random search keeps track of the best $\theta_*$ (with the lowest loss value) seen so far, and for each update, it randomly samples a number of parameters $\\{{\theta_i}\\}\_{i=1}^N$, evaluates their individual loss in parallel, and updates $\theta_*$ with the best $\theta_i$ if it has an even lower loss. From this perspective, we need to constrain global random search to also __use the same number of environment steps per update__ just as we have constrained the ES methods. For example, on the Swimmer task, because we make both FullES and NRES use only 6000 environment steps (a total of 6 episodes) per update, global random search will also be allowed to evaluate only $6$ different randomly drawn $\theta_i$ (instead of $10^{50}$) in parallel for an update (which would cost an episode-length ($T=1000$) number of sequential unrolls). Thus to evaluate $10^{50}$ episodes in total, global random search would require $10^{50} / 6 \approx 1.67 \times 10^{49}$ updates and a total of about $1.67 \times 10^{52}$ sequential steps. Global random search would therefore be expected to perform quite poorly against the methods considered in our paper. Thank you for engaging with us during the rebuttal period. Please let us know if you have any further questions, and in particular if there are any other points from your initial review that have not yet been addressed.
Question 2: If I am not overlooking something, either the paper is wrong or the rebuttal answer. The paper states: „To do this, we measure a method’s performance as a function of the number of sequential environment steps it used. Sequential environment steps are steps that have to happen one after another (e.g., the environment steps within the same truncation window). However, steps that are parallelizable don’t count additionally in the sequential steps.“ Global random search can evaluate random search points in parallel - the do not depend on each other. Thus, a global random search algorithm that evaluates N independent random policies/search points only has a constant cost independent N of when under the suggested performance measure as defined in the submission (e.g., the answer to my original question should briefly be "yes"). This is not just nitpicking. The performance measure is important for the evaluation of the different approaches. The performance measure was picked to stress a particular (and important) property of the proposed algorithm - which makes it look better compared to other methods that do not have this property. But if „sequential environment steps“ as defined in the manuscript is the relevant measure, then trivial methods seem perform at least on par with the suggest algorithm. I guess you need to introduce an additional constraint on the number of environmental steps per iteration. This would make a hyperparameter in the performance evaluation explicit - and methods will rank differently based on the choice. Question 1: The chaotic behaviour is stressed in the manuscript (the word occurs three times on the first page), therefore my question. The characterisation „a loss objective function that exhibits extreme sensitivity with respect to small changes in its input learnable parameter“ makes sense, but is not necessarily sufficient for the underlying system being chaotic in the strict mathematical sense. While I would suggest a different wording, I am happy with the explanation you gave. (In your explanation in the reply, note that the Lorentz system is only chaotic for certain choices of parameters, in particular the „default“ parameters that are the optimum of the considered optimization task. This means that during optimization the behaviour of the underlying dynamical system need not be chaotic during the whole search process - but it will be around the optimum.)
Thanks very much for your prompt response. We follow-up below: __Question 1__: Thank you for acknowledging our explanation. We originally chose the term “chaotic” following its usage in related works (TES, PES), but we agree with the reviewer that the word has different meanings depending on the context, and so we will further clarify its usage to describe loss functions in unrolled computation graph in our revision. We also agree that the underlying dynamical system doesn’t need to be chaotic to have a chaotic loss function on its induced unrolled computation graphs (nor do we claim this in the paper). For example, we do not claim that the dynamical system in the learned optimization application (where the state variables are the inner model parameters) is a chaotic dynamical system. However, as we have shown through Figure 1(a), the loss surface of this learned optimization task has extreme sensitivity to small changes in its learned optimizer’s parameters, and thus we refer to this loss surface as chaotic. __Question 2__: Similar to what the reviewer mentioned, _we have indeed introduced an additional constraint on the number of environmental steps per update_ when we measure the number of sequential steps for all the ES methods. __Thus there is nothing wrong or inconsistent with our paper/rebuttal response.__ For example, as we have explained on Line 289, FullES and NRES use the same number of steps ($6000$) on the Swimmer task. In our revision we will discuss this constraint earlier (Line 279-282), when we introduce the concept of _sequential steps_, to make the constraint more clear. The reviewer claims that under different values of this constraint, the methods will rank differently. Here we wish to make a few clarifications: 1. __How would global random search perform under different values of the constraint?__. It is true that when the limit approaches infinity (e.g. $\ge 10^{50}$ episodes per update), global random search can dominate any iterative update method that only explores locally (including the methods we consider but also other methods like CMA-ES) by using only one update and thus $T$ sequential steps. However, under practical values of this constraint (e.g., $\le 10^{3}$ episodes per update), _global random search is not a favorable method compared to the ES methods we have considered as it suffers from the curse of dimensionality_ — the total number of evaluations needs to be __exponential__ in the search space dimension $d$ to thoroughly evaluate the space, thus requiring a significantly larger number of updates and sequential steps than the ES methods need to solve the tasks. We are unaware of prior work that uses global random search for the tasks we have considered likely due to this issue. 2. __How do the methods we focus on compare under different values of the constraint?__. For the ES methods we consider in our paper (which do not include global random search), our proposed method _NRES would still outperform the other three methods_ __regardless of the size of this constraint__. This is because: - NRES outperforms FullES because it uses $T/W$ times fewer sequential steps per update under any fixed constraint value due to its extra ability to parallelize over the time-step dimension (line 191-195). - NRES outperforms PES because NRES gradient estimates will always have a lower variance (line 171-184) than PES estimates whenever both methods use the same number of workers, which would be true under the same number of environment steps per update. - NRES outperforms TES because TES’s estimated gradients suffer from truncation bias (line 94-96), making it unable to converge to an optimal solution regardless of the number of steps used per update. Finally, we note that we have picked the current constraint values for each experiment (e.g. 6000 steps per update on Swimmer) by tuning on the method FullES (but not on our proposed method NRES). Hence this shouldn’t give NRES an unfair advantage in our experiments. __Summary__: Thanks again for engaging with us in this discussion and for your questions and feedback. To summarize: * We will make the definition “chaotic loss surface” more explicit in the paper when we describe __Loss properties__ in line 58. * We will highlight earlier on that when comparing iterative update methods in terms of sequential steps, we constrain the number of steps used per update by __adding the following sentences after line 282__: “As all the methods we compare are iterative update methods, we additionally require that each method use the same number of environment steps per update when measuring each method’s required number of sequential steps to solve a task.” * We emphasize that the _conclusion that NRES offers improvements over the other ES methods considered in our paper would still hold regardless of the number of steps allowed per update_. We will add the discussion/explanation provided above on this in our revision.
Thank you for the detailed response. It seems that the biggest improvement of NRES over previous works is the parallelization ability. Given the score indicates accept, I meantain this score.
Thanks very much for your reply and your recommendation to accept the paper. Following the reviewer’s remark about the reason for the improvement of NRES, we briefly summarize the main benefits of NRES over prior works (also illustrated in the table in Figure 1(b)). NRES improves over prior ES methods for a number of reasons: 1. As the reviewer mentions, _NRES improves over FullES_ because NRES is an online method unlike the offline algorithm FullES and it thus has much __better parallelization__ ability than FullES. The gradient estimate from NRES can also have lower variance than FullES under the same total compute. 2. _NRES improves over the online method TES_ because TES suffers from truncation bias whereas NRES is __unbiased__. 3. _NRES improves over the online method PES_ because NRES has a significantly __lower variance__ than PES due to its noise-reuse property. These properties together make NRES a particularly compelling approach relative to prior ES methods.
Follow up hYKW
Dear Reviewer hYKW, We would appreciate if you would you be so kind as to acknowledge and respond to the authors' rebuttal. This is crucial to ensure the reviewing process is conducted adequately. AC
Decision
Accept (poster)