Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control

Summary

This paper explores the concept of return landscapes in deep reinforcement learning agents for continuous control. The authors demonstrate that the mapping between policy parameters and return can exhibit high-frequency discontinuities, leading to variations in agent performance. They introduce the concept of noisy neighborhoods in the return landscape and show that different policies can have substantially different distributional profiles, even when they achieve similar average returns. The authors also investigate the impact of noisy neighborhoods on learning dynamics and propose a distribution-aware procedure to improve policy robustness.

Strengths

- The paper introduces a fresh perspective on the instability of deep reinforcement learning agents in continuous control by studying the return landscape. - The concept of noisy neighborhoods and the characterization of post-update return distributions provide valuable insights into the relationship between policy parameters and return. - The experiments and visualizations presented in the paper help illustrate the findings and support the claims made by the authors.

Weaknesses

This paper is well-written, and the authors thoroughly explain the key concepts and conclusions by employing extensive experiments and visualizations. However, I believe there are areas for improvement: 1. In Section 4.2, the authors state, "while Figure 5 demonstrates that we can smoothly interpolate between levels of stability along a single gradient direction, Figure 1 illustrates that many gradient directions are not so forgiving. As such, we posit that the stability of policies over the course of training can be improved simply by rejecting gradient updates that lead to policies with less favorable post-update return distributions." I find the introduction of Algorithm 1 to be somewhat abrupt, as it does not seem closely related to the conclusions drawn earlier in the paper. For instance, an important observation made is that "we can smoothly interpolate between levels of stability along a single gradient direction." In algorithm design, can we not only reject policy updates that create noisier neighborhoods but also leverage policy updates that result in more stable neighborhoods? 2. The paper designs a simple yet effective algorithm based on crucial conclusions derived from in-depth analysis. However, due to the algorithm's conciseness and clarity, it becomes difficult for readers to link it to other existing methodologies and place it appropriately within the field. Therefore, a more thorough comparison or discussion with related work is necessary. For example, in the related work section, the authors have mentioned that their work is related to studies based on rejection/backtracking strategies.

Questions

1. I am particularly interested in whether distributional RL algorithms exhibit the issues mentioned in the paper. Given that distributional RL methods take into account reward distribution during policy optimization (albeit with significant differences from the post-update reward distribution studied in this paper), can these methods produce smoother reward landscapes? 2. Can the authors analyze the limitations of Algorithm 1? In particular, are the main conclusions and Algorithm 1 still valid for policies updated using different methods (beyond PPO, SAC, and TD3)? 3. On average, how many times will Algorithm 1 reject an update during a single policy parameter update in actual operation? Will the efficiency of the algorithm be severely affected if the number of rejected updates becomes too high?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are not explicitly discussed by the paper. Some comments and questions regarding limitations of the work are discussed in the section above.

Summary

The paper investigates the mapping from policy parameters $\theta$ to returns $R(\theta)$ and the sensitivity of $R(\theta)$ to minor perturbations to $\theta$. The authors show that popular RL algorithms tend to traverse "noisy neighborhoods" in policy parameter space where the return is highly varied even for small changes to $\theta$. They find that policies with comparable average returns can have drastically different return distributions. Additionally, interpolating between policies from the same training run allows for transitioning from noisy to smooth return landscapes without degraded performance (the same is not true for policies from different runs). Finally, a practical algorithm is proposed which improves stability by rejecting gradient updates that lead to noisier return distributions.

Strengths

Overall, I thought the paper was enjoyable to read and appears fairly novel (to the best of my knowledge). The figures are great and in general the story of the paper flows very well; Figure 2 is fantastic. The landscape-inspired approach to deep RL is extremely interesting in my opinion. Finding novel ways to stabilize training by considering the local return landscape around the current set of policy parameters is highly relevant to scaling up and robustifying existing RL methods. I appreciate that the paper doesn't rely on overly technical jargon and, combined with the great illustrations, the paper is fairly easy to grasp without much math at all.

Weaknesses

Section 4.1 makes pretty strong claims about interpolating between policies from the same run and from different runs, but it is only really substantiated by a few (six) qualitative examples. I suspect there is some way to quantify the claims in this section across a larger population of policy pairs, which I think would significantly strengthen this section. The results in Figure 6 (and the description in Section 4.2) are a bit confusing to me. Is this just showing that the proposed algorithm succeeds in reducing LTP after 40 gradient steps? Why don't we compare the mean / variance of returns for complete training runs that do or don't use rejection? The 'x' markers show points reached by full training runs, so why can't we show that with/without the proposed approach? Or perhaps I'm misunderstanding, and Algorithm 1 is meant to by applied after a training run in order to robustify an already learned policy? I felt the experiments could be more comprehensive w.r.t environment diversity. Given that the main claims of the paper are substantiated only empirically, I think it is potentially a problem that the paper only considers a few DMC environments (ant, walker2d, halfcheetah, hopper) which all involve locomotion.

Questions

"For our analysis, we use 200 policies generated by different runs of TD3. From these we select pairs of policies with different post-update return distributions, as measured by their standard deviation or left-tail probability, but similar mean" (L214-216). I don't fully understand how pairs of policies were selected using these criteria. Were these handpicked? Is there a more systematic way of picking pairs of policies? Is it possible for us to consider more than (a handful+1) examples? What is the claimed contribution of Algorithm 1? Are the authors claiming that this update scheme should be used practically? Is this just an illustration due to other considerations (e.g. computational cost). I think you should probably mention distributional RL [1] as related work. In Figure 3, labeling the successful/failing trajectory either in the plot or in the caption would be nice. === I have read the rebuttal and bumped up my score slightly in light of the additional experimental results.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

No

Summary

In this study, the authors explore the underlying causes of the instability observed in the performance of deep RL policies. Their investigation revolves around examining the relationship between a policy and its corresponding return, with a focus on analyzing how minor variations in policy parameters impact the overall return. The authors posit that regions where even slight perturbations in policy parameters result in significant changes in return are susceptible to failures. To address this issue, they introduce an algorithm designed to navigate away from these volatile neighborhoods, thereby enhancing the robustness of the policy. Lastly, the authors provide a set of experimental results to demonstrate their findings and validate the efficacy of their optimization algorithm through another set of experimental results.

Strengths

Overall Strengths: 1. The organization and writing of this paper are of good quality. The authors have succeeded in presenting the content in a clear and easily understandable manner, which greatly enhances the readability of the paper. A notable aspect is the structure of the experiments sections (Section 3 and Section 4), where the authors first introduce the research questions, followed by a detailed explanation of the experimental setup, and finally present the results along with their corresponding conclusions. This approach significantly aids in the comprehension of the experiments, in such heavily experiments-based paper. 2. I personally find the high-level idea of studying why the performance of RL policies so unstable well-worth pursuing.

Weaknesses

1. This paper would benefit from providing additional explanation and motivation regarding the significance of investigating the mapping between policy parameters and return. A more thorough exploration of why this particular landscape is crucial to the focus of the work is needed. Expanding on the importance of understanding this relationship would enhance the reader's understanding and emphasize the relevance of the research. 2. The experimental setup is limited. I would personally be interested in how different algorithms compare with each other when trained on the same task as well as how a certain algorithm behaves when trained on different tasks.

Questions

1. What will the results look like among policies trained with different algorithms on the same task? 2. I have seen recent works studying how the mapping between the state-space and the policy look like and how this landscape evolves during training. How do those findings relate to the findings in this paper? (For reference, look at “Understanding the Evolution of Linear Regions in Deep Reinforcement Learning” from NeurIPS’22) 3. I personally find figure 2 hard to understand. But for now, I don’t have any particular suggestions to improve the readability of these. 4. It would be invaluable if the same experiments would be repeated for policies trained with behavior cloning. To first see how policies trained with behavior cloning on some task compare to policies trained with other RL algorithms on the same task. And second, see whether different runs of behavior cloning demonstrate non-noisy return landscapes.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Please refer to the previous sections.

Summary

This paper looks at visualizing the return landscape in typical continuous control environments of deep RL. The authors propose a statistic over the returns profile, namely the left tail probability, and try to present this as a suitable measure of policy quality in a series of experiments and visualizations. Finally, they propose a high-level rejection sampling process for the policy updates using the proposed statistic and argue that it can lead to similar-mean but less-variable return profiles in the trained policies.

Strengths

The idea of looking at return surfaces is not original (e.g., see [1], [2], and [3] among many others). However, trying to propose a remedy based upon such visualizations has been less common. In terms of quality, the paper tries to narrow down its focus to control and robotics applications, which is admirable. As for the clarity of the paper, too much theoretical complications did not hinder the point of the paper. Unfortunately, the results are less significant than expected, but the studied topic (i.e., better understanding of the return landscape for deep RL in control environments) is certainly of great importance and has been under-looked for too long. # References [1] Sullivan, R., Terry, J. K., Black, B., & Dickerson, J. P. Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments. ICML, 2022. [2] Bekci, Recep Yusuf, and Mehmet Gümüş. "Visualizing the Loss Landscape of Actor Critic Methods with Applications in Inventory Optimization." arXiv preprint arXiv:2009.02391 (2020). [3] Ilyas, A., Engstrom, L., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Madry, A. (2018). A closer look at deep policy gradients. ICLR 2020. [4] Hwangbo, Jemin, et al. "Learning agile and dynamic motor skills for legged robots." Science Robotics 4.26 (2019): eaau5872. [5] Saleh, E., Ghaffari, S., Bretl, T., & West, M. (2022). Truly Deterministic Policy Optimization. Advances in Neural Information Processing Systems, 35, 8469-8482.

Weaknesses

## Presentation and Writing First and for most, the paper lacks a clear set of contributions. I encourage the authors to dedicate the last paragraph of their introduction to list 3 or 4 main contributions, and commit to delivering them in the entire paper. This is a standard practice, and lacking this makes it extremely difficult to evaluate the merit of the work. Second of all, the paper lacks a main message and a proper effort to convince the reader about the validity and the importance of it. As is, the manuscript looks more like a technical report draft rather than a conference paper. In particular, note that most NeurIPS papers focus on driving the main message in the initial sections, and avoid detailing the methods/data until the experiments section. However, this paper lacks such a structure, and tries to narrate a story by jumping from one experiment to another, while losing a cohesive context and spending a lot of space on detailing experimental details. ## Rigor The paper is not driven by any amount or kind of theory. Instead, the authors choose or abuse vague terms and intuitions to describe a series of hypotheses. For example, the notion of "stability" of a policy gets thrown around a lot in the paper (e.g., Line 246) without any mathematical standing or definition. This is despite the fact that the stability of a policy is a well-defined topic of research, and has many measures such as the Lyaponuv stability criterion. I doubt the authors were targeting this original notion of stability, and this is just one example of a technical term being abused. Many other vague terms are consistently used throughout the paper, including, but not limited to, (1) the "quality" of a policy, (2) "noisy" neighborhoods, (3) "failure" of a policy or a trajectory, (4) "safer" behavior, and so on. ## Premise 1. Most deep Policy Gradient methods (e.g., PPO, TRPO, DDPG, TD3, SAC, etc.) optimize the mean of the discounted return in the MDP framework. Other statistics of the return distribution (e.g., the variance, the mode, left tail probability, etc.) are irrelevant to the optimization objective. When evaluating methods and policies, it is standard that higher mean payoffs are preferable even if the return distribution has higher variance. If a particular application requires consistent returns from a policy, the user either has to (1) engineer the reward so that such variations lead to worse mean returns, or (2) use PG methods that optimize a different objective (e.g., the worst-case performance with min-max optimization methods). Both approaches are commonly used in control and robotics applications. Of course, consistent policy performance is preferable but it shouldn't be confused with the main performance metric. The entire paper is written on the premise that variations in the return are inherently undesirable, and I don't necessarily agree with that. In my experience, Deep RL algorithms such as PPO, SAC, and TD3 gravitate towards such "noisy" regions at the end of the training because these regions can potentially yield higher returns than the stable areas, even if the return improvement is small. When evaluating such PG methods, only the mean return should be considered. The only way the paper's findings could be relevant is if high-variations in the return profile were shown to correlate with lower mean-returns at the end of training. I'm afraid the paper is not showing that. 2. Deep PG methods do not directly optimize deterministic policies. Each algorithm, even the deterministic ones such as DDPG and TD3, have an exploration parameter $\sigma$. These parameters cause a filtering/convolutional effect on the return landscape, and smoothen the effective optimization objective. The paper mainly focuses on oscillations in the deterministic landscape, whereas the effectively optimized landscape can be much smoother and I'm not sure the same findings are applicable to the filtered landscape. ## Arguments The experiments and figures in the paper aren't always convincing or helpful. In particular, 1. Figure 5 doesn't prove a meaningful hypothesis. Such parameter interpolation behavior may not even be specific to RL problems, and "non-noisy" problems can also exhibit the same behavior. In other words, interpolating parameters within the same run in a deep classification task can lead to reasonably-performing parameters while inter-run interpolations wouldn't perform well. I'm not sure how this observation is a solid evidence for the existence of special patterns or connections, even if there were many gradient steps in between interpolated parameters. 2. Figure 1 consumes a lot of space, but carries little information. 3. Also, the paper spends a lot of time speculating. For instance, Lines 49-52, Lines 236-243, and Lines 202-207 are a few examples. Line 245 speculates about the existence of "paths" (I'm not even sure what it means), and such claims about high-dimensional optimization problems cannot be easily proven by the experiments conducted in this paper.

Questions

I would like the authors to address my concerns in the weakness section regarding the premise of the paper and the made arguments.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

Given the the recent reproducibility issues with deep RL, papers in this are must be held to a higher standard. In particular, for this paper: 1. Many depicted training curves are based upon single trainings. 2. 20 seeds were used for running the experiments. I encourage the authors to run the experiments for at least a 100 seeds. 3. Most figures lack any information about the uncertainty of the statistics. Finding ways to incorporate confidence intervals in the plots is important for this paper. 4. The number of environments considered in the paper are limited. Hopper, Walker, Half-Cheetah, and Ant were shown in the main paper, while there are many more environments in the standard gym benchmarks. 5. The quality of the environments in the paper could also be improved. In particular, while standard gym benchmarks are a good start, they do not depict realistic robotic artifacts. There are many realistically modeled robotic environments, such as the ones used in [4] and [5], that can be better representatives for validation of findings. 6. While the paper is specifically targeting continuous control, the authors never discuss the physical aspects of each environment and do not make physical variations to the underlying physical models. 7. The code is not shared open-source in the paper. While this is not a requirement for the conference, it is certainly of great important to the RL community. I encourage the authors to release an open-source link to the code in the main paper.

Summary

This work proposes a new perspective for investigating the stability of RL algorithms -- the return distribution of post-update policies. A simple rejection technique is developed correspondingly to increase the stability in that sense.

Strengths

The paper is very well written and easy to follow. This new perspective of return distribution of post-update policies appears novel and makes much sense. I can imagine many followup directions of this work. The rejection technique is neat while effective.

Weaknesses

There is indeed a few weaknesses in the treatment of the problem. 1. Why do the authors limit the investigation to deterministic transitions and continuous actions? In my understanding the concept the authors propose apply to stochastic environments and discrete actions as well. 2. Maybe it's better to rename the work as "Policy Optimization in a Noisy Neighborhood: On Return Landscapes in MuJoCo" since the work in its current form does not have any other domains. In other words, it is interesting to see whether the observation is general across various domains, especially the stability of interpolation of policies inside a single run. That being said, I believe the novelty of this work is significant and am happy to increase my score based on the authors' response to my concern.

Questions

see above

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

see above

Thanks for the additional result

I have increased the score from 6 to 7 in light of the author response. I do encourage the author to include more ALE games in next revision -- apparently the authors have the required computational resource.

Response to Rebuttals

I thank the authors for their response, and I’ve raised my score. I found the additional experiments were very valuable. That being said, I still find it difficult to recommend this paper for acceptance. The following are just some of the reasons: 1. The authors mostly did not commit to making changes to the paper in the rebuttals. Instead of isolating the response to the reviewer, I wished to note the specific changes & improvements the authors would make to the paper itself. The wealth of questions here indicate interest in the topic, but also show that the writing and the clarity of the paper could be improved. 1. The nomenclature problem is still persistent: * The parameter to return function is mostly deterministic in this paper. The “noisy” naming implies stochasticity in the returns, which is not the main feature of the study. Note that “chaotic” functions, where small input variations can lead to large variations in the output, are very different from “noisy” functions. * The response “these concepts are defined in the paper or in related work” is inadequate for the justifying the misuse of such an established notion as “stability”. Stability in continuous control has been a major topic of research for the better part of a century, and there are tens/hundreds of papers in NeurIPS alone on the notion of stability in non-linear control systems. Any traditional control textbook from 50 years ago have already used the same term with an entirely different meaning. To be clear, recent works such as Reference 21 (Nikishin et. al in the UAI workshop of 2018) did not justify this naming as well. * “Safety” in reinforcement learning is a separate topic of research that this paper is not about, and this word also must be replaced with an alternative and a more proper notion. * “Paths” may better be replaced by linear interpolations throughout the paper; the former is just too general and vague. 2. Dismissing the connection between the optimization objective and the return landscape, on the basis of a limited scope or irrelevance to the work, is not a valid argument in my opinion. The two are significantly connected to each other, and they must be studied and motivated in conjunction with each other. This is a glaring shortcoming with the paper as it is, and I tried to make this point as clear as possible in my earlier review. 3. There are 12 plots in Figure 5, and each one is produced by picking one or two trainings, sampling two policies from those training, and then interpolating between the policies. The small number of trainings and environments involved in this process make this evidence merely anecdotal. Adding more plots is also not a scalable approach to make this experiment more comprehensive. The authors should have found better ways of visualization to present a more statistically reliable evidence. 4. The authors mention that running more seeds can only make the plots more crowded. This, on itself, should have been an alarming bell with the experimental design in this case. To contrast this with an example, most papers report the mean statistics with a confidence interval which only gets smaller and more reliable with more seeds. The authors could have found better statistics and methods of visualization that could scale to many more seeds and environments without sacrificing any clarity. 5. I hope the authors don’t confuse the “initial sections” with the “introduction”. In general, Lines 126-140 could have been out-sourced to the appendix, and Figure 1 is using too much space without presenting a lot of valuable information. Similarly, I don’t find Figure 4 necessary in the main text, as those failure visualizations are too known for deep RL practitioners and roboticists. Instead, more important and missing experiments could have been included in the main body.