A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

Summary

The article introduces an extensive empirical analysis of plasticity loss in on-policy reinforcement learning (RL), focusing on Proximal Policy Optimization (PPO). The main findings include that plasticity loss is also present in on-policy RL and that “regenerative” methods that regularly grow network parameters work well in this setting. The empirical evaluation consists of ProcGen and ALE benchmarks. Additional environmental distribution shifts were introduced to study the plasticity loss root cause thoroughly. 8 previously introduced methods to counteract plasticity loss in supervised and off-policy RL are examined, namely L2 Norm, LayerNorm, CReLU activations, Regenerative Regularization, Resetting final layer, Shrink+Perturb, Plasticity Injection, and ReDo. The work presents and analyzes metrics, such as weight magnitude, number of dead neurons, and gradient norm, previously related to the loss of plasticity.

Strengths

The strong part of the work is the extensive evaluation setup of on-policy RL. The claims are well supported by experiments. The paper also interestingly indicates that the problem of warm-start might be partially mitigated by “regenerative” methods that target weight magnitude growth. This has the downstream effect of minimizing the number of dead units in the network previously connected to the plasticity loss.

Weaknesses

While the experimental setup is comprehensive, the findings could benefit from deeper analysis. The article does not present new insights into plasticity loss nor propose novel methods to mitigate it, primarily noting that the problem also exists in on-policy RL. A more thorough exploration of the interactions between methods and their impact on plasticity would significantly enhance the work. In particular, it would be beneficial to understand the differences in plasticity loss dynamics between off-policy and on-policy methods. Additionally, some of the figures are overly cluttered, making them difficult to interpret quickly. To improve clarity, consider summarizing the results from Figures 4 and 5 into a single scalar per method and moving the detailed figures to the appendix. I also suggest using the rliable [1] library for more effective results aggregation. [1] https://github.com/google-research/rliable

Questions

Regarding deeper insights: 1. In off-policy methods, it has been shown that the critic network mainly loses plasticity. Can the authors comment on their on-policy experiments through this lens? Specifically, is resetting only the actor's or only the critic's head more or less beneficial? What role does the common backbone play in this problem? If you separate the actor and critic networks, which can overall impact performance [1], will the conclusions remain similar? 2. What happens if we increase the number of epochs? If we combine this with the examined methods, how will it impact the final performance? Does increasing the number of epochs hurt performance due to more and more outdated samples, or is it primarily due to plasticity loss? [1] Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., ... & Bachem, O. (2020). What matters in on-policy reinforcement learning? A large-scale empirical study. arXiv preprint arXiv:2006.05990.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

yes

Summary

This work studies the loss of plasticity phenomenon in the on-policy continual deep RL setting, where previous work has focused on studying and identifying mitigation strategies for the off-policy RL or supervised learning settings. They conduct experiments over a variety of settings (gridworld, CoinRun, and Montezuma’s revenge) over different variants of environment distribution shift, demonstrating that loss of plasticity still occurs in the on-policy regime. They perform a further analysis of the correlations in both train and test performance with various quantities studied in previous works, across several mitigation strategies previously presented. Based on this, they provide several hypotheses for what properties are needed for successful intervention at addressing plasticity loss as well as ensuring good generalization performance.

Strengths

- Paper is overall well-written and particularly introduces the loss of plasticity phenomenon/warm start problem well. Empirical investigations of loss of plasticity for continual RL are important for the community. - Experiments are comprehensive and presented clearly, including domains of varied complexity. The authors took care in implementing several intervention methods that have arose previously in the literature, and report correlation results comprehensively. - There are new insights about previously successful intervention methods not working in their setting, such as concatenated ReLUs and plasticity injection. From their correlation results, they provide connections between their most successful methods (regularization-based) to the greatest predictor of plasticity loss (weight magnitude, and surprisingly not gradient magnitude or magnitude of weight change). - There are interesting next directions exploring mechanisms of plasticity loss and using these metrics as an indicator for what is occurring in the optimization landscape for continual reinforcement learning.

Weaknesses

- The graphs are difficult to read, given the number of interventions and the colors chosen. It’s difficult for me to see, eg. how much combining soft shrink + perturb with LN improves upon only doing one or the other. - In Appendix D, it might be useful to highlight the significantly improved intervention methods for each environment and shift condition, and some written insight about the table results. Minor typos: - Line 24: arrive -> arrives - Line 142: withing -> within

Questions

- What is the reason for adding LN to soft shrink and perturb, versus original shrink and perturb? It seems that shrink and perturb seems to be doing better than the soft variant for most of your results, and I would guess that the procedure being applied after each step of gradient descent instead of longer intervals could hurt performance. - Do you have any insights why gradient magnitude/weight change magnitude did not have a significant correlation with plasticity or generalization in your setting, in contrast to previous work? - Your Montezuma's Revenge experiments only analyze two intervention methods. How should I interpret the reward numbers in Figure 6? Do you have similar quantities (weight magnitude, gradient norm, dead units, etc.) for these methods?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Authors acknowledge their correlation plots do not attribute causality. Experiments are comprehensive in testing intervention methods except Montezuma's Revenge, which the authors address.

Summary

This paper studies the problem of plasticity loss in on-policy deep RL. The study is quite wide as it covers many environments, types of non-stationarities, and solution methods. The first main result is the demonstration of plasticity loss in various settings. The second main result is the analysis of existing methods in the problems. The results show that regularization methods effectively mitigate plasticity loss in this setting.

Strengths

This is the first study that extensively studies plasticity loss in on-policy RL. Although some prior work has shown plasticity loss in on-policy RL, this study is much more extensive. The paper is generally well-written. The paper claims to be an extensive study of plasticity loss in on-policy learning, and it supports that claim by evaluating a wide range of methods in a wide range of environments. There are some minor weaknesses, but the overall paper is good and would be a good contribution to the plasticity loss literature. I recommend accepting the paper.

Weaknesses

The statistical significance of the results is unclear. How many runs were performed for all experiments? 5? And what do the shaded regions represent in the figures? All the figure captions should contain details about the number of runs and shaded regions.

Questions

What values of $\beta$ were used for Adam? Dohare et al. found that equal $\beta$s in PPO (particularly when used with ReLUs) significantly improve the performance when evaluated over a long time horizon, as may be the case in your experiments. Dohare et al. Overcoming policy collapse in Deep RL. EWRL 2023.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately discuss the limitations.