Deep Reinforcement Learning with Plasticity Injection

Summary

This paper studies the loss of plasticity in deep RL and proposes a method for mitigating it at the cost of more overhead memory and computation, but doing so in such a way that does not necessitate adding more trainable parameters or affecting predictions. The latter two are key when using their approach for the diagnosis/analysis of plasticity loss in deep RL agents (as they remove confounders due to changes in exploration or representational capacity). They also show that their method can be used to increase the size of the agent network during the course of training to better deal with plateaus in training and save computation cost (i.e. starting with a smaller network to save computation and increase the size when plateauing). This method is also a nice addition to the Reincarnating RL framework, whereby reusing previously trained agents with plasticity injection could help both utilize what was learned in a previous iteration as well as keep the network flexible to absorb new knowledge.

Strengths

- Loss of plasticity in continual deep RL is a very interesting topic and this paper focuses on a methodology to both help analysis/diagnosis of this phenomenon as well as to mitigate it. - The proposed method has other benefits such as allowing us to increase the size of the neural net during the course of training without retraining or affecting current predictions. - Paper is well written and does a good job at introducing and motivating the problem, discussing related works, and exposing the ideas - Perhaps what I'm most excited about in this paper is the following possibility with their *plasticity injection* approach: "*An exciting avenue for future research is understanding trade-offs between architectural design decisions: RL agents typically employ networks that were originally proposed for stationary problems, but perhaps dynamically growing networks would suit the non-stationary nature of RL better.*"

Weaknesses

- The methodology of this paper in the short run would be beneficial, but it's a way around the real underlying problem: the real solution to this problem would likely emerge in the deep-learning toolbox. - Full-sweep experiments would have helped with illustrative experiments in smaller domains without any exploration confounding altogether (e.g. Q-learning but with a full sweep over the full state-action space.) - I like the experiment in Figure 1; helps illustrate the phenomenon nicely. However, the description of what constitutes tasks at each interval could help understand the nature of this illustrative experiment better; e.g., how were the states selected using frozen policies (from every 10M frames), how many such states were selected (as many as the tasks?), how was MC applied to estimate values.

Questions

1. I agree that analysing plasticity in RL is confounded by exploration. But why not study this phenomenon in manageable finite environments and perform full sweeps over all states or state-action pairs? This could at least serve as a minimalistic way to fully remove the impact of exploration and solely focus on plasticity loss. 2. Regarding this claim (lines 298-300) "*However, the idea of plasticity injection is agnostic to the choice of the architecture. For example, it can be applied for residual blocks in ResNets or decoder blocks in Transformers.*" I agree. But do we know whether such architectures still suffer from loss of plasticity when used properly in deep RL? 3. Is anything known about whether the root cause of plasticity loss in deep RL would differ from that of deep learning? 4. What measure do you use to automatically identify loss of plasticity or plateaus during training?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations have been discussed sufficiently well.

Summary

The author propose a solution to the problem of loss of plasticity (decrease in learning effectiveness over time). Their solution is to freeze the existing network and add two new heads, one free to learn, the other frozen to the negative of the free-learning new head. The result is that the intervention has zero effect on outputs at the time of plasticity injection, while allowing a freshly reset head to learn from scratch. They suggest that the method can be used as a diagnostic tool, avoiding possible confounds from other methods that rely on resets and create large changes in outputs. They also suggest that the method improves performance better than existing plasticity-restoring methods, while being less costly than training a larger network from time 0. == Update after rebuttal == Considering the authors' reply, I have slightly updated my evaluation to lean more towards acceptance.

Strengths

The method is novel. The experiments seem thorough (though some clarifications are needed, see below).

Weaknesses

The main difficulty is that the method actually imposes a considerable computational cost and further complexity, while apparently providing a small benefit vs. the much simpler alternative of training a larger network from scratch (<10% total wallclock time, IIUC from Figure 6) It also seems that the method introduces an additional parameter (when to inject plasticity) against the simpler method of training a larger network from scratch.

Questions

- Figure 1: The successive “tasks” are not random. There is an order induced by the fact that they represent results of a slowly learning and improving policy: the policy at 20K vs 10K or 4K is not just different, it is better trained. This might be a confound if the training impacts (say) the complexity of the actual value function. What happens if you do the same figure but randomizing or inverting the order of the successive policies whose value is being predicted? (i.e. at iteration K, the task is to predict the values for K’-th frozen policy, with K’ != K) - l. 233: affect -> affected - Figure 4: “We take the maximum score among the agents with plasticity injection after 25M, 50M, and 100M steps" - come on! Even adding random noise to the values would provide an apparent "net improvement" under such a selection method! At the very least least show us the results with injection at 50M for all. - In Fig 6a, at what time do we perform the interventions? 50M? - IIUC the difference between "Width Scale" in Fig 6a and "Larger Net" in Fig 6b is that the former only enlarges the networks at 50M (?), while the latter is larger from time 0. Is this correct? If so, it should be stated in the main text. - There is an alternative to the tripling of head parameters imposed by the method. Just add one new freshly initialized head, with a scalar multiplier on its output initially set to 0. Presumably this should reduce the added cost while providing equivalent results. What happens if you do that?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors do mention some limitations of the work, especially the cost.

Summary

The work attempts to study the role of plasticity loss in deep reinforcement learning. The paper proposes a new tool to diagnose plasticity loss. The new tool is designed to minimize the effect of other confounders like exploration. The results on Atari show that plasticity loss affects the performance of double DQN, and plasticity injection often breaks performance plateaus.

Strengths

The paper's main contribution is a new diagnostic tool for plasticity loss. The complexity of deep RL makes it challenging to isolate phenomena like loss of plasticity and study them carefully. The proposed tool, plasticity injection, does an excellent job of isolating the effect of plasticity loss while minimizing the impact of other confounders. The empirical results show that double DQN suffers from plasticity loss in Atari games. And plasticity injection improves performance in almost all games. The improved performance is particularly exciting because agents like Double DQN are tuned to achieve high performance in 200M frames. My guess is that if we let these agents run for longer, their performance will plateau in most games. So, the effect of plasticity loss will be more apparent if these agents learn for longer, say 1B frames. And in such long-lived agents, loss of plasticity will become a more dominant phenomenon. So it's interesting to see plasticity loss even with parameters tuned for good performance in 200M frames.

Weaknesses

The proposed diagnostic tool is well thought out and works as intended. However, other contributions in the paper have some issues. The main problem lies when comparing other methods like SnP and Width Scale with plasticity injection. It is not clear if other methods were tuned properly. The appendix said that some parameters of SnP were tuned, but the paper does not mention if other parameters, like learning rate, replay ratio, or buffer size, were tuned for any of these methods. Unfortunately, that is the downside of using only large environments like Atari; we can not perform rigorous empirical investigations, limiting the conclusion we can draw from some of the experiments, like Figure 6. A small issue with the paper is that it only studies value-methods methods and not policy gradient methods, so its title is a bit misleading. A better title would be _Value-based Deep RL with Plasticity Injection_. However, these issues do not affect the paper's primary contribution, so I recommend accepting this paper.

Questions

The results with L2 regularization in the appendix are surprising. Prior results on plasticity loss have shown that L2 regularization helps mitigate it. Figure 1 in the paper by Sokar et al. (2023) shows that L2 regularization improves the performance of DQN. Why do you think there is a discrepancy between the effect of L2 regularization in your results and previous results? I guess that because the parameters of double DQN are not tuned for L2, we might need to tune all the parameters of double DQN to benefit from L2. Sokar, G., Agarwal, R., Castro, P. S., & Evci, U. (2023). The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302.12902.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The main limitation is that other methods are not properly tuned, so the conclusions we can not draw strong conclusions from experiments in Section 5.3.

Summary

This paper provides a deep dive into the issue of plasticity loss in deep reinforcement learning and introduces a novel approach, "Plasticity Injection". The authors employ this technique to not only diagnose plasticity loss but also effectively dissect confounding factors, such as exploration mechanisms. Through extensive experiments, the authors verified that Plasticity Injection significantly increases the agent's performance with minimal computational overhead. The comprehensive investigation of the role of resets in the plasticity viewpoint further enriches the paper's contributions to the field.

Strengths

1. Building upon the reset mechanism, this paper offers a successful strategy for tackling plasticity loss in Deep RL, while dissecting the role of exploration. 2. Plasticity Injection module is simple and straightforward, highlighting its wide potential applicability across various algorithms and architectures. 3. The authors conduct an in-depth analysis of the elements contributing to performance improvement. 4. I liked the qualitative analysis of why the score of Assualt gets stuck.

Weaknesses

1. While the Plasticity Injection module was successfully integrated into the Atari environment, it would benefit from broader testing across various environments, considering that the reset mechanism [1] has consistently performed well in the Deepmind Control Suite environment. 2. Related to Weakeness 1, recent work from Lyle et al [2] have found that the last layer reset does not improve the model's plasticity in their synthetic experiments. Can we really confirm that the reset mechanism is improving the performance on the Atari environments because it is improving the model's plasticity? [1] The primacy bias in deep reinforcement learning., Evgenii Nikishin et al. [2] Understanding plasticity in neural networks., Clare Lyle et al.

Questions

1. Can the authors provide the results of the experiments conducted for the Adaptive Criterion for Injection and L2 Regularization? It seems that these results are missing in the appendix section. 2. A recent investigation from Lee et al [1] has claimed that the reset mechanism is closely tied to the non-stationarity of labels in reinforcement learning while having minimal impact in terms of the inputs' non-stationarity. I would like to see whether the Plasticity Injection (Reset without exploration) behaves similarly to the Reset. [1] Enhancing Generalization and Plasticity for Sample Efficient Reinforcement Learning., Hojoon Lee et al.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

N/A