Reinforcement learning (RL) is inherently rife with non-stationarity since the states and rewards the agent observes during training depend on its changing policy. Therefore, networks in deep RL must be capable of adapting to new observations and fitting new targets. However, previous works have observed that networks trained under non-stationarity exhibit an inability to continue learning, termed loss of plasticity, and eventually a collapse in performance. For off-policy deep value-based RL methods, this phenomenon has been correlated with a decrease in representation rank and the ability to fit random targets, termed capacity loss. Although this correlation has generally been attributed to neural network learning under non-stationarity, the connection to representation dynamics has not been carefully studied in on-policy policy optimization methods. In this work, we empirically study representation dynamics in Proximal Policy Optimization (PPO) on the Atari and MuJoCo environments, revealing that PPO agents are also affected by feature rank deterioration and capacity loss. We show that this is aggravated by stronger non-stationarity, ultimately driving the actor's performance to collapse, regardless of the performance of the critic. We ask why the trust region, specific to methods like PPO, cannot alleviate or prevent the collapse and find a connection between representation collapse and the degradation of the trust region, one exacerbating the other. Finally, we present Proximal Feature Optimization (PFO), a novel auxiliary loss that, along with other interventions, shows that regularizing the representation dynamics mitigates the performance collapse of PPO agents.
Paper
Similar papers
Peer review
Summary
The paper investigates the phenomenon of feature collapse in the popular on-policy RL algorithm PPO. It demonstrates that increasing the sample reuse (number of epochs) in PPO deteriorates the feature rank and plasticity. It also finds that the clipping operation for the conservative update in PPO does not prevent this feature collapse. Furthermore, it shows that feature collapse leads to the degradation of the trust region, and vice versa. Finally, the paper proposes several techniques to prevent feature collapse, including feature regularization, network sharing, and Adam hyperparameter adjustments, though these techniques do not fully resolve the issue.
Strengths
- To the best of my knowledge, this is the first work to investigate the phenomenon of feature collapse in on-policy algorithms. - The experimental design is strong. They appropriately select benchmarks: Atari for discrete action spaces and MuJoCo for continuous action spaces. Additionally, the experiments are extensive and well-conducted. - The connection between feature collapse and trust region is intriguing and distinguishes this work from previous studies on feature collapse in value-based methods.
Weaknesses
- They demonstrated that training policy and value networks for more epochs leads to feature collapse, but this is not typical in PPO settings. It is uncommon to see PPO run for 6 or 8 epochs (Figure 1, 2, and 3). - While they attempt to explain feature collapse in the critic networks, the explanation is insufficient and requires further elaboration (Section 3.1). - While they emphasize that the focus of the paper is not on improving the performance of PPO, they do not provide a clear solution to mitigate feature collapse in PPO (Figure 6).
Questions
- In Section 3.1 and Figures 7, 8, and 9, there appears to be no clear correlation between the number of epochs and the plasticity loss of the critic network, which is counterintuitive since it contrasts with findings from value-based plasticity studies. Could you explain more about this? - In Section 3.2, you state that the clipped sample’s ratio will have their probabilities continue to go beyond the clip limit in PPO. How about replacing the clipping operation with a KL penalty? This technique was proposed in the original PPO paper [1], and it has shown no significant difference in performance compared to the clipping operation in some environments [2]. What impact do you think replacing the clipping operation with a KL penalty would have on feature collapse? - Regarding PFO, instead of applying regularization in the feature space to ensure the current representation does not deviate from previous representations, how about applying regularization in the weight space using EWMA update? - It would be beneficial to investigate Phasic Policy Gradient (PPG) [2]. It trains the actor network to predict the value of the old policy, which is similar to PFO. It would be interesting to analyze feature collapse in PPG to see if similar patterns emerge as observed in PFO. [1] John Schulman et al., Proximal Policy Optimization Algorithms, arXiv 2017 \ [2] Karl Cobbe et al., Phasic Policy Gradient, ICML 2020
Rating
6
Confidence
5
Soundness
3
Presentation
3
Contribution
2
Limitations
N/A
Thank you for your responses to my questions. My answers to some of your responses are shown below. > Q1. … leads to feature collapse, but … it is uncommon to see PPO run for 6 or 8 epochs (Figure 1, 2, and 3). A1. Thank you for the detailed explanation! Increasing the number of epochs makes sense to me now. > Q2. While they attempt to explain feature collapse in the critic, the explanation is insufficient. A2. I would like you to elaborate on Lines 182-184. However, I understand that the main focus of this work is not on the plasticity of the critic network. Thanks! > Q4. … no clear correlation between the number of epochs and the plasticity loss of the critic network … it contrasts with findings from value-based plasticity studies. A4. I think I missed the caption. Thank you for letting me know. > Q5. How about replacing the clipping operation with a KL penalty? … A5. Thank you for clarifying my question! Including these results will strengthen your paper. ~All of my concerns have been well addressed, and therefore I am increasing the score from 5 to 6.~ Updated: After reviewing feedback from other reviewers, I found that similar studies had been conducted prior to this work. Therefore, I have decided to maintain the original score.
Can the reviewer elaborate on their update about prior similar studies? Which studies does the reviewer refer to and to what extent are they similar to our work? We clearly acknowledge building on previous work in value-based methods studying representation dynamics and reference previous works on PPO exhibiting issues with non-stationarity and performance but no connection to representation dynamics. To the best of our knowledge, our work is the first to draw connections between representation collapse, trust-region issues, and performance collapse.
Summary
The paper examines the loss of plasticity and its connection representation collapse in policy networks in online reinforcement learning (as opposed to previously studied value networks in offline reinforcement learning). The paper establishes the problem in the Atari game domain including the growth in the norm of feature pre-activations of the policy network. The analysis isolates and illustrates the problem of increasing representation state collinearity and representation collapse in the trust region of a policy in a toy domain. To mitigate representation collapse the paper proposes a feature regularization loss term based on these pre-activation features of the policy network. The evaluation compares methods to mitigate policy network representation collapse including the new loss, sharing the actor and critic feature trunk, and modifying the Adam optimizer (by resetting moments or faster decay of second moments) in the ALE and MuJoCo environments. The results show the loss has does not impact episode returns but does mitigate loss of plasticity.
Strengths
# originality Good - Investigating plasticity loss in offline RL is established, but has not been extended to online RL. - The analysis provides new insights on trust region optimization. # quality Good - The toy domain isolates the effect of interest to strengthen the empirical results and motivate the new regularization term. - the experiments show modest evidence for improvements and report variation in results over runs. # clarity Mostly good - The exposition on trust region weaknesses and collapse could be clearer and make the main points sooner. The rest of the text was good. # significance Good - Loss of plasticity is an important problem, particularly in the case of online RL. - The paper will be of interest to researchers in continual learning and RL communities; both theoretical (for the trust region points) and practical (for the regularization term) researchers.
Weaknesses
The paper would benefit from outlining the trust region insight sooner as it is a core idea that only gets explained in detail on page 7 of the paper. The results on the regularization term are a bit confusing and mixed. The narrative would benefit from clarifying the aspects that remain unresolved or ambiguous that merit further attention. The Gravitar analysis of sparse reward environments was helpful. But I was not really sure what to make of the effects on episode returns compared to representation collapse. Some way of teasing these effects out more cleanly would help.
Questions
- line 143: "do not include normalization layers" - Does this help plasticity preservation? (even if only to cite prior work on the effects) - lines 151-153: "We vary the number of epochs as a way to control the effect of non-stationarity, which gives the agent a more significant number of optimization steps per rollout while not changing the optimal target it can reach due to clipping, as opposed to changing the value of ε in the trust region for example." - It might be interesting to contrast the effects of varying $\epsilon$ as well. - line 280 (minor): $\pi_\theta(a_i | s)$ - $a_i$ should be $a_1$ in this case - The toy setting was good demonstration of the problems with trust region constraints. It may help to introduce this sooner as I was confused about the trust region claims being made until that point. - Figure 6 (minor) - The caption only mentions "Top" and "Bottom", omitting the middle row of "NameThisGame".
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes. The paper acknowledges remaining opportunities to relate the findings on representation collapse to other research on plasticity loss. And to incorporate more complex model architectures (like memory). There is no need for additional comments on potential societal impacts.
Summary
The paper addresses non-stationarity in RL and its impact on deep learning networks, focusing on PPO. It identifies that networks in PPO, like those in off-policy methods, suffer from representation rank deterioration, leading to performance collapse. The authors propose Proximal Feature Optimization (PFO), a new regularization on the policy representation that regularizes preactivation changes, which can mitigate performance collapse and improve the agent’s performance.
Strengths
- The work exhibits a coherent structure, and the author adeptly elucidates the underlying motivation. The author adeptly elucidates the shortcomings of PPO and offers enough reasons. This paper is well-written. - Section 3's Q1 and Q2 effectively describe the potential problems of PPO, and I concur with the underlying motivation. - The paper not only identifies problems but also suggests practical interventions, such as PFO and sharing the actor-critic trunk, to regularize representations and reduce non-stationarity, showing promising improvements in performance. - The authors provide details of the experiment design, and it's clear.
Weaknesses
The authors claim to open data and code; however, I could not locate them. Therefore, I apologize if I overlooked their presence. Refer to the question section for other weaknesses.
Questions
- Please provide a more detailed explanation on Eq.2. I don't understand the meaning of $\phi$, and should the formula be $s_t$ instead of $S_t$? - In section 4, the authors introduce an intervention that allows the actor and critic to share all the layers. The experiment results illustrate that this can bring some improvements but not significantly. I am more curious about whether the reason for this is the sparsity of reward or the environment itself. I think the latter is more likely, and we can verify this by setting up a control experiment with different reward functions in the same environment.
Rating
6
Confidence
5
Soundness
3
Presentation
4
Contribution
3
Limitations
N/A
Summary
This work provides an empirical study of the feature rank deterioration and loss of plasticity of the Proximal Policy Optimization (PPO) algorithm on Atari and Mujoco tasks. Then links the deterioration of the performance to representation collapse and hence the break of the trust region. From there, the authors propose an auxiliary loss to PPO to maintain the representation rank.
Strengths
The study is very interesting and brings some new insight into how PPO works in practice. The experiments are conducted in clear logic and the analysis and observations are all novel and interesting.
Weaknesses
Figures can be plotted with better quality, don't overlap the labels (in Figure 3), and maybe set the titles to the quantity of interest. I understand that several works are measuring the feature rank at the last policy/critic layer. But why that is sufficient evidence that the policy is losing rank? For instance, if my action space is binary, then the policy's last layer could learn to be rank-2. More information might be stored in previous layers. From another perspective, isn't it expected that the policy learns to lose rank, by compressing state information to only correlate with actions? The proposed Proximal Feature Optimization objective is a bit confusing. First it should be a $\ell_2$ norm? What does it mean by $(\phi - \phi_\text{old})^2$. Secondly, from a high-level, isn't this just a regularization loss as in continual learning that prevents forgetting? But based on my knowledge it will exacerbate the the loss of plasticity. If you want to preserve rank, should not you add some reconstruction loss?
Questions
Could the authors address my questions in the weakness section? From Figure 6, does it mean that the proposed method does not consistently improve the loss of plasticity? If the main goal is to prevent the preactivation norm from going larger, why not just regularize it to be smaller?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Ues
Summary
The paper investigates the phenomenon of loss of trainability and performance collapse with PPO agents in standard RL benchmarks. The same phenonema from other settings is found to hold with PPO agents and, moreover, increasing the number of training epochs exacerbates the effect. Further investigation finds that PPO's trust-region constraint is not maintained when the feature rank decreases and a mechanism for this finding is explained. The authors propose a regularizer, PFO, which softly constrains the change in features is shown to mitigate trainability loss throughout the learning process.
Strengths
The paper investigates the topic of trainability loss in a less-explored context, on-policy policy optimization, and focusing on a popular algorithm, PPO. Better understanding this algorithm and how it interacts with neural networks could lead to widespread impact as the research direction is developed. A wide variety of experiments are conducted with some interesting findings. To me, the most intriguing insight is the connection between low feature ranks and increased violations of PPO's trust region. The investigation is fairly thorough, examining various metrics in tandem throughout the training process. Experiments seem to be conducted and reported in a reasonable fashion. I appreciate the simple, clear example to demonstrate how a poor representation may lead to the clipping constraint being ineffective (line 270-...).
Weaknesses
Currently, the paper feels a bit disorganized and it is a bit difficult to follow the train of thought. There are many different experiments done and placing more focus on the most important could help streamline the paper. For example, the section at line 193 feels a bit out of place since this line of investigation is not mentioned earlier. Another example is that the research questions Q1. and Q2. do not mention the trust-region although it is a large part of the paper. Additionally, various phrases (or claims) in the paper could use more support or evidence. These are discussed more specifically in the Questions section. I am also concerned about the link between feature rank and trust-region violation, a central point of the paper. In Fig.4, the correlation between the prob ratios and dead neurons or prob ratios and feature rank seem weak. There is only a substantial dip in the prob ratios once the feature rank is near zero or the number of dead units is very high. In other words, only when the agent is doing very poorly can we see a strong relationship, which may limit the applicability of the finding. Ideally, one would avoid this regime since performance is poor to start with.
Questions
I would be willing to revise my score based on answers to the following. _Clarification questions_ - Line 308. Could you clarify how the average probability ratio is computed? As written, my interpretation is that, given some window of updates, you consider all ratios above $1+\epsilon$ and take their mean. Then, you do the same for the ratios less than $1-\epsilon$. Finally you divide the first mean ratio by the second mean ratio. Is this correct? If so, this seems potentially misleading because the means are computed conditionally, considering only ratios that are above 9or below) the clipping ratios. Then, we cannot tell what the mean ratio is overall, and whether there are many violations of the trust-region occurring. Moreover, ratios can be misleading since they may overemphasize small probabilities in the denominator. I would suggest also reporting how often the clipping criteria is violated. E.g. compute on collected states which proportion of them have a ratio outside the clipping region after a round of updates on a batch of trajectories. Perhaps it would also be useful to consider ratios less than 1 or greater than 1 seperately instead of computing a ratio. - Line 164 Using the term "plasticity loss" as a metric is confusing because it also refers to the overall phenomena of neural networks being unable to learn additional tasks after training on other tasks. Does this term refer to the capacity to fit random labels? In that case, I would use a term that is more descriptive e.g. random label trainability (RLT). Also, how are these random labels generated in this context? In the original paper, a regression task was made since a value-based algorithm was used. With a policy-based algorithm, how is the plasticity evaluation task set up? - Fig.1 Why does the preactivation norm plateau at 10^4? - Which data is used to produce the box plots? The caption for Fig. 6 mentions "A boxplot includes 15 runs with the different epochs". Does this mean the box plot conains data across training times and epochs? - In the box plots, how are outliers determined? We can see they lie outside the interquartile range indicated by the "whiskers" but what additional criteria is there? - How is the PFO regularizer related to adding a regularizer in parameter space? Say $\ell_2$ regularization between the current parameters and the previous parameters? Basically, this seems similar in spirit to adding a target network to value-based methods. - Line 94-99 (and elsewhere) "PPO introduces additional non-stationarity...": I do not think it would be fair to see PPO has more nonstationarity than other policy gradient methods. REINFORCE and PPO both have nonstationarity due to the changing policies. REINFORCE can also be interpreted as a trust-region-like method. Since REINFORCE is essentially gradient ascent on the expected return objective, we can view it as minimizing a linear approximation with a quadratic constraint (see paragraph containing eq. 5.5 in [4]). In this light, by doing a single optimization step before changing the surrogate objective, we could say REINFORCE has _more_ nonstationarity. - Line 257. "The underlying assumption...approximately orthogonal..." Could you give some more support to this claim? As mentioned above, with state aggregation, this clipping strategy could be effective without having orthogonal features. - Line 8-9: "...policy optimization methods which are often thought capable of training indefinitely". I would disagree with this claim. Deep RL algorithms have often be thought to be high-variance and unstable so I do not think it would be common to believe they can train indefinitely. I suggest replacing this phrase and simply mention the phenonmenon has been underexplored in the context of on-policy policy optimization. - The parameter norm has been linked to plasticity/trainability loss in neural networks. Is there a reason why the paper measures feature norms instead? _Broader questions_ - The performance degradation from training more epochs could be attributed to difficulties of offline RL. As the number of epochs is increased, the more the procedure ressembles offlne RL---a setting in which standard RL algorithms are not suitable for. What do you think of this? - I find the proposed solution (Proximal Feature Optimization) to be slightly unsatisfying, Generally, I can see how using regularization on the policy and features can mitigate the performance collapse but would this be simply delaying the problem? If we train for longer, would we still expect to see the same performance degradation? - Interestingly, the value network is not the problem. It is the policy network that suffers from representation collapse. Do you have any thoughts about this? I think this could be an interesting avenue of investigation. _Suggestions_ - I would suggest removing or summarizing the paragraph at line 193 into a single sentence. It is clear that the policy would be the same for all states if the features are constant. This would imply the policy is uniform. Currently the paragraph seems to explain this in a roundabout manner when a concise explanation suffices. - A suggestion would be to try the Atari environemnts identified in [1] to benefit from plasticity injection. On many Atari games, little or no benefit was found, so it may be more meaningful to focus on those where there is more evidence for trainability loss. - I would be careful about mentioning that low feature rank necessarily means that the representation is poor. Successful training of deep neural networks often involves a reduction in feature rank and the relationship between rank and performance can be complex [3]. - As a sidenote: even if there is a low dimensional representation, this does not necessarily imply the clipping criteria will be violated. For example, if we use state aggregation, then multiple states are clustered into one aggregate state. Then, the policy will be exactly the same for all these states, so if the policy exits the clipping region for one those states, there will no longer be any incentive to update the policy any further on any of them. - A minor suggestion is to use stable rank [2] as a measure of the rank instead of effective rank since it is a continuous quantity in the singular values. [1] "Deep Reinforcement Learning with Plasticity Injection" Nikishin et al. [2] https://nickhar.wordpress.com/2012/02/29/lecture-15-low-rank-approximation-of-matrices/ [3] "An Empirical Study of Implicit Regularization in Deep Offline RL" Gulcehre et al. [4] https://www.stat.cmu.edu/~ryantibs/convexopt-S15/scribes/05-grad-descent-scribed.pdf
Rating
5
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
These are discussed.
Addressing the remaining points
>**Q5.** Using the term "plasticity loss" as a metric is confusing because it also refers to the overall phenomena … **A5.** Yes, we use this metric to compute the loss from fitting a random initialization's trajectories. It is defined in the background (114-122) and to fit the actor we use a KL divergence. In the paper, **we distinguish between the metric called plasticity loss and the phenomenon called loss of plasticity.** This is the value of a loss, that’s why we call it plasticity loss. However, the phenomenon has also been termed plasticity loss by Lyle et al. (2024), so we agree that reusing it for the loss can be confusing. We propose to state our distinction between the two forms in our background, making it less confusing. Otherwise, we can change the loss term to to random fit error. >**Q6.** Fig.1 Why does the preactivation norm plateau at 10^4? **A6.** Typically, the preactivation norm increases until all the neurons of the feature layer are dead (in the case of ReLU). After that, all the gradients for the weight matrices are 0, so the norm flattens out. >**Q7.a** Which data is used to produce the box plots? … **A7.a** Each run is summarized with an average over the last 5% of its progress. The 15 runs consist of 3 epoch values (4, 6, 8) x 5 seeds. This is presented in lines 307-308 and further detailed in the appendix lines 641 - 644, 658-659. >**Q7.b** In the box plots, how are outliers determined? … **A7.b** The whiskers are determined by the highest observed datapoint below Q3 + 1.5 IQR (similarly for the lower one) (default of matplotlib). The outliers are points outside of the whiskers. We thank the reviewer for the question and will add this information to Appendix lines 659+. >**Q9.** I do not think it would be fair to see PPO has more nonstationarity than other policy gradient methods… **A9.** We agree that, from this perspective, REINFORCE could be considered as more nonstationary than PPO. **We will remove this "more nonstationary" observation; it is not critical to the work.** The critical point of in work is that PPO performs multiple epochs on the same data and, in this sense, can be more prone to "overfitting" to previous experiences, which can become worse with more epochs. >**Q10.** Line 257. "The underlying assumption...approximately orthogonal..." Could you give some more support to this claim? … **A10.** The presumed orthogonality here is between the states where the policy should act differently, like with two different groups of aggregated states. Otherwise, for the states where the policy should be similar (same group of aggregated states), some alignment in the features and a reduction of dimensionality are indeed desired for generalization. However, with clipping, this alignment should be associated with the right relative feature norm; otherwise, the clipping will be violated, as observed in Figure 5, where a larger alpha would make the relative deviation of the action probabilities larger. >**Q11.** Line 8-9: "...policy optimization methods which are often thought capable of training indefinitely"... **A11.** This is indeed arguable. We believe that trust region methods aim to overcome this high variance and instability. However, this is not a critical claim, and we can remove it. We thank the reviewer for the suggestion. >**Q15.** Interestingly, the value network is not the problem. … **A15.** **Yes, this is one key point of our work.** Policy networks also suffer from representation collapse independently of value networks, as noted in the caption of Figure 1. Our intuition and the main motivation of this work is that, similarly to value networks, policy networks are also subject to the non-stationarity of their inputs and outputs (background, lines 88+), and optimizing deep neural networks under non-stationarity is known to cause issues. >**Q16.** A suggestion would be to try the Atari environemnts identified in [1] to benefit from plasticity injection… **A16.** Phoenix is one of the environments we have in common and where we see the most impactful results. For this work, we did not want to select environments with prior knowledge of existing collapse in value-based methods to demonstrate the collapse in policy optimization; we rather used the unbiased sample recommended in Atari-5 (Aitchison et al. 2023) because **we did no want to unconsciously cherry-pick environments, where certain approaches may perform better.** Moreover, we would like to highlight that, on Atari, we could not find any large-scale study of on-policy RL algorithms and PPO demonstrating the collapse of the actor’s representation from which we could pick our environments at the time of submission. >**Q17.** A minor suggestion is to use stable rank [2] as a measure of the rank … **A17.** **We have tracked 5 different measures of the rank, spanning continuous or discrete and relative or absolute metrics.** We conducted a thorough comparison of these in Appendix E.
I appreciate the clarifications and explanations. After reading responses to other reviewers, I have to agree that certain points of Reviewer qwmj are still concerning, including the ones around the performance of the baseline. I am also still a bit underwhelmed by the fact that trust-region failures seems to only occur when the representation has already collapsed and the performance is poor. I will keep my score at present.
We are happy that our clarifications addressed all your previous concerns. > I have to agree that certain points of Reviewer qwmj are still concerning, including the ones around the performance of the baseline. **We understand the concern; however, this is a matter of clarification.** We take the thoroughness of our implementation with utmost importance and **have clarified all the necessary bits in our reply to the reviewer,** which we summarize below: 1. Recall that **our results hold on Atari and we have already replicated them with the CleanRL** implementation the reviewer is using. 2. Recall that **our implementation for MuJoCo is based on recent implementations** for continuous action spaces influenced by seminal work studying implementation details in PPO and that **we find this setting more relevant to our audience.** 3. **Replicate our implementation in CleanRL, which is still exhibiting collapse,** so the reviewer can more easily inspect the code. 4. **Adapt the default CleanRL implementation the reviewer is using with a fully state-dependent action distribution and observe collapse** in the setting they are familiar with and interested in. > I am also still a bit underwhelmed by the fact that trust-region failures seem to only occur when the representation has already collapsed and the performance is poor. **Unfortunately, this is a fact.** We extensively investigated the link in the early stages of the project to find strong evidence throughout training, **which to us was also the more exciting claim to prove; however, countless examples made us realize that the link did not necessarily hold throughout training but only around the collapsing regime.** Using only positive examples or a narrow view of the correlations, to **claim that such a link is present throughout training would be extremely misleading and has been an issue in offline RL.** The work by Gulcehre et al. (2022) that the reviewer references is an excellent example of this. They conclude that previous exciting associations between performance and rank in offline RL are misleading and do not hold when considering a large experimental scope. Our conclusion is that trust-region violations become more evident when representations are about to collapse. **This does not mean that one should only be concerned about the link when performance is starting to deteriorate.** **The representations don’t collapse all of a sudden; they deteriorate throughout training until they reach collapse. So, mitigating representation degradation should happen throughout training and not only when around the collapsing regime.** We hope these clarifications address your concerns. Thank you for your input, and we look forward to your feedback.
Summary
This paper presents a series of experimental studies to diagnose the learning issues of PPO under non-stationarity in Atari-5 and MuJoCo tasks. Based on the results, this paper establishes a connection among feature rank/norm, plasticity loss and trust region violation and learning performance. To mitigate the issues in feature representation, this paper proposes Proximal Feature Optimization (PFO) and demonstrates its effectiveness in mitigating the learning issues under non-stationarity. Besides, the effects of sharing network parameters and adapting Adam are also evaluated in the same context.
Strengths
- Most existing works on plasticity loss or learning under non-stationarity focus on value-based RL or off-policy AC methods. In contrast, this paper focuses on on-policy algorithms, mainly PPO, which helps the RL community gain a better understanding of this problem. - The experimental studies contains the results with multiple metrics, i.e., feature rank, feature norm, plasticity loss, dead neuron, excess ratio. These results will be a useful reference to the audiences. - This paper reveals the connection among factors like feature rank, plasticity loss, excess rate, and learning performance.
Weaknesses
- The writing is almost clear. The empirical results and conclusions can be better organized and made prominent. - The proposed method PFO is closely related to DR3 (Kumar et. al, 2022) and RD [1], both of which propose regularizing the feature representation. More discussions are necessary. - Although PFO are demonstrated to mitigate the feature rank, plasticity loss, excess ratio in Section 4, it does not bring clear improvement in terms of learning performance like episode return (according to Figure 18 to Figure 27). This cripples the significance of PFO and also the potential causal connection between the learning issues investigated and the learning performance. --- Reference: [1] Reining Generalization in Offline Reinforcement Learning via Representation Distinction. NeurIPS 2023
Questions
1. The epsiode return curves in Figure 24 and Figure 26 (i.e., for MuJoCo Hopper) look strange. Is there any explanation to the collapse? A regular implementation of PPO should work in Hopper when the epoch num is 10. 2. The authors mentioned 4 MuJoCo tasks and 5 Atari games in the Experimental Setup paragraph. However, it seems only 3 Atari games (Phoenix, NameThisGame, Gravitar) and 2 MuJoCo tasks (i.e., Humanoid, Hopper) are used for the evaluation in Section 4. Are there more results on the remaining tasks/games?
Rating
3
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
The limitations are discussed in Section 5. However, one major limitation of this work is that the results in this paper do not provide sufficient support for the point that "mitigating the learning issues in feature rank, plasticity loss of PPO can improve the learning performance in terms of episode return".
| Implementation detail | CleanRL’s default | ClearnRL's default adapted | Our implementation | | ------------------------- | --------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------------- | | Network output | State-dependent mean / State-independent std | same / State-dependent std | same / State-dependent std | | Transformation of the std | exponential | same | Softplus (recommended by Andrychowicz et al., 2021) | | Action distribution | Normal | same | TahnNormal (recommended by Andrychowicz et al., 2021) | | Reward transforms | Normalize and clip | same | None (to keep the default non-stationarity) | | Observation transforms | Running normalization and clip | same | Normalization at initialization (to keep the default non-stationarity) | | Layer initialization | Orthogonal with custom scale | same | Default PyTorch initialization | | Learning rate annealing | True | True, False (collapses for both) | False | | Value loss clipping | True | False (out of scope) | False (recommended by Andrychowicz et al., 2021)
Response to the rebuttal
I appreciate the authors' careful response and the additional results. > More discussion about PPO implementation Thanks for the implementation details provided by the authors. According to Figure 8 (PPO at MuJoCo without intervention) and Figure 24 (PPO at MuJoCo-Hopper with intervention), the results show that (1) the PPO implementation used in this work exhibits collapse in HalfCheetah and Hopper with epoch=10 basically after 1M steps (Figure 8), and the collapse remains when different kinds of intervention are applied. I still have a concern about the validity of the PPO implementation used in this work. After a quick run with github CleanRL implementation of PPO, I did not observe collapse for HalfCheetah and Hopper after 1M steps. With the details provided by the authors, I realize that it should stem from the difference in implementation. On my side, the learning rate gradually decays to 0.1 * [init_learning_rate], and the variance vector is state-independent (I think this is a convention). The authors mentioned "We do not decay the learning rate (line 154) we are interested in agents that can ideally train indefinitely", which does not quite make sense to me as "train indefinitely" is meaningless after convergence or collapse in a single-task RL training process. And a proper learning rate decay (e.g., decay to 0.1 * [init_learning_rate]) should also allow the agent to learn within a long horizon. > More discussion about evaluation environment choice The authors mentioned, "We selected the environments where the collapse was the most consistent to test multiple interventions while maintaining a reasonable compute budget". According to Figure 8, it looks like the collapse is more severe in Ant and HalfCheetah than in Humanoid. This turns out to be a bit contradictory with the response above that Humanoid is used for evaluation but Ant and HalfCheetah are not included. > More discussion about the collapse The results in Figure 24 show that all the interventions fail to address the collapse in Hopper. And based on my personal experience, PPO with proper implementation does not collapse in MuJoCo tasks (as mentioned above). My main concern remains after the rebuttal.
We sincerely thank the reviewer for promptly engaging with our rebuttal and clarifying their main concern. **We take the thoroughness of our implementation with utmost importance** and understand the concern of the reviewer. **We have been preparing a complete answer with sufficient experimental evidence to address your concerns.** We address the implementation concerns in MuJoCo below, but first, we would like to highlight that **the main results in our paper are also shown with the Atari environments** and that we have included a strong replication of our results in Atari with the **CleanRL codebase. It exhibits the same collapse we observed with our implementation.** As the reviewer, we wanted to rule out the collapse happening because of any bugs or implementation details with as much confidence as possible. In addition, note that we have referenced Dohare et al. (Overcoming Policy Collapse in Deep Reinforcement Learning, 2023), who also observed a collapse on MuJoCo. > I did not observe collapse for HalfCheetah ... I realize that it should stem from the difference in implementation. We understand the concern and have taken two actions to help the reviewer increase their trust in our results: 1. **Clarify the motivation behind our setting, replicate it on CleanRL, and observe collapse:** In addition to sharing our code with the reviewers, we replicated it in CleanRL as we did with Atari so that it is easier for a reviewer familiar with CleanRL to inspect. We would like to highlight that our setting is fully described in Appendix B.2, and its differences from **CleanRL stem from using recent implementation details for continuous action spaces since the original PPO paper** (as CleanRL is a faithful replication of it). Several works we cited have studied the implementation details of PPO and have given recommendations that shape how PPO agents are implemented these days (Andrychowicz et al., 2021; Engstrom et al., 2020). **We believe that this setting is more relevant to the community than the initial implementation of PPO in 2017.** A table with all the differences can be found below. 2. **We have taken the ClearnRL implementation used by the reviewer and applied minimal changes that resulted in a collapse.** - Remove value loss clipping. This is an unnecessary trick that complicates the analysis (we are primarily interested in the actor). It is also not recommended by Andrychowicz et al. (2021) - Make the standard deviation output of the action space dependent on the state. Since our research focuses on the problems related to state representation, the study inherently makes more sense if the standard deviation is state-dependent. **If the standard deviation were state-independent, it would not be influenced by the representation collapse we are investigating, thus undermining the core premise of our study.** **With this, we have obtained similar collapsing curves as seen in our submission with and without an annealed learning rate.** We have communicated the implementations and training curves to the AC. > The authors mentioned "We do not decay the learning rate (line 154) we are interested in agents that can ideally train indefinitely" … Moreover, let us clarify the wording and context of our work, we acknowledge that we should have provided a more detailed explanation in our initial submission. **We are interested in an online learning setting where the agent receives the experiences continuously, and it is not straightforward to determine when the learning process should terminate.** The continual and online learning aspect of our experimental setup makes it challenging to apply standard annealing schedules, and a primary goal of our research is to maintain the plasticity of representations. Therefore, we choose not to anneal the learning rate to better study the behavior of representation collapse under these conditions. **The environments we use are single-task environments to ablate additional MDP non-stationarity, but they are complex enough for the agents to keep improving when trained for longer than our common benchmark limits.** In the tasks we have tried, the policies trained with a constant learning rate collapsed while apparently still improving and before stabilizing/converging. **We do not claim that our approach is the only way to study this phenomenon**, and we acknowledge that annealing the learning rate could be a method to delay or prevent collapse. In the additional runs we have provided, we observe collapse even when annealing the learning rate when training for long enough. > Choice of eval environments **We have communicated the results of all tasks to the AC. This should resolve this issue.** We considered the severity of the collapse regarding both Tanh and ReLU activations. The default hyperparameters of Ant don’t collapse on either, and we favored Humanoid for its large action space over HalfCheetah, which seemed redundant to us with Hopper.
Reply
I thank the authors for their rebuttal and clarification. By "setting the title to the quantity of interest" I just meant that you could directly set an informative title for the figures. I will maintain my scores.
We thank the reviewer for their clarification. We will add this to our final version.
Thank you for addressing my questions. I do not see any substantial changes that would alter my score. After discussion with the other reviewers I am lowering my score in light of the weakness of the causal evidence for what drives collapse and the lack of strong results showing cases where PFO substantially improves over alternative methods.
We thank the reviewer for the follow-up. > “in light of the weakness of the causal evidence for what drives collapse” Can the reviewer elaborate on this? Some reviewers have raised concerns about the scope of the cause evidence, which holds primarily around the collapse regime**, but it does not seem like concerns have been raised about the weakness of the evidence.** If the reviewer refers to **the scope of the collapse, then we have addressed reviewer YzGo’s concern about the underwhelming scope in our reply to them.** Quoting: “**Unfortunately, this is a fact.** We have extensively investigated the link in the early stages of the project to find strong evidence throughout training, **which to us was also the more exciting claim to prove; however, countless examples made us realize that the link did not necessarily hold throughout training but only around the collapsing regime.** Using only positive examples or a narrow view of the correlations, **to claim that such a link is present throughout training would be extremely misleading and has been an issue in offline RL.** **This does not mean that one should only be concerned about the link when the performance is starting to deteriorate. The representations don’t collapse all of a sudden, they deteriorate throughout training until they reach collapse. So mitigating representation degradation should happen throughout training and not only when around the collapsing regime.**” > and the lack of strong results showing cases where PFO substantially improves over alternative methods. **"PFO substantially improving over alternative methods" has never been a claim made in this work. This is a misunderstanding raised by reviewer qwmj, which was clarified in our reply to their review.** PFO is not meant to improve best run performance (lines 330-332), but to mitigate collapse, and to show that mitigating representation collapse mitigates performance collapse. Quoting our reply to reviewer **qwmj:** “the **significance of PFO is realized by a) successfully mitigating collapse** as observed in Figure 6 with a significantly higher median performance indicated by the black lines, and b) improving the representation metrics and obtaining a better trust region, therefore **strengthening the relation we’ve drawn between representation, trust region, and performance around collapse.**”
I thank the authors for their thorough discussion and replies here and with the AC. The further clarifications have been illuminating and adjusted my views overall. I've raised my score as a result of seeing: - the presentation was weaker than initially believed (due to the confusions being discussed and potentially resolved) - the results are narrower in scope than I originally understood (which remains valuable but narrows the audience of interest somewhat) - the results are empirically stronger around the circumstances of collapse and potential understanding they yield than I most recently believed
Important inquiry regarding claims and positioning relative to existing works
I appreciate the thorough experiments that reviewers are already assessing, but something has come up for which I wanted to give you an opportunity to respond. The following two sentences of the abstract seem to be important claims/motivations for the paper. (1) > it has been overlooked in on-policy policy optimization methods which are often thought capable of training indefinitely (2) > revealing that PPO agents are also affected by feature rank deterioration and loss of plasticity. However, there are several papers, including one well-recognized in the loss of plasticity community, that already refute (1) and establish (2): Dohare, S., Sutton, R. S., & Mahmood, A. R. (2021). Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325. Dohare, S., Hernandez-Garcia, J. F., Rahman, P., Mahmood, A. R., & Sutton, R. S. (2023a). Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306.13812. Dohare, S., Lan, Q., & Mahmood, A. R. (2023b). Overcoming policy collapse in deep reinforcement learning. In Sixteenth European Workshop on Reinforcement Learning. I would like to hear from the authors about how they plan to position their claims/work within the context of this existing literature. Furthermore, there are results given on four Mujoco tasks in the latter paper, which are highly relevant for this paper. It would be natural to build on these results. However, these existing results are not acknowledged. It is important because the acknowledgment would have allowed the reviewers to compare your results. As we look closely now, some of the results do not match. For example, Dohare et al. (2023b) show that adam-equal-betas do not collapse on Hopper, whereas in your experiment, it does. This baseline is also not used in the two new Mujoco environments that were submitted later. PPO+L2 is another important and effective baseline in the latter work that seems relevant to your work but is not mentioned or compared. Moreover, claims regarding loss of plasticity are an important part of the paper, but references to this literature seem insufficient, except for Kumar et al. (2021), Nikishin et al. (2023), and Lyle et al. (2022, 2024). A wider coverage of existing work, e.g., the above-listed ones and the following, would help better understand your work in the context of the literature. Sokar, G., Agarwal, R., Castro, P. S., & Evci, U. (2023). The Dormant Neuron Phenomenon in Deep Reinforcement Learning. In International Conference on Machine Learning 2023. Kumar, S., Marklund, H., & Van Roy, B. (2023). Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308.11958. Abbas, Z., Zhao, R., Modayil, J., White, A., & Machado, M. C. (2023). Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents (pp. 620-636). PMLR.
Rebuttal and concerns about misleading claims
**We sincerely thank the AC for their commitment to a thorough review process** and for including us in the discussion of important concerns that were raised during the private discussions between the reviewers that we would not have otherwise had an opportunity to respond to. **This clarifies the recent comments we have received and allows us to discuss them constructively**. We hope to address them during the time left for the discussion phase. > several papers that already refute (1): it has been overlooked in on-policy policy optimization methods which are often thought capable of training indefinitely… and establish (2) revealing that PPO agents are also affected by feature rank deterioration and loss of plasticity. **We respectfully disagree. These works do not refute (1) nor establish (2). This is a misleading claim.** First, the “it” in (1) refers to “networks … exhibit a decrease in the rank of their representations, a decrease in their ability to regress to arbitrary targets, called plasticity” (line 5 abstract, and lines 25-56 introduction.) and “raises the question of how much PPO agents are impacted by the same representation degradation attributed to non-stationarity.” (lines 34-36 introduction). We agree that the “thought capable of training indefinitely” bit is controversial and that it is common to believe the opposite. We discussed this with reviewer YzGo and mentioned that it was not a critical finding of the paper and that we would remove it. However, **none of the works mentioned discusses the representation degradation of PPO, i.e., the “overlooked” bit of (1) and the main point of (2).** **Dohare et al. (2021)** have **a single RL experiment** with an environment **with non-stationary transition dynamics** and **only track online performance**. There is **no curve about representation degradation** (representation rank, dead units, feature norm, etc.), and the **“plasticity loss” refers to the degradation of the online performance** of this environment with non-stationary dynamics. This confuses assessing the ability to learn a sequence of optimal policies with the evolution of the learning network to fit the same arbitrary function, which we measure with the random label fitting capability. **Dohare et al. (2023a)** track dead units and effective rank but **only in the supervised learning experiments, confirming our point that representation degradation is overlooked in online RL. They also perform the same experiment as Dohare et al. (2021), using a non-stationary environment** to measure only online performance. **Dohare et al. (2023b)** have experimented in stationary online RL with PPO as we do, but again, they **only track online performance in the complex tasks (MuJoCo), with no figures of representation degradation and no figures of plasticity loss. With all due respect, all the claims made in the paper about plasticity appear to be speculations.** Representation is plotted once for a toy example, but it is one-dimensional, so already collapsed and does not show an evolution from rich to poor. These works are important in developing a better understanding of training deep neural networks under non-stationarity; however, they do not specialize in the specific topic of representation dynamics we discuss in our work. Importantly, **it is specifically tracking representations that allows us to draw a connection between representation and trust region degradation.** > Mujoco tasks in the latter paper, which are highly relevant ... natural to build on these results. However, these existing results are not acknowledged. It is important because the acknowledgment would have allowed the reviewers to compare your results. **To the best of our knowledge, Dohare et al. 2023b do not provide code with their paper, and their setting is not fully described.** Several implementation details are missing, e.g., whether they decay the learning rate, the value of the Adam $\beta$ hyperparameters used for the plots (unlike for L2, which is well described), and the action distribution used. Furthermore, some of the design choices in their setting are unusual, e.g., a 4x larger minibatch size effectively dividing the number of gradient updates per batch by 4 (less overfitting) and no gradient norm clipping. **For these reasons and because we had a more modern setting in mind with an action space fully dependent on the representations, basing our implementation on a replication of theirs did not seem like a natural choice to us.** It is also not clear if highlighting their specific results on MuJoCo or basing our implementation on theirs would have provided a stronger reference to compare our results. **To the best of our knowledge, the paper has not been replicated, the code is not available, and the runs have not been thoroughly reviewed. In fact, reviews of the paper in EWRL16 state “reviewer points out some issues with the empirical results”.**
Continuation
> some of the results do not match ... adam-equal-betas do not collapse on Hopper, whereas in your experiment, it does. **This is likely due to the difference in settings. Our code and run histories are available for inspection, but it does not seem like theirs are.** Moreover, it’s not clear what values of the Adam moment coefficients Dohare et al. (2023b) used. It seems that these were tuned per environment, and while their narrative suggests decaying the moments faster, i.e. decreasing the coefficients, they tuned $\beta_1$ on values strictly larger than its default. In our experiments, we followed their narrative and set the second-moment coefficient to match the smaller first-moment coefficient. This is fair to the other interventions, which were also not tuned. **Like any of the other interventions, we do not deny that further tuning the coefficients can yield better results.** > This baseline is also not used in the two new Mujoco environments ... **We sincerely apologize for this.** We ran them but used an older version of the script to generate the figures. **This is now fixed, and the updated plots can be found at the same link. We observe the same results as in the submission, that the intervention is not sufficient to prevent collapse.** > PPO+L2 ... It is important to note that the loss of plasticity subfield is rapidly growing, and several solutions are being suggested; however, no established solutions or baselines exist yet. If we claimed to have designed a new SOTA solution, we would have to run as many of the previously established solutions as possible to compare against them. However, this is not the purpose of this work, which is centered around building understanding. As mentioned in this discussion phase, **PFO, among the other interventions we have tried, is used to strengthen the connection we draw between representation, trust region, and performance. We found that none of the interventions suggested before directly targeted the degrading representations symptom with a connection to trust region, so we found that adding PFO to the interventions would strengthen the narrative.** **Our narrative does not focus on the norm of the weights**; instead, we focus on the norm of the features because the collapse happens at the feature level, which depends on both inputs and the model architecture; **therefore, weight decay did not seem like a crucial intervention to prioritize.** > claims regarding loss of plasticity are an important part of the paper, but references to this literature seem insufficient, > A wider coverage of existing work, e.g., the above-listed ones and the following, would help better understand your work in the context of the literature. Given an extra page, we are happy to cite and relate to all the aforementioned works to give more context to our work. **Kumar et al. (2023)** provide a comprehensive comparison and categorization of methods used to mitigate plasticity loss in continual supervised learning, it is not clear if solutions in this setting translate to solutions in RL; however, this setting is an important stepping stone towards maintaining plasticity. **Sokar et al. (2023)** provide an alternative characterization of plasticity loss in RL using dormant neurons, however, like Lyle et al. (2022), their study only includes value-based methods. **Abbas et al. (2023)** study representation metrics such as feature norms like our work but in value-based methods. They observe a sparsification of the norm while we, taking the dying neurons out of the equation, find that the preactivations blow up. **Overall, the findings in these works and our work are complementary, and further discussing them on an additional page of related work is mutually beneficial.** ### Conclusion > I would like to hear from the authors about how they plan to position their claims/work within the context of this existing literature. **We commit to discussing all the related work raised in this rebuttal, however, we strongly believe that this does not change the position of our work,** which provides a complementary view of representation degradation and plasticity loss in on-policy policy optimization and a novel connection to trust region methods explored both empirically and theoretically. **The plasticity community is growing with great complementary perspectives that we hope to add to. We express strong concerns against the misleading private claims raised using the line of work by Dohare et al. to undermine the contributions of our work.**
Thanks for responding promptly. I am here to help you and commit to my best effort to resolve the confusion fairly. Let us continue a respectful and productive conversation to help us resolve the confusion. > This clarifies the recent comments we have received and allows us to discuss them constructively. > We express strong concerns against the misleading private claims raised using the line of work by Dohare et al. to undermine the contributions of our work. I believe the above concerns are unwarranted. The inquiries were mine and not covered by other reviewers. Sometimes, I see meta-reviews bringing new concerns that the authors cannot respond to. Hence, I am bringing these up with you for clarification as soon as I discover them. I do not see any of the recent comments by reviewers reflecting these concerns, and hence, no reason for the authors to be concerned. About the content of the response: While you claim that (1) is not refuted, you also agree to remove “thought capable of training indefinitely,” which is part of (1). Hence, claim (1) does not stand as it is. It was already shown to be not true. Moreover, the “it” of (1) in the abstract is described as the following phenomenon: > previous works have observed that networks in off-policy deep value-based methods exhibit a decrease in representation rank, often correlated with an inability to continue learning or a collapse in performance. It is the “an inability to continue learning or a collapse in performance” part that the works by Dohare et al. already show. Hence, a more nuanced and more accurate claim could be something along the lines of “although an inability to continue learning or a collapse in performance for PPO was shown in prior works, its relation with a decrease in representation rank was not studied.” Regarding the measure of loss of plasticity, there is a different, but not necessarily incompatible, one compared to yours, which considers loss of plasticity to be the phenomenon where, on a task of similar difficulty, the learner performs worse than when learning from scratch, such as in Abbas et al. (2023). I believe loss of plasticity claimed in Dohare et al. (2021, 2023a) is in a similar sense rather than merely speculative. Hence, the existence of these results, in my opinion, still weakens the claim that “revealing that PPO agents are also affected by … loss of plasticity”, which is part of (2). Regarding performance collapse, Dohare et al. (2023b) clearly show it, too, weakening the strong claim of (1) again. It is expected that newer works will be more thorough than previous ones, which I have also acknowledged for your work. However, acknowledging highly relevant prior results, such as those by Dohare et al. (2023b), helps readers assess the work more comprehensively and in the context of the literature. The weaknesses of this prior work are not a valid cause to avoid acknowledging or positioning your work relative to it. You already refer to Dohare et al. (2023) later, for which these weaknesses indeed were not a problem. I do find your reasoning for the discrepancy in results compared to Dohare et al. (2023b) acceptable. I also find your argument regarding PPO+L2 to be reasonable. > Given an extra page, we are happy to cite and relate to all the aforementioned works to give more context to our work. > Overall, the findings in these works and our work are complementary, and further discussing them on an additional page of related work is mutually beneficial. These sound reasonable to me as well. > We commit to discussing all the related work raised in this rebuttal. However, we strongly believe that this does not change the position of our work I think the position might change due to the weakening of claims (1) and (2). However, I do not think a change in the position necessarily weakens the work itself. A more accurate, fair, and comprehensive position, for example, something similar to what I suggested above regarding a revised claim, may rather strengthen the claims and the contributions.
We thank the AC for the prompt and detailed response and for clearing our concerns. This is very much appreciated, and we recognize the contribution of the AC, which is well beyond expectations. We apologize to the community if, while expressing our concerns, any of our comments came across as overly critical. Our intention is not to undermine any previous work. Instead, we want to grow the understanding of plasticity by acknowledging previous work and building on previous limitations, as future work will build on ours. > … you also agree to remove “thought capable of training indefinitely,” Moreover, the “it” of (1) in the abstract is described as the following phenomenon: … exhibit a decrease in representation rank, often correlated with an inability to continue learning or a collapse in performance. **The phenomenon we refer to in the quoted text is “representation collapse”,** which can be measured by tracking various metrics we discussed in our paper. **This is what we claim is overlooked. The inability to continue learning and collapse in performance are correlations to it, which can also correlate with many other undesired learning dynamics.** However, we acknowledge that when combined with the “thought capable of training indefinitely,” **readers may interpret that all parts of the correlation have been “overlooked in on-policy policy optimization.”** **To avoid any misinterpretation and better contextualize our work, we will, as the AC suggests, reformulate this statement** and other statements referring to previous investigations of “performance collapse” and “inability to continue learning in policy optimization” to explicitly build on Dohare et al.’s work (Dohare et al. 2021, 2023a for inability to continue learning in non-stationary environments, and Dohare et al. 2023b for collapse in stationary environments). **We will position our work as motivated to understand this previously observed collapse and inability to continue learning from the lens of representation dynamics used in value-based methods.** As the AC suggests, rather than weakening our work, **it will make it connect different previous works, providing a better overview of it and strengthening the “understanding” narrative of the submission.** > Regarding the measure of loss of plasticity ... We agree. The community has still to tease out the different notions of “plasticity loss,” and this has been raised by a reviewer in the discussions. We agree that picking one definition (random label fitting) and formulating strong claims of novelty with respect to it (2) can seem to undermine the claim with another but relating definition (Abbas et al. 2023) of the same name. **We suggest making our statement precisely about random label fitting and elaborate on the similarities between the two notions in our related work section, highlighting that Dohare et al. (2021, 2023a) also show a loss of plasticity in policy optimization from Abbas et al. (2023)’s perspective.** Again, this will give more context to our readers and avoid misunderstandings. > It is expected that newer works will be more thorough than previous ones, which I have also acknowledged for your work. … The weaknesses of this prior work are not a valid cause … We thank the AC for highlighting the thoroughness of our work, as stated at the beginning of this reply and in the last sentence quoted, we are happy to refer to all previous work that can better contextualize our work and grow the understanding of the plasticity community, and help the trust-region subcommunity that we bring with this work, navigate the previous works in plasticity. We are glad the rest of the AC concerns have been addressed. Below, we summarize the major points that we will rephrase and help to reposition our work. - **Adding an extra page of related work,** including **Abbas et al. (2023), Sokar et al. (2023), and Kumar et al. (2023) for broader discussions of plasticity** as discussed in our previous reply, **the line of work by Dohare et al.** as discussed in this reply, and **those from discussion with reviewers like PPG (Cobbe et al 2020), DR3 (Kumar et al. 2022), and RD (Ma et al. 2024).** - **Rephrasing the claims about overlooked collapse and inability to continue training in policy optimization: it is rather the connection to representation dynamics that has not been investigated.** - **Rephrasing the contribution of PFO refocusing its contribution on strengthening our analysis rather than emphasizing raw performance.** - **Clarify the scope of the connection between representation collapse and trust region collapse highlighting that although the connection mainly holds in the collapse regime, mitigating representation collapse should be throughout training.** We thank the AC once again for the productive feedback, and we look forward to continuing this dialogue and refining our manuscript to achieve the best outcome for the readers.
Thank you very much for further clarification, and no worries at all. I understand very well how intense this short period of rebuttal can be. I also realize that we are perhaps coming from slightly different perspectives. I greatly appreciate the authors' effort to help these perspectives converge. This effort, if reflected in the paper, will benefit the community and strengthen the contribution by situating it well within the existing literature. This will perhaps be the most scholarly outcome of our discussion. My inquiries have been fulfilled. I do not have any further questions at this point. Thanks again for the response.
We are grateful to the AC for their understanding. We thank all the reviewers and the AC for their valuable suggestions and active engagement during the discussion phase. We commit to reflecting all these efforts and the converged perspective in the paper.
Decision
Accept (poster)