Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks

Summary

This paper explores implementation details from Dreamer-v3 in the context of PPO to examine the generality of these implementation tricks to DeepRL. They identify 5 unique implementation details from Dreamer-v3 and do a thorough examination of its effects on both continuous and discrete control environments (i.e. Deepmind Control Suite and Atari respectively). They offer solid evidence for and against certain implementation tricks in the context of PPO and find that environments with poorly normalized, unbounded rewards were where these identified tricks helped most.

Strengths

### Originality --- This paper follows a line of literature that investigates implementation tricks for DeepRL algorithms. I believe its valuable and necessary research given the large amount of implementation tricks in DeepRL. It was interesting and original for the authors to specifically study tricks from an on-policy model-based DeepRL algorithm to the model-free DeepRL algorithm. ### Quality and Clarity --- I believe they did a very thorough evaluation on very large benchmark domains. It is impressive that they were able to do both all of Atari and DMC control. The writing was clear and I believe the authors were very clear about what they were investigating and presenting from the very beginning. ### Significance --- The per-environment learning curves from the ablations and experiments for both DMC and Atari (in the supplementary) is a significant contribution that democratizes this information to many researchers who do not have great compute access. Although there wasn't a strong clear implementation trick that drastically improved performance, across DMC and Atari, I commend the authors for reporting all both what worked well and not well which is very useful for practitioners moving forward.

Weaknesses

* It would be great to see hyperparameter ranges tested for each implementation detail. Perhaps this is already done through the code-release but would be nice to have in writing with some insight from the authors * Perhaps diving a bit deeper into subcategories of tasks within the benchmark suites and doing some analysis there. For example, some environments are long horizon, sparse/dense reward, small/large action space, etc. Given all of the per-environment data already documented and available, it would be great if the authors could provide some insight on why some implementation tricks worked/didn't work based upon features of the task. * Some more insight on which implementation tricks were more tailored towards World Model learning and specifically Dreamer's actor-critic learning. It would be interesting to implement these implementation tricks with A2C rather than PPO to test whether these tricks were indeed for the world model or for the RL optimization. (I would suggest PPO for Dreamer-V3 but that also sounds very non-trivial).

Questions

My questions align with the points raised in the "weaknesses" section: * Other than the reward clipping ranges, did you find these implementation tricks very sensitive to hyperparameter? * Where there other insights about types of environments within a benchmark suite and its relationship to the implementation tricks? * What was the main rationale for choosing PPO over A2C when applying Dreamerv3 tricks? How many of implementation details do you think were specific to World Model training?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, they have adequately addressed the limitations.

Summary

This paper aims to assess the generality of DreamerV3's stabilization techniques by adapting them to the widely-used model-free algorithm PPO. The authors then proceed to conduct a thorough ablation study across Atari games and the DeepMind Control Suite to evaluate their effectiveness.

Strengths

- The paper is well written and easy to follow. - The obtained findings are intriguing and highlight the need for a more comprehensive ablation study of these tricks in the model-based context of DreamerV3. - Demonstrating a negative result poses a challenge as poor outcomes can be attributed to implementation errors. In this regard, the authors have done a rather good job: - They chose PPO, an extensively studied algorithm, and conducted experiments based on the widely-accepted CleanRL implementation. - They perform add-one and drop-one ablations across the complete set of DMC environments and Atari games, while adhering to the recommended Atari wrappers. Multiple seeds were used, and the results were reported using the recommended metrics from RLiable. - They rely on the open source implementation of DreamerV3 and communicated with DreamerV3’s authors to implement theses tricks. The few differences with DreamerV3 are well-explained and justified. - The code is open source, which enables the community to further validate their study and potentially build upon it.

Weaknesses

- Originality: The sole innovation of the paper lies in the adaptation of these tricks to PPO, which, on its own, may not be considered a significant contribution. - Impact: While the experiments demonstrate technical rigor and the publication of negative results is valuable, the overall impact of the paper appears to be relatively limited. - Minor writing details: - There are instances in the bibliography where names with accents have been incorrectly rendered. - Some references should probably be included (L120, relevant papers on Twohot encoding for distributional RL; L148, explicit the previous work, etc).

Questions

The paper is coherent and does not present any confusion, so I don't have any specific questions. I am open to discussing the impact further, as it is undoubtedly challenging to evaluate in advance.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

2 fair

Limitations

The authors discussed the scientific limitations of their work. The societal impact is not discussed, but not very relevant in this case I believe.

Summary

This paper empirically tests the effectiveness of the DreamerV3 tricks on PPO. The authors conduct a comprehensive study on each trick, especially, extensive ablation studies, including add-one and drop-one ablations. Based on these experiments, the authors give some insights into the effect of the DreamerV3 tricks.

Strengths

- Extensive empirical study on each of the DreamerV3 tricks; - Experiments are conducted on various environments, including the environments from Atari 100M and DeepMind Control Suite, and using the rliable metrics to present robust performance comparisons; - Based on the experiments, the authors give some insights.

Weaknesses

- The largest weakness of this paper is that they test the effectiveness of DreamerV3 tricks on only a single RL algorithm, i.e., PPO. Given the diversity of RL algorithms, the conclusions derived on PPO, at a large possibility, cannot generalize to other RL algorithms. Indeed, the observations given by this paper are valuable for the community as a technical report, but I don't think the contributions of this paper are enough for a NIPS paper. - The authors claim that "our method performs comparably to PPO on Atari games with reward clipping and significantly outperforms PPO without reward clipping". However, it can be seen from Figure 2 (e) and (f) that, the algorithms with reward clipping generally outperform the ones without reward clipping, which suggests that reward clipping is a useful technique. Therefore, I don't think it is valuable to achieve a SOTA performance under the condition without reward clipping. The vanilla PPO with reward clipping is still the SOTA method on Atari. - Furthermore, on DeepMind Control Suite, the vanilla PPO is also the SOTA algorithm. So it can be concluded that almost all the DreamerV3 tricks do not have a positive effect on PPO, leading the conclusions of this paper to be much less meaningful.

Questions

- Do the observations of this paper applicable to other RL algorithms?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

The authors investigate the architectural and algorithmic improvements proposed in the DreamerV3 paper, but applied to PPO. The tricks are each compared on the DMC and Atari suites.

Strengths

Thorough benchmark papers are a valuable asset to the RL community due to the difficulty in accurately comparing them and typically large error margins. The paper is well-written and easy to read, and cites related work DreamerV3 and PPO are both very relevant.

Weaknesses

The conclusions of the experiments were not really surprising. While I think a benchmark / reproducibility paper like this still has value, I have some concerns with that: * First, it misses the obvious opportunity to first reproduce the DreamerV3 tricks on the Dreamer RL algorithm itself. It would not have to be every benchmark, but getting similar results on e.g. Atari would have helped support the claim that these tricks were correctly implemented, as well as reproduced a recent method of interest. * Second, why PPO? Is it even a good target for these tricks? According to the introduction, a big motivation for these were to enhance stability of actor-critic approaches, which typically struggle with stability. While PPO has some similarity to actor-critic approaches due to taking multiple steps and value function/GAE estimation, it was my impression that it behaved more like a policy gradient approach (which are typically more stable but on-policy and less efficient in theory). As such, the DreamerV3 tricks might have been a more natural fit for e.g., SAC. --- After rebuttal: I think the authors gave adequate answers. Although they did not remedy all weaknesses, empirical studies like this are very important for RL research and it would be prohibitively expensive to test more algorithms. As such I think it would be useful for the community even though it has some limitations.

Questions

During the course of writing this paper, did you try or manage to reproduce any results of DreamerV3? Why the choice of PPO instead of e.g., SAC?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Seems clear.

Thank you for the rebuttal.

I thank the authors for the reply. I do think benchmark papers important, even if I would have preferred to validate their implementation against some of the published results of Dreamerv3 first before extending the tricks to method that is quite different (model-free PPO). I will increase my score.

Thanks for the rebuttal - Updating to 7

I thank the authors for their response. I concur with reviewer 75kB's suggestion that including a comparison with DreamerV3's results would serve as an effective sanity check. However, I understand that this could be challenging due to the different codebases used for PPO and DreamerV3. Since implementation details are crucial in deep RL, I believe investigating transferability and reproducibility is important and should be encouraged. I'm changing my score from 6 to 7.

Rebuttal response

I appreciate your detailed response, and I acknowledge that PPO is a prominent algorithm that has been thoroughly investigated. However, I firmly believe that exploring the application of these tricks to a variety of algorithms is crucial. As it's common knowledge that the same tricks may yield disparate results when applied to different algorithms, limiting the study to a single algorithm might constrict the paper's overall contribution. I have also taken into consideration the perspectives of other reviewers and understand that the extensive experiments conducted in this paper can be seen as valuable to the research community. Nevertheless, I'm concerned that leveraging a significant amount of computational resources to execute an analysis that might not be highly innovative, and arriving at conclusions that might not be particularly surprising, does not seem fitting for acceptance at a prestigious conference like NIPS. Moreover, from a practical standpoint, the experimental results don't provide enough guidance for a user to choose anything other than the original PPO with reward clipping, neglecting the dreamerV3 tricks. Finally, it's important to note that some of my concerns were left unaddressed in the rebuttal, such as those related to the state-of-the-art performance. Thus, I must stand by my initial rating. I encourage the authors to consider expanding the scope of the paper to include more algorithms and strive for conclusions that can add more substantial value to the field.