We focus on the task of approximating the optimal value function in deep reinforcement learning. This iterative process is comprised of solving a sequence of optimization problems where the loss function changes per iteration. The common approach to solving this sequence of problems is to employ modern variants of the stochastic gradient descent algorithm such as Adam. These optimizers maintain their own internal parameters such as estimates of the first-order and the second-order moments of the gradient, and update them over time. Therefore, information obtained in previous iterations is used to solve the optimization problem in the current iteration. We demonstrate that this can contaminate the moment estimates because the optimization landscape can change arbitrarily from one iteration to the next one. To hedge against this negative effect, a simple idea is to reset the internal parameters of the optimizer when starting a new iteration. We empirically investigate this resetting idea by employing various optimizers in conjunction with the Rainbow algorithm. We demonstrate that this simple modification significantly improves the performance of deep RL on the Atari benchmark.
Paper
References (53)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The authors argue that Adam's internal parameters should be reset with each iteration. The authors demonstrate the effectiveness of this approach in the Atari domain.
Strengths
- Results are convincing for Rainbow. - Novelty is very low, but potential impact is high, if the result generalizes, there is little reason not to use this method in every DQN-style RL algorithm.
Weaknesses
- As mentioned by the authors, novelty is low compared to Bengio et al. - Results are only shown on Rainbow and do not appear to work for SAC (with reason) -- but does raise the question if the method is effective for other RL methods. - There is limited insight. Can the authors show that initializing Adam's parameters with 0 is better than using the parameters from the previous iteration in a more concrete way? Such as examining the behavior of the actual values. The fact that not resetting is seemingly better at low values of K suggests that not resetting can provide a reasonable initialization for the parameters of Adam. Minor - The y-axis is unlabelled in several figures.
Questions
As mentioned in weaknesses: - Does this result generalize to other methods besides Rainbow? Such as DQN or more modern deep RL methods. - Can the authors show that initializing Adam's parameters with 0 is better than using the parameters from the previous iteration in a more concrete way? Such as examining the behavior of the actual values.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
No concerns.
Summary
The paper addresses the issue of using modern optimizers, such as Adam, which maintain internal parameters that are updated over time, potentially contaminating the optimization process. To mitigate this effect, the paper proposes a simple strategy of resetting the internal parameters of the optimizer at the start of each iteration. Empirical investigations using different optimizers and the Rainbow algorithm show that this modification enhances the performance of deep reinforcement learning on the Atari benchmark.
Strengths
### Writing The authors effectively communicate their ideas and concepts, ensuring clarity and coherence throughout the paper. The logical structure and well-reasoned arguments contribute to the overall quality of the essay. The article excels in providing the reader with a clear understanding of the problem's context and significance. By effectively conveying the goals and challenges of the study, the authors enhance the reader's comprehension of the subsequent experiments. Overall, the writing is of high quality, facilitating a smooth and engaging reading experience. ### Method The paper presents an easy-to-use approach by introducing a method that is not only easy to implement, but also easy to apply, which enhances the potential adoption and practicality of the proposed approach.This user-friendly feature makes the method highly accessible and beneficial to researchers and practitioners in various fields. The used code bases and hyperparameters are provided, allowing the results to be reproduced.
Weaknesses
While I appreciate the proposed method's ease of use, I believe that the authors could have conducted a more comprehensive and statistically rigorous analysis of their approach, considering its simplicity. One notable limitation of the paper is the absence of confidence interval plots and statistical analysis, which could have been derived from [1], to enhance the clarity and precision of the findings. Incorporating these elements would have allowed readers to better understand the level of uncertainty associated with the reported results, thus bolstering the overall robustness of the study. Furthermore, the authors only rely on a single seed for the initial analysis, without providing a compelling rationale for this choice. Although using a single seed can streamline the experimental process, it diminishes the validity of the findings by disregarding potential result variations arising from multiple seeds. A more thorough explanation or a comparison of outcomes based on different seeds would have added value to the introduction, ensuring a more comprehensive analysis. I appreciate that the authors included continuous control tasks in their study; however, these tasks are not thoroughly explored. While the authors provide hypotheses to explain the unexpected results, a deeper analysis would have been expected. In line 293, the authors reference a follow-up paper on resetting approaches but fail to cite the original work [2], which states in the section "What and how to reset" that resetting the optimizer has almost no significant impact due to quick updates of the moments. This contradicts the findings of this work. Which brings me to the conclusion that I believe that the paper shows promise and the authors have taken a positive direction. However, in its current form, the paper falls short of being acceptable. It is essential to include comparisons to other baselines, such as [2], to provide a more thorough understanding of the opportunities and limitations, and to gain a clearer understanding of the internal effects in order to explain the aforementioned points. ### Minor - The protocol for the random resets is not easy to understand and should be specified more clearly [1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320. [2] Nikishin, Evgenii, et al. "The primacy bias in deep reinforcement learning." International Conference on Machine Learning. PMLR, 2022.
Questions
- How does this method compare to other resetting approaches in terms of effectiveness? - What is the level of statistical significance observed in the results? - Why is resetting not effective for continuous control tasks? - Are there any experiments demonstrating the impact of contamination on the tasks discussed in this paper? - What are the consequences of reducing the frequency of optimizer resets beyond K=8000? - How can an optimal value for $K$ be determined? - Which ADAM/optimizer parameters are relevant when performing resets, i.e. have an effect when reset? - Are the observed effects still present when modifying the ADAM/optimizer hyperparameters? - Do different loss functions used in various DQN versions (e.g., MSE, Huber, Quantile) exhibit similar behaviors?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The authors have made some effort to address the limitations; however, it is crucial for them to conduct a more comprehensive investigation into these limitations, as mentioned in the "Weakness" section.
Summary
This paper questions the standard use of Adam-type optimizers in deep RL. The paper argues that solution methods in deep RL are best thought of as solving a sequence of optimization problems. And that the standard use of optimizers leads to "contamination" of the optimizer's internal parameters. The paper then proposes to reset the optimizer's internal parameters to fix this "contamination." Finally, the experiments on Atari show that resetting the optimizer's internal parameters leads to significant performance improvement.
Strengths
The main strength of the paper is that the key idea of the paper, resetting optimizer parameters at the beginning of each iteration, is simple and effective. I liked the general theme of the paper, i.e., we need to understand better the tools we borrow from other fields. The paper is well-written and easy to understand, making it accessible to a wide audience. The experiments in section 4.3 are performed on 55 Atarti games with ten seeds each, which might mean the results are statistically significant. Resetting seems to be beneficial for multiple optimizers like RMSprop, Adam and Rectified Adam.
Weaknesses
The paper has two major weaknesses: 1. The paper claims at many points that a "contamination" effect plagues RL (for example, lines 107-109) and that many updates are wasted to unlearn the effects of the previous iteration. However, the paper does not describe what exactly this "contamination" means, and neither does it show the presence of any "contamination." All the paper shows is that there is a performance boost when we reset the optimizer's internal parameters. This performance boost is not direct evidence of contamination from iteration to iteration. However, this weakness can be easily overcome. The paper first needs to contain a definition of "contamination," maybe the authors mean that the internal parameters($m$ and $v$) are too far away from their true values at the beginning of each iteration. One way to measure this difference could be to measure the cosine similarity between the current value of $m$ and the true value of $m$. The true value can be measured by taking the gradient of all the samples in the buffer and taking steps using that gradient. A large difference between the true and current value of $m$ would mean contamination. The paper also suggests that this contamination is particularly large at the beginning of each iteration compared to a random time in the learning process. Again, this can be easily shown by showing that the difference in true $m$ and current $m$ is larger at the beginning of the iteration compared to any random time in the learning process. 2. None of the results in the paper except the ones in section 4.3 are statistically significant. The paper only shows results for a single random seed. I should note that the authors are aware of this weakness (line 138). The best way to look at the current experiments in Sections 4.1 and 4.2 is that they are used to tune hyper-parameters for the experiments in Section 4.3. The authors might be limited in their computational resources, but in that case, it is better to present statistically significant results in smaller environments like MinAtar[1] than unreplicable results in a big environment. Other than these two main problems, there are a few other minor issues in the paper. 1. The update equations for Adam in Section 3 are wrong. Instead of using $m$ and $v$ for the final update, new variables $\hat{m}$ and $\hat{v}$ are used. See to the original Adam paper for the correct equations. 2. Line 176 says that $K=1$ corresponds to vanilla gradient descent. But, that is not true. For $K=1$, the update is similar to the Rprop optimizer, not SGD. For $K=1$, the update only takes into account the sign of the partial derivative but not its magnitude. 3. The value of $K$ is not properly tuned. In Figure 4, the difference between $K=1000$ and $K=500$ is insignificant. So, the optimal value of $K$ could be smaller. I suggest the authors also try smaller values of K, like 250, 125, etc. I like the ideas presented in the paper. However, I can not recommend accepting the paper in its current form in light of these weaknesses. [1] Young, K., & Tian, T. (2019). Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176.
Questions
What would be a good definition of "contamination"?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
1 poor
Presentation
3 good
Contribution
3 good
Limitations
No confidence intervals are reported for any experiment in the paper. I recommend the authors report the 95% bootstrapped confidence intervals for their results. [2] and [3] provide good guidelines for properly reporting experimental results in deep RL. [2] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320. [3] Patterson, A., Neumann, S., White, M., & White, A. (2023). Empirical Design in Reinforcement Learning. arXiv preprint arXiv:2304.01315. EDIT: I have updated my score based on the new results provided by the authors.
Summary
The paper studies optimization in value-based deep reinforcement learning. The key insight is that when using target networks for action-value function training, changes in the target parameters yield a change in the optimization problem the online parameters are solving. Because of that, the authors argue that preserving the adaptive optimizer statistics (e.g. of Adam) might or might not be desirable. The paper then studies the effect of resetting the optimizer state after (hard) target updates mostly using the Rainbow algorithm on Atari games as a testbed yielding a slight positive aggregate improvement.
Strengths
The main strength of the paper is the simplicity of the contribution; the paper is well-written and easy to follow, and the method is motivated and described well. The experimental protocol is solid: it uses the full set of 55 Atari games and a standard Rainbow implementation.
Weaknesses
The main weakness of the paper is the mixed empirical results. Granted, the median human-normalized performance improves from ~1.75 to ~2.25, however, per-game effects from resetting the optimizer are highly heterogeneous, yielding performance deterioration in ~14 environments. The soundness of the paper could have been higher if, at least, an explanation (supported by evidence) for the negative effects was given.
Questions
Target network parameter updates indeed change the loss landscape that the online parameters are navigating. In addition to that, updating the replay buffer changes the distribution of inputs and hence the optimization problem for online parameters. Do you have ideas on how an optimizer could be changed to adapt to the input shifts?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
“We hypothesize that this can contaminate the internal parameters of the employed optimizer in situations where the optimization landscape of the previous iterations is quite different from the current iteration.” (L9) The reviewer didn’t find an empirical verification of this assumption. Many deep RL algorithms use moving average target updates after each step instead of periodic hard updates. The authors demonstrate preliminary evidence that in soft actor-critic that uses such a practice, the optimizer resets do not improve the performance. Having said that, the reviewer appreciates the transparency about the negative results. Again, one of the limitations is that in some environments resetting the optimizer yields negative results. It implies that a better alternative could be triggering the optimizer reset using a criterion (e.g. based on a measure of the loss landscape change / by performing a lookahead and assessing whether the reset was helpful)
Thank you for your answer
For some reason I misread the citation [47] and thought you were referring to this follow-up paper: D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P. L., Bellemare, M. G., & Courville, A. (2023). Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. In The Eleventh International Conference on Learning Representations. Please apologize and disregard this comment. I also appreciate the clarification of why Nikishin et al. come to a different conclusion about the optimizer reset, which at the same time gives some intuition about the results for the continuous control setting. > What are the consequences of reducing the frequency of optimizer resets beyond ? This question was probably badly worded. But as guessed correctly, I meant an increase of $K$, which implies a decrease in the frequency of optimizer resets. > We would like to remind that we did not introduce this parameter. Therefore, we do not think it is incumbent upon us to answer this question in our paper as this is simply beyond the scope of our work. Also, not knowing the optimal is not at all reflective of a weakness of the resetting strategy, because we also need to tune when not resetting the optimizer. That said, this is a good empirical question, one that we would be super excited to focus on after publishing this paper. Thank you for the answer. With this question, I did not mean to imply that choosing an appropriate $K$ is a weakness of the proposed method, since, as you mentioned, it has to be done regardless. I was just interested in how the added purpose of $K$ for the optimizer resets affects the hyperparameter selection of $K$. > Are the observed effects still present when modifying the ADAM/optimizer hyperparameters? This question was more related to changing the hyperparameters of the respective optimizers than to the optimizer itself, e.g. the betas for ADAM. Granted, the default values of 0.9, 0.999 in the case of ADAM are usually the best settings. I would just be curious if you have tried other values besides the default and possibly seen a difference. Other than that, I appreciate the effort that went into the rebuttal answers as well as the additional results and seeds. They definitely further support the proposed approach. I will take this into account when discussing the final recommendation with the other reviewers and AC.
Changing Adam's Hyperparameters
In terms of the Adam optimizer, we used the default values of Adam because deviating from the default values could have been interpreted as our trying too hard to show the effectiveness of resetting. In particular, other than $\beta_1$ and $\beta_2$, we also did not change the default step-size value of Adam used in baseline RL algorithms to keep the comparison as grounded as possible relative to the literature. That said, we totally agree that it would be interesting to look at the effects of these hyper-parameters on the validity of our conclusions. For example, the moment estimates may become less or more stale depending on the choices of $\beta_1$ and $\beta_2$, and this may ultimately affect the gap between the resetting and non-resetting agents. However, in light of our lack of ability to edit the rebuttal pdf, it seems impossible for us to carry this experiment, so unfortunately we can only promise to add this experiment in our final paper. We are delighted to see the reviewer's engagement with our rebuttal, as well as the reviewer's stating that they will take into account the added seeds, the contamination-measurement experiment, and the IQN/DQN results in their final recommendation. However, we also worry that our average score has remained low, and that this can negatively affect our chances in the discussion. While we appreciate if the new considerations could also be reflected in the final score, we nevertheless really respect the reviewer's engagement with our rebuttal.
Thank you for additional details.
Thank you for providing some more insight on that matter. I have updated the score of the original proposal.
Thanks
We appreciate the reviewer for their reconsideration and for supporting our paper. Please let us know if there was any new or lingering question.
Thanks for the response and the additional results. At this time I don’t intend on increasing my score but continue to favor acceptance of the paper.
I thank the authors for adding more seeds and experiments to measure contamination. The results with more seeds are largely consistent with the results in the original manuscript. The new results alleviate most of my concerns about the statistical significance of the results in the paper. The new contamination results are striking. The cosine similarity effectively reduces to zero after about 20M frames. These results seem to support the claim that there is contamination. However, they are not fully satisfactory. The authors approximate the true gradient from the first mini-batch in the next iteration. But I don't know if the first sample gradient is a good approximation of the true gradient. The true value is the average over a sample of all the data in the buffer (1M?), but the mini-batch is only 128(?). I realize that computing the gradient over the full buffer will be computationally expensive. Still, we can reach a middle ground by measuring the average gradient for 10,000 or 50,000 transitions in the buffer. A few questions to the authors: - Why does the cosine similarity decrease over time? And why was it exactly 1 in the beginning? If there is contamination from iteration to iteration, shouldn't the cosine similarity always be zero? Why do you think that is the case? - What do the error bars report? Is it the 95% bootstrapped confidence interval or IQM or something else? - Did you get to do more runs for rainbow without resetting (Figure 4) for K=0? I'm very curious to see if Rainbow with K=0 and K=8000 perform the same. If that is the case, it would mean that we can effectively remove the target network from Rainbow. As a side note, the authors do not describe the hyper-parameters used for all the algorithms in the paper. The authors should add a section in the appendix describing all the hyper-parameters for all the algorithms used in the paper. I light of the authors adding more runs to make their results statistically significant and adding a measure for contamination, I have changed my score to accept the paper.
Answering Additional Questions
We thank the reviewer for their continued engagement with our rebuttal, and for their neat suggestion about the contamination experiment which truly strengthened our paper. - The true value is the average over a sample of all the data in the buffer (1M?), but the mini-batch is only 128(?). You are correct in saying that the gradient is noisy because it is only estimated over a minibatch of size 64, however, we think the low cosine similarity is not due to the fact that we use stochastic gradients and it is due to the fact that the optimization landscape actually changes from one iteration to the next one. That said, to further address the reviewer's concern, we just started another experiment where we change the batch size from 64 to 2048, and will add this to the paper. However, please keep in mind that even if a much larger batch size can increase cosine similarity, in practice we do not want to use a larger batch size anyways due to 1) computational considerations and 2) that in deep learning very large batch sizes actually performs poorly due to their converging into sharp minimizers with poor generalization gap (Keskar et al). - Why does the cosine similarity decrease over time? And why was it exactly 1 in the beginning? If there is contamination from iteration to iteration, shouldn't the cosine similarity always be zero? Why do you think that is the case? This is a very good question. We think the early cosine similarity we observe is due to the fact the network is initialized completely randomly, and so the optimizer roughly moves the weights in the same direction irregardless. However, this is a very quick and transient phase and the cosine similarity quickly vanishes to zero. - What do the error bars report? Correct, it is the 95\% bootstrapped confidence intervals.
Answering Additional Questions (2)
- Did you get to do more runs for rainbow without resetting (Figure 4) for K=0? I'm very curious to see if Rainbow with K=0 and K=8000 perform the same. If that is the case, it would mean that we can effectively remove the target network from Rainbow. Yes we did, and we still observed that Rainbow (without resetting) performs equally well with the extreme value of $K=1$. In fact, we see that the performance of Rainbow remains flat with respect to $K$. In our experience, only when using extremely large values of $K$ (bigger than 20000) we get a negative impact on Rainbow's performance. We concede that this is a somewhat surprising result, but note that in the literature we have seen examples where a much smaller value of the original $K=8000$ is used for Rainbow. For example, Agent 57 used $K=1500$ (Badia et al). Overall, akin to the reviewer, we are also really surprised that Rainbow works equally well with $K=1$ and that the choice of $K=8000$ seems to just be a legacy from previous papers such as DQN. The reviewer correctly identifies that by removing the frozen target network in Rainbow without resetting we will not affect the performance negatively. - As a side note, the authors do not describe the hyper-parameters used for all the algorithms We will be sure to add a section on hyper-parameters. Note, however, that we have used the standard hyper-parameters from the Dopamine implementation (Castro et al). The main hyper-parameter we worked with was $K$, which we have reported in the paper. That said we agree that having a hyper-parameter section would make the paper self-contained, and we will add this to the paper. Again, thanks for your constructive feedback. It really helped improve our paper. Please let us know if any question has lingered at this point. Keskar et al, ''On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima", 2017 Badia et al, ''Agent57: Outperforming the Atari Human Benchmark", 2020 Castro et al, ''Dopamine: A research framework for deep reinforcement learning", 2018
Dear Authors, thank you for quickly answering my questions. I want to follow up on a couple of points. I want to clarify that when I suggest using a larger sample (say 10k) to estimate the true gradient, I'm not suggesting running the entire experiment with a 10k mini-batch size. I'm suggesting using a mini-batch of size 64 and, at transitions, passing a mini-batch of 10k through the network to get an estimate of the true gradient but do not use that estimate to update the weights as that will change the underlying algorithm. The result showing that removing the target network from Rainbow does not hurt the performance is very exciting. I think the authors should spend some paragraphs in the paper and maybe a couple of lines in the conclusion pointing out this result. I realize this is not the paper's main message, but this result will be an important data point for future research.
Rainbow's performance with K=1
We definitely agree that expanding the discussion about Rainbow without resetting and $K=1$ can really be helpful for future research, especially the kind of research that aims to better understand the frozen target network, or to potentially even remove it altogether! In particular, we also have found that, unlike Rainbow, DQN really struggles with small values of $K$ and works terribly with $K=1$. While this explains why the original $K=8000$ choice was "grandfathered in" from DQN to Rainbow, it seems to us that one (or more) of the components of Rainbow is obviating the need to use larger $K$. We don't have a definitive answer yet, and so this should be further investigated in future. Also, like you said, we did not focus on this result too much in the submission because it was only tangentially related to the main message of our paper. We really hope the reviewer would reinforce their advocacy of our paper so we could present the surprising Rainbow($K=1$) result in the final camera ready. As for larger batch size, yes, we agree and understood the reviewer's intent. Please stay tuned and we will get back to you on the final outcome of the experiment. It seems like we can not edit the pdf and also cannot provide outside links, but we will state what we observe here as a comment.
We thank the reviewer for their continued engagement and for advocating our work. Per your request, we ran the experiment with a larger batch size and measured the cosine similarity after target updates on all 12 games from our ablation study. We show results averaged over all games for the original batch size (64) and the much larger batch size (2048). | number of frames | cosine similarity (64) | cosine similarity (2048) | | ------ | ------ | ---- | | 0 M | 0.982 $\pm$ 0.1374 | 0.9884 $\pm$ 0.0802 | | 10 M | 0.1034 $\pm$ 0.2808 | 0.1872 $\pm$ 0.2652 | | 20 M |0.0239 $\pm$ 0.1052 | 0.0259 $\pm$ 0.0655 | | 30 M | -0.0085 $\pm$ 0.0842 |0.04 $\pm$ 0.0683 | | 40 M | -0.0039 $\pm$ 0.103 | 0.0122 $\pm$ 0.0746 | | 50 M | -0.0132 $\pm$ 0.0684 | -0.0154 $\pm$ 0.0871 | | 60 M | -0.0121 $\pm$ 0.0751 |0.0339 $\pm$ 0.0658 | Overall, under both batch sizes we see a similar trend where the cosine similarity goes to zero. We also conducted a related experiment where we measured the cosine similarity right before target updates. Please refer to our last comment to reviewer Ks5C.
Thanks for the response. I concur that it is encouraging to see aggregate improvements from such a simple, hyper-parameter-free technique. What I was mostly pointing at was a lack of explanation for why some of the games experience worse performance from applying optimizer resets. The cosine similarity experiment is indeed interesting. However, it seems like it is unable to explain the effects of resetting the optimizer. For instance, based on Figure 8, DemonAttack is an example of a game with positive effects from opt-reset and Breakout is an example with neutral effects. For both of these games the cosine similarity between the Adam 1st moment and the gradient after the target update behaves almost identically, suggesting that low cosine similarity isn't explaining the opt-reset effects. Additionally, for a complete picture, we need a similar plot _with cosine similarity right before the target update_. Thanks for also validating opt-reset on DQN and IQN — was the aggregate score calculated on all 55 games?
Further Clarifications
Thanks for your carefully reading the paper and the rebuttal. The reviewer makes a good observation. We acknowledge that measuring the cosine similarity alone cannot fully predict the utility of resetting. Note that in doing this experiment, our intention was merely to show that a contamination effect does exist, but even with different amounts of contamination it is still possible for the non-resetting agent to occasionally outperform the resetting agent due to the presence of other confounding factors in deep RL. To take a step back and focus on the bigger picture, in this work we presented the view that mainstream RL algorithms should better be thought of as solving for a sequence of optimization problems. This stands in contrast to the view that treats RL algorithms as solving for a single optimization problem. Fully adopting this view means that the most natural thing to do is to reset the RL optimizer in situations where the optimizer is moment-based. This surprisingly simple technique was somehow absent in the literature, and while we do not have all answers about when it works best, we observe that over many settings of RL algorithms and optimizers it is quite beneficial to reset. We believe that a paper that takes a first step to present this optimization view and to propose the resetting strategy can serve the RL literature well. - Thanks for also validating opt-reset on DQN and IQN — was the aggregate score calculated on all 55 games? To answer your question, we did the new 1) DQN with Adam and 2) IQN with Adam experiments only on 12 games. These are the same 12 games from section 4.1 used in the original submission to show the benefits of resetting on 3) Rainbow with Adam, 4) Rainbow Pro with Adam, 5) Rainbow with Rectified Adam, and 6) Rainbow with RMSProp. So taken together, we have 6 examples of resetting's improvement on the same set of 12 games. We chose these 12 games completely randomly when we began the project. Our experiments on the full 55 games are only on 1) Rainbow with Adam and 2) Rainbow with Rectified Adam.
Could you give the plot with cosine similarities before the target updates? I'd also like to put on your radar the paper by Dabney et al. [1] that treats RL as a sequence of prediction problems. [1] Dabney, Will, André Barreto, Mark Rowland, Robert Dadashi, John Quan, Marc G. Bellemare, and David Silver. "The value-improvement path: Towards better representations for reinforcement learning." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 8, pp. 7160-7168. 2021.
Yes, we are working to report this experiment. We submitted it a few hours ago and because Rainbow takes roughly a day to complete 25 Million frames we expect to have a meaningful update in 2 or 3 days. Also notice that we cannot edit the pdf, and cannot provide links per the AC guidance, so we have to be creative in terms of how we report this result. Pardon us in advance for the inconvenience. - I'd also like to put on your radar the paper by Dabney et al. [1] that treats RL as a sequence of prediction problems. Yes, we are aware of the innovative work of Dabney et al. and we thankfully did not fail to cite it in the original submission (reference [30]). We also discuss it in the paper (lines 276 to 280). This paper is presenting a similar view that the RL agent should best be thought of as one that solves a sequence of value-prediction problems rather than a single one. Adopting this view led them to design their representation learner in a sequence-aware manner. More specifically, they argue that the representation learning aspect of the RL agent should aim to learn a representation that is well-equipped to predict, not just the current value function, but the sequence of value functions observed so far. They show that this will help learn better representations in the context of Atari. While Dabney et al. looks at this through the lens of representation learning, we were interested to explore the ramifications of this view through the lens of optimization. This led us to resetting which is the most natural thing to do having adopted the sequence view. We believe that adopting the sequence view can have interesting ramifications beyond optimization (our paper) and representation learning (Dabney et al.) as well. Even in the context of optimization, one can think of other ramifications, for example exploring different ways of initializing the optimization problem at each iteration in light of the sequence view, or looking at the previously solved optimization problems to better guide the agent in solving the current optimization problem. Our belief is that these are really exciting but under-explored questions, and we hope that by allowing us the opportunity to share our results with the community we can instigate further work in this direction. We are happy to add this discussion to the paper to better situate us relative to Dabney et al.
We thank the reviewer again for their engagement. Per your question, we measured cosine similarities after and before target network updates. We did so on the same 12 games we ran all our ablation studies on. See results below: In the first table we report the average cosine similarity across learning per each game. | Game | average cosine similarity (after) | average cosine similarity (before) | | ------ | ------ | ---- | | Amidar |0.0036$\pm$ 0.0021 | 0.4008$\pm$ 0.2396 | | Asterix |0.0025$\pm$ 0.0011 | 0.3941$\pm$ 0.146 | | BeamRider | 0.0059$\pm$ 0.0008 | 0.3763$\pm$ 0.1601 | | Breakout | 0.0021$\pm$ 0.0015 | 0.3622$\pm$ 0.1531 | | CrazyClimber |0.0074$\pm$ 0.0047 | 0.4034$\pm$ 0.2733 | | DemonAttack | 0.0003$\pm$ 0.0002 | 0.3972$\pm$ 0.1961 | | Gopher |0.0069$\pm$ 0.0032 | 0.4139$\pm$ 0.1272 | | Hero | 0.0048$\pm$ 0.0005 | 0.39$\pm$ 0.1471| | Kangaroo | 0.0063$\pm$ 0.0052 | 0.3371$\pm$ 0.1605 | | Phoenix |0.0042$\pm$ 0.0024 | 0.3996$\pm$ 0.1576| | Seaquest | 0.0048$\pm$ 0.0014 | 0.4073$\pm$ 0.3206 | | Zaxxon | 0.0045$\pm$ 0.0043 | 0.3933$\pm$ 0.1956 | In the second table we report cosine similarity during learning averaged over all 12 games: | number of frames | cosine similarity (after) | cosine similarity (before) | | ------ | ------ | ---- | | 0 M | 0.982 $\pm$ 0.1374 | 0.9303 $\pm$ 0.1627 | | 10 M | 0.1034 $\pm$ 0.2808 | 0.4106 $\pm$ 0.2574 | | 20 M |0.0239 $\pm$ 0.1052 | 0.3493 $\pm$ 0.2074 | | 30 M | -0.0085 $\pm$ 0.0842 |0.4155 $\pm$ 0.1462 | | 40 M | -0.0039 $\pm$ 0.103 | 0.3148 $\pm$ 0.1537 | | 50 M | -0.0132 $\pm$ 0.0684 | 0.4046 $\pm$ 0.1222 | | 60 M | -0.0121 $\pm$ 0.0751 | 0.3366 $\pm$ 0.2077 | Overall, we see small cosine similarity after target updates, and we also see cosine values before target updates converging to values in the interval of [0.3, 04].
Thank you for the update! These experiments are insightful and I recommend including them into the paper. In response to providing evidence about the gradient contamination claim, I increase the score.
Thanks
We highly appreciated your insightful suggestion and your commitment to engage with our rebuttal. We will be sure to add these to the paper. Thanks for reinforcing your support.
Decision
Accept (poster)