Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agents

Summary

Overfitting in deep Q-learning agents is a recent topic of interest in the RL community, and several methods have been proposed to mitigate this problem, including data augmentation (DrQ), random ensembles (RedQ, DroQ), and resets (Nikishin et al.). This paper builds upon prior work on periodically resetting weights and addresses one of its core limitations: by resetting weights, an agent will perform poorly immediately after resetting, although it eventually recovers and often exceeds its performance before resetting. This paper proposes to learn an *ensemble* of agents and periodically reset only one of the agents at a time (in sequence). Experiments are conducted on tasks from DMControl, Minigrid, and Atari100k, and indicate that resetting with ensembles is effective at mitigating deterioration of performance immediately after a reset.

Strengths

This paper is well written, considers an interesting problem, and experiments appear sound. The description of the proposed method is easy to follow, and especially Figure 4 is useful for understanding how the ensemble resetting works in practice.

Weaknesses

- It would be useful to include more discussion on related works that seek to understand and mitigate overfitting in deep Q-learning besides resetting. There is a lot of literature in this area and I imagine that the authors are familiar with the literature, so I will refrain from mentioning any specific references (besides the methods mentioned in my summary) to remain impartial. - I would like to see more ablations to get a deeper understanding of the trade-offs in ensemble resetting vs. prior works. How does sample-efficiency and performance drop change when the number of ensemble agents $N$ varies? How often should one reset agents? Is the rate of resets and number of agents dependent on the replay ratio? What is the computational cost of using additional agents vs. increasing the replay ratio? Would the proposed method benefit from using the ensemble in other ways as well, e.g. by using the RedQ trick for computing TD-targets? Addressing some or all of these questions would likely increase the impact of the work.

Questions

I would like the authors to address my comments listed in "weaknesses". If the authors are not able to conduct some or all of the experiments required to answer my questions during the rebuttal, I'd like to see a discussion of what the authors would expect the results to be based on their experience.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

There is sufficient discussion of limitations.

Summary

This paper combines the resetting method proposed by (Nikishin et al., 2022) as a remedy to the primacy bias affecting deep RL algorithms with the use of ensembles of agents. The proposed RDE method, apart from generally improving performance, has the goal of minimizing the regret associated to a learning agent that uses periodic resets, mitigating the severeness of the performance drops it experiences right after a reset. To do this, without sacrificing too much on exploration and online data collection, the probability of executing an action in the environment is evaluated according to the oldest value function. Empirical results show benefits in using this approach, at both low and high replay ratio, in standard robotic locomotion and navigation tasks, as well as a safety domain.

Strengths

**Originality**: despite the combination of resets and ensembles is not particularly original, the focus on developing a technique for leveraging their combination to avoid performance collapse while resetting is, to the best of my knowledge, new. **Quality**: the quality of the work is generally good. The experiments cover a reasonable number of domains and the ablations mostly answer natural questions. **Clarity**: the clarity of the writing is good. The paper would benefit from some small tuning to the presentation here and there, but the overall flow makes clear what the contribution is. **Significance**: harnessing the performance benefits originating from the mitigation of the primacy bias while at the same time not incurring a cost in terms of regret is a worthy research direction which could be interesting to many practitioners and researchers.

Weaknesses

**Major Concerns** - If I understand it correctly from Algorithm 1, a different policy could be potentially selected at each step in the environment. This could conceptually create problems in terms of inconsistent behavior, since each policy will have to deal with the actions previously sampled from a potentially very different policy, and in terms of lack of "deep exploration", since this would harm temporally-consistent behaviors. A reader would benefit from a discussion or analysis of this aspect, to understand whether this is happening at all, or, if not, why that might be not happening in this kind of tasks of setting. - The idea of combining periodic reset and ensembles of agents in continuous control has been explored in "Unleashing The Potential of Data Sharing in Ensemble Deep Reinforcement Learning" (Lin et al, 2022). I find the idea of the paper of using this combination to avoid performance drops to still be valuable, but discussing the relationship with that paper could better contextualize the contribution. **Minor Concerns** - Y and X labels are missing from all Figures in the paper, making it hard to parse the plots at first sight. For most plots, it is either quite easy to infer the quantities of interest, or they are explicitly mentioned in the caption, but it can still be very misleading for many readers. - The paper would benefit from the extension of some of the ablations to more tasks. In particular, I find the performance of the ensemble-based approach without any resetting on top to be an important baseline to contextualize the results in Figure 3, and I think results concerning it would be a good addition to the Figure.

Questions

I ask the authors to provide answers to my concerns expressed above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors do not discuss any computational consideration resulting from their use of ensembles of agents. I encourage the authors to add such a discussion to the paper.

Summary

The work proposes an extension to the resetting strategy proposed by Nikishin et al. The extension intends to mitigate the catastrophic performance collapse often observed for the simple resetting strategy while keeping the properties that help avoid the primacy bias. To this end, the work makes use of ensembling techniques, such that the policy/value network of an RL agent is not completely reset, but can fall back to different checkpoints. The method is evaluated on a variety of environments and a further extension is presented that enables application in safety critical systems.

Strengths

* The presented method cleverly combines ensembling with resetting to improve RL agents. * It seems general enough that it should be usable as a plug and play method for a broad variety of RL agents. * The work mostly easy to follow * It is demonstrated that the method is flexible and can incorporate auxiliary information, such as safety-critical information to provide a safer method than the vanilla resetting one

Weaknesses

The presentation could be improved: * A lot of the discussed "preliminaries" seem irrelevant for the content of the paper. For example, the paragraph on "Off-policy RL." seems not necessary and the content in "Primacy Bias and Resetting Deep RL Agent." is largely a repetition of the introductory text. * Algorithm 1 is never discussed in the text and feels wholly redundant with Fig. 1. This half page might be used to show more experimental results. * Algorithm 1 is not consistent with the text. From the Algorithm it looks like every ensemble member is reset every $N\times T_{reset}$ time-steps and not $T_{reset}/N$ as stated in line 149. * Lines 178 - 180 are concerned with expressing that the "oldest Q-function" is used to normalize in the selection mechanism. This can be expressed in much clearer terms "oldest Q-function" as is shown in line 183. * In line 195 it is claimed that RDE effectively prevents performance collapses, however I disagree with this wording. It can mitigate it to some extent, but the experiments clearly show that there is still performance collapses happening. * The description of Figure 2 is wholly confusing. The long sentence explaining what the y-axis is showing is expressed in a very convoluted way. * In figure 2c it should not be possible that the performance of the baseline "Base" is below 1 when using an RR of 1. * Design decisions are often not well enough explained. * Figure 1 does not explain what RR is * The paragraph heading in line 204 should be "Baselines & RL Agents" not "Baselines & DNN Architectures" * Line 246 claims that there is no significant drop on the humanoid-run example. The reward for the RDE method drops from 150 - 100. This represents a significant drop in my opinion since 1/3 of the performance is lost. * The choice for designing the selection mechanism is very unclear. The discussion about using the oldest Q-function in the selection mechanism seems more to point to only using the oldest Q-function. Indeed, this seems to be supported in the experiments and should have been at least an ablation. Additionally the initial paragraph of Section 3.2 seems to also point to only using the oldest Q-function * It is claimed that a $\beta$ set to 50 "nearly eliminated" performance collapses (line 270). However, the performance collapse in the Figure is again the 1/3 loss in performance. * Section 5 feels like an added afterthought. I don't see why it warrants an own section. It repeats some of the discussion of safe RL from the preliminaries section. Instead the "safe" selection mechanism should have been discussed in Section 3 and then only the results should be a subsection of Section 4. * Where does the value for the reset frequency $4\times 10^5$ in line 221 come from? The experiments seem to have an unfair comparison and are likely not reproducible: * All details about how hyperparameters were determined seems to be missing * The value of $N$ is never stated. Only from the plots can it be assumed to be 2 * The ablation does not take into account all confounding factors and should be redone * It is claimed that using a reset frequency for the individual members being equal to the single network case is fair. I fail to see how this is fair. This seems to just give benefit to the RDE method since it can benefit $N$ times as much from resetting. * Without stating how hyperparameters were set of the methods, it seems like an arbitrary comparison of the methods. * N is not really ablated and it's not clear how the ensemble increases computational overhead. The description of an MDP is wrong. An MDP is an abstract representation of an environment. The MDP does not consist of an agent. The bounds for the discount factor in line 74 should be $\gamma \in [0,1]$ not $\gamma \in [0,1)$. It is totally valid to have undiscounted cases. Overall, the work would need significant rewriting and an overhaul for the experiments for me to accept it. I am very doubtful that this can be done in the time-frame of a rebuttal.

Questions

How would PBT style methods compare to resetting strategies. Are they doing some form of resetting?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations of the method have not been discussed. An obvious limitation is that ensembles likely will increase the computational overhead. For example, in line 131 it is said that the simple resetting strategy often requires high replay ration and therefor more resources. This should likely be worse for the presented method and the conducted experiments are not convincing enough to show that the novel method would require fewer resources.

Summary

This paper proposes a novel reset-based method that leverages deep ensemble learning to address the primacy bias issue in deep reinforcement learning. The authors construct N-ensemble agents and reset each ensemble agent sequentially to prevent performance collapses and improve sample efficiency. The proposed method is evaluated through experiments on various environments, including safe RL scenarios, and the results demonstrate its effectiveness and potential for real-world applications.

Strengths

1. The paper addresses an important issue in deep reinforcement learning, namely primacy bias, and proposes a novel method to mitigate its effects. This is valuable as primacy bias can lead to overfitting and performance deterioration, which affects the applicability and efficiency of deep RL algorithms. 2. The use of deep ensemble learning in the proposed method is innovative and practical. Deep ensemble learning has shown effectiveness in domains such as image classification and RL, and leveraging the diversity gain of ensemble agents can enhance performance and prevent performance collapses. 3. The paper provides a comprehensive analysis of the proposed method, including its underlying operations and how it effectively prevents performance collapses. This analysis adds clarity and depth to the understanding of the proposed method.

Weaknesses

1. The paper stated that RDE was conducted for tasks with safety requirements, but it only did one experiment on safe RL benchmark. Thus the paper lacks effective validations for RDE on violations. Simply stating that RDE does not cause performance collapse in general continuous control tasks cannot explain its role in safe RL.

Questions

1. Can you show more experimental results on safe RL benchmark?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper stated that RDE was conducted for tasks with safety requirements, but it only did one experiment on safe RL benchmark. The paper has no negative social impacts.

Rebuttal Reponse (Increased score)

Thank you very much for the detailed response and the many additional experiments. My concerns and issues have been mostly addressed and I am much more positive towards to presented work. I increase my score form 3 to 5. You stated in the rebuttal that you tuned the hyperparameters. How was this done? A grid search? Random search? How much tuning is required to get a good performance of the method? Did you use the same tuning budget for all methods? I am still concerned that the work requires substantial rewriting which might warrant a resubmission. Since other reviewers have not raised that point though I will discuss that in the reviewer discussion.

Thanks for the response

I am satisfied with the response from the authors. Provided that they will do the modifications to their paper (concerning figures, related work and other tweaks proposed in responses to other reviewers), my opinion is that the paper should be accepted. I am raising my score.