Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control

Summary

This paper investigates the sample efficiency problem in continuous control. The authors propose the BRO algorithm, i.e., Bigger, Regularized, Optimistic. The authors find that strong regularization allows for effective scaling of the critic networks, which, paired with optimistic exploration, leads to quite good performance. BRO achieves strong performance on numerous continuous control benchmarks, and is the first model-free reinforcement learning algorithm that can learn meaningful performance in DMC dog and humanoid tasks.

Strengths

- this paper is easy to follow and easy to understand - the studied topic is important to the RL community. It is vital to develop stronger and more powerful model-free RL algorithms for continuous control problems - despite that this work combines numerous previous well-developed tricks and strategies, the authors selectively incorporate them into one framework and demonstrate that such design choice incurs quite good performance. - the experiments are extensive and solid - the developed BRO algorithm is the first model-free RL algorithm that can achieve meaningful performance in DMC dog and humanoid tasks

Weaknesses

this paper has the following drawbacks, - the quality of the figures could be significantly improved. Please try to export pdf with matplotlib instead of taking screenshots by convention. - BRONet seems to be a simplified version of ResNet. I do not seem to observe any significant network architecture difference between them. The authors should not over-claim on the network architecture. Any clarifications here? - It is often unclear how the figures are plotted and which environments they cover, e.g., Figure 4, Figure 6. This should not be vague and ought to be clearly stated in the main text - I am a bit concerned with the claim that *Algorithmic improvements matter less as the scale increases* (Line 184). Do you think that this is always correct? One should focus more on scaling instead of algorithmic improvements in the context of RL? - missing baselines and references. The authors should compare against some other recent strong model-free RL algorithm, e.g., TD7 [1]. Meanwhile, the authors should cite the REDQ [2] paper when referring to replay ratios. REDQ should be included as a baseline approach in the paper. Moreover, a recent paper introduces sample multiple reuse (SMR) [3] that updates a fixed batch multiple times to boost sample efficiency. I think this can be a very relevant work and should be included and discussed in the paper. Also, the authors write that they introduce another actor network, and incorporate regularization techniques into critics, it reminds me of another work, DARC [4], where they leverage double actors for exploration and introduce critic regularization for better performance. It should be included in the paper. [1] For sale: State-action representation learning for deep reinforcement learning [2] Randomized Ensembled Double Q-learning: Learning Fast Without a Model [3] Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse [4] Efficient Continuous Control with Double Actors and Regularized Critics Despite the aforementioned drawbacks, this is a solid paper that may have a potentially large impact on the RL community. I would be happy to reconsider the score if the aforementioned flaws are addressed during the rebuttal phase.

Questions

- how do you expect BRO to be applied in the discrete control tasks? Will BRO beat other stronger MCTS-based approaches in Atari games? Any comments here? - can you elaborate more on explaining why scaling the actor network is not effective?

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have a good discussion of the potential limitations of this work in the main text.

Summary

This paper investigates how reinforcement learning (RL) can benefit from parameter scaling. The authors introduce BroNet, a variant of ResNet with LayerNorm, as a well-regularized network structure for the critic that improves performance when scaled. They also find that when the critic is properly regularized, the common Clipped Double Q-learning trick can be replaced with optimistic exploration, further boosting sample efficiency. These findings are combined into a new algorithm called BRO. To demonstrate the efficiency of their approach, the authors conduct extensive experiments on 40 challenging tasks across 3 benchmark suites. They also provide comprehensive ablation studies to justify their design choices.

Strengths

- The paper addresses the important and timely topic of scaling in model-free off-policy RL. - It offers extensive large-scale studies on various design choices, providing valuable insights to the RL community. - The proposed BRO algorithm shows promising results across a wide range of tasks.

Weaknesses

As acknowledged by the authors, the study primarily focuses on state-based off-policy RL. The transferability of their conclusions to other domains such as image-based problems and offline RL remains unclear, which limits the paper's broader impact.

Questions

- The benefits of scaling the critic appear to saturate after 5M parameters. What are your projections for further scaling? Do you anticipate additional benefits beyond this point, or do you believe there are fundamental limitations? - Could you provide more details on the evaluation metrics used, particularly for the MetaWorld benchmark? Given the various reporting methods in the literature for MetaWorld success rates, it would be helpful to clarify which specific metric was employed in this study.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

Yes, the authors have discussed the limitation in section 5.

Summary

The paper studies how to scale up RL algorithms in the continuous domain and introduces the BRO (Bigger, Regularized, Optimistic) algorithm, designed to enhance sample efficiency with (relatively) large models. The authors conduct extensive experiments to verify the effectiveness of factors like replay ratio, regularizations, optimistic exploration, and quantile Q-values when scaling up RL algorithms. The findings from these extensive experiments lead to the novel BRO algorithm, which consists of a novel architecture with proper regularization and exploration. Empirical results demonstrate that BRO achieves state-of-the-art performance on 40 complex tasks from the DeepMind Control, MetaWorld, and MyoSuite benchmarks, outperforming leading model-based and model-free algorithms, especially in the challenging Dog and Humanoid tasks.

Strengths

1. This paper tackles an important problem of scaling up in reinforcement learning, especially in continuous action domains. 2. The authors conduct extensive experiments on the effects of different methods on scaling up, which I found very informative. 3. The proposed algorithm, BRO, achieves strong empirical performances on various domains, especially on the Dog & Humanoid domains. 4. The paper is well-structured and well-written.

Weaknesses

Usually, scaling up benefits more when a large amount of data is available, where large models can lead to positive transfer or generalization across different tasks. However, the current setup is the same as the standard setting, where the agent is trained for 1M steps on each task separately. The work would be more significant if the model is trained on and can be transfered across different tasks.

Questions

See the weakness section. Is there any evidence that the proposed method can benefit from training on diverse tasks?

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

The authors discussed their limitations well in the paper.

Thanks for the additional results

I appreciate the author providing additional results on offline RL and image-based RL. However, the experiments seem to deviate from the goal of this paper. Changing the network from MLP to BroNet (with the same parameter size, I presume?) only shows the efficiency of BroNet other than "scaling also helps offline RL and image-based RL". The correct experiments should be also changing the size of BroNet like what you did in the main paper. Especially as you argue the the results saturate with 5M since the tasks are too simple, maybe you can prove that on the image-based tasks. I understand these experiments could be too expensive during the rebuttal phase. Thus, I would encourage you to provide them in the final version.

We thank for the rebuttal

We sincerely thank the reviewers for their valuable feedback and insightful questions during the discussion phase. We are confident that incorporating their suggestions has significantly enhanced the quality and value of our manuscript. We will be ready to answer new questions if any arise. Best regards, Authors