Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design

Summary

This paper focuses on the problem of learning update rules in reinforcement learning. By combining the advantages of the Learned Policy Gradient (LPG) algorithm and the Unsupervised Environment Design (UED) technique, the authors propose a meta RL algorithm, namely, GROOVE (General RL Optimizers Obtained Via Environment Design), to address the generalization problem for meta RL algorithms are applied to unseen environments. Accounting for the curricula generation of the meta-learned optimizer, the authors also design a metric named algorithmic regret (AR) to evaluate the regret during the meta-training, then be used to guide the learning of meta-learner. A series of experiments also present that GROOVE outperforms baselines on the Atari games.

Strengths

1. This paper is well-organized, and the authors have a throughout introduction for the related work. 2. Experiments show that the designed AR outperforms existing metrics.

Weaknesses

1. The approximation of algorithmic regret seems heavily relies on the choice of an oracle algorithm as UEDs, i.e., the choice of antagonist agent, which would be a performance bottleneck for GROOVE. Thus, there would be better if the authors demonstrate an ablation study on using different antagonist agents, i.e., training different RL algorithms. 2. The novelty is limited, as GROOVE is a direct combination of UED and LPG.

Questions

Whether different antagonist agent will affect the performance?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

The paper "Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design" proposes an automatic curiculum approach for meta-learning of RL optimizers. It is based on a notion of regret of the optimizer, to choose environment parameters that are at a good level for the optimizer to leverage feedback from RL training steps. Since the notion of regret is intractable to be exactly computed, authors introduce a notion of algorithmic regret, which is an approximation. Results show intersting properties of the method.

Strengths

- Meta-learning of RL optimizers is an important problem - The proposal looks to contain innovative components - Interesting results

Weaknesses

- The paper is very hard, not to say impossible, to follow, as many many different components are not sufficiently defined (when not defined at all). To be useful, a research paper must contain every formalization and definition to allow an educated researcher to understand and reproduce the contribution. While I am not fully familiar with meta-learning of optimizers, I have a strong background in RL and I am still unable to put everything together in that paper to understand the proposal. I really think the paper should be fully rewritten before considering publication anywhere. Here is a (non-exhaustive) list of things that are really difficult to understand: - U_\eta is introduced in 2.1 with y_t,\pi_t=U_\eta(x_t|x_t+1...x_T). I understand it is an LSTM on sequence x_t+1...x_T but why giving x_t| here ? Is it a distribution of x_t given the sequence ? This does not fit with the fact to feeding this to variables y and \pi. Is it the result of a LSTM step after feeding x_t in the sequence ? but x_t contains y_\theta(s_t) so it does not make sense to me. - In 2.1 we do not know how is trained U_\eta (\eta discussed only in 3.1) so it is very diffcult to understand what it captures. - what is y_\theta and why does it subscripts \theta as the policy \pi does ? policy and boostrap share same parameters ? what is it supposed to capture ? - in 2.2 V gets a meta-optimizer F as input, in 3.1 it takes parameters and in 3.2 an RL algorithm - G in 3.1 is not defined : is it the regret ? - \cal{H} is not defined - F_\eta not well defined, we do not really know what controls \eta - Algo 1 : "Compute meta-gradient for \eta with updated \theta (eq 2)" ==> eq 2 is an expectation of expected return, not a gradient - Algo 1 : "update \Lambda with R and phi" => what does it concretely mean ? You store the regret associated with phi in the buffer ? - Algo 1: What means initialize lifetimes? How is it done ?

Questions

see weakness above

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

3 good

Limitations

..

Summary

The paper presents an approach, i.e. GROOVE, to learning an update rule for generalization on unseen tasks automatically, based on the idea of Unsupervised Environment Design, where a student agent is trained on an adaptive distribution of environments proposed by a teacher and the teacher seeks to propose tasks which maximize the student’s regret. The authors also introduce a new concept of algorithmic regret, which is used to approximate regret and automatically generate curricula. The results of experiments comparing GROOVE and LPG demonstrate the superiority of GROOVE in terms of generalization.

Strengths

1. The proposed method is well-motivated and the authors provide a clear explanation of the underlying concepts. 2. GROOVE, especially the use of algorithmic regret, is simple to understand and effective that is well-supported by the experimental results and analysis.

Weaknesses

1. The paper could benefit from listing its contributions, e.g. in the first chapter, which helps readers capture its novelty faster. 2. More detailed discussion of the connection between GROOVE and related works in the field of RL and meta-learning (i.e. in the part "Alternative Approaches to Meta-Reinforcement Learning") will be helpful for readers to address this work.

Questions

1. Have the authors tried other RL algorithms as the baseline in the Algorithmic Regret? if so, how do they perform?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

as stated in the weakness

Summary

The authors present a meta-learning method to enhance an RL algorithm's performance across diverse RL tasks. They utilize the concept of unsupervised environment design (UED) methods for training RL agents and adapt this approach for the task of meta-learning a policy optimizer. The authors propose a novel algorithmic regret (AR) to facilitate UED and confirm its superior performance to L1 value loss and positive value loss through experiments. The experiments verify the effectiveness of proposed method.

Strengths

1. The paper is well-structured and easy to understand. 2. A thorough review of related studies is provided. 3. The proposed idea, while simple, appears logically sound. However, as I am not intimately familiar with the domain of reinforcement learning, I am unable to evaluate its novelty.

Weaknesses

1. Could the authors further elaborate why the proposed Algorithmic regret is better than the approximation method before? Some intuitive discussions are necessary in addition to only experimental results. Moreover, AR is compared to L1 value loss and positive value loss in Sec. 4.5. It seems there are also some other loss in Lines 16-25. Could the authors clarify the reason that the comparison between AR and these loss is not conducted? 2. It would be informative if the authors could discuss the scenarios depicted in Fig. 1 where Groove performs less effectively than LPG. 3. "The result is our method" in abstract could be rephrased. 4. It is mentioned that "By examining how characteristics of the meta-training distribution impact the generalization of these algorithms". It is unclear to me where it is reflected in the paper. Could the authors further illustrate it? Minor: Line 50 Section 4.4 -> Section 4.5

Questions

Please refer to the 'Weaknesses' section above for questions and clarifications.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

No.

Summary

This paper proposed a new framework, GROOVE, to solve a new Meta-UPOMDP problem. It also introduces algorithmic regret (AR) to approximate the regret to update the curator and generator (sampler?). The results show that the meta-optimizer learned by GROOVE can be used to improve the games in Atari.

Strengths

1. The proposed problem Meta-UPOMDP is important. 2. The experiments show that the method can work even from grid-world to atari. 3. The AR idea is novel. 4. The results of AR in Fig. 6 is great.

Weaknesses

1. Several definitions are absent, including the sampler in Fig2 and the symbol \eta mentioned in line 74. 2. It would be beneficial to introduce PMO earlier. The headline and abstract might lead readers to expect traditional RL and few-shot meta-learning. 3. It is recommended to conduct tests on additional environments, such as Procgen. Even just for evaluating the meta-optimizer learned in Grid-World. 4. Since AR constitutes a crucial aspect of the work, further insights should be provided. 5. The majority of the work involves combining existing works (LPG and PLR).

Questions

1. How do you select algorithm A in equation 4? Including the hyperparameter of the algorithm. 2. The claim in 209 seems similar to [1]. Please explain the difference. 3. What is the probability of using the generator and the curator? [1] Li, Y., & Zhang, C. (2022). On Efficient Online Imitation Learning via Classification. Advances in Neural Information Processing Systems, 35, 32383-32397.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Mentioned in weakness.

Thanks

Thanks to authors for their insightful answers that help le better understand the contribution. Despite this and ither revierwers opinions, still think the paper should be reorganized to help the reader. At least the global objective should be clearly formalized at the begining of section 2. Maybe 2.2 before 2.1 would be easier to read ? There is also still no clear definition of what is a lifetime that is core of the algorithm. I understand arguments if authors that claim that algo1 is high level picture, but I feel that in that form it does not help. Giving some further details in it would be very insightful. Also, a picture that would include all notation to illustrate the learning process would be very helpful (figure 1 is not). I still think that this paper, even if I am sure present very interesting contribution that I would like to see published soon, is not ready for publication at this round. Although, I won't oppose to its acceptance if AC and other reviewers like it. Thus, I increase my score to bordeline reject to reflect this.

Disappointing engagement

Unfortunately, after a week of the discussion period, we received only a single response to our rebuttal (thank you to reviewer 4F7T). **Please may reviewers 8DkV, qWd8, TbjH, and f1yc take a look at our rebuttal**, which includes multiple new results and edits, and consider updating their scores if we have adequately addressed their feedback?

Thank you

Thank you for your response to my comments. I acknowledge the efforts made to address the questions I raised. Because I am not quite familiar with the reinforcement learning topic, I will maintain my score and be attentive to the other reviewers' thoughts.

Official comment

1. Thank you for providing the results of Procgen. However, the experiments do not train for enough steps. Normally, Procgen needs to train for 200m steps. Hence, it is hard to tell about the improvement. 2. Thank you for providing experiments on using different algorithms as regret antagonists. However, it seems like most algorithms are bad besides A2C. These results lowered my confidence in the method.

Thanks for your response

I thank the authors for their response, and I'll keep my score as I do not think there is a significant novelty.

Official comment

I don't have any additional questions. I'm inclined to accept this paper, contingent on the consensus of the other reviewers.