Recurrent Hypernetworks are Surprisingly Strong in Meta-RL

Summary

The paper investigate the performance of a specific type of architecture, RNNs coupled with hypernetworks, in meta-reinforcement learning. Meta-RL aims to address the sample inefficiency of RL algorithms by learning to perform few-shot learning when given a distribution of related tasks for meta-training. The authors note that while specialized and complex meta-RL methods have been proposed in the literature, recent work suggests that using an off-the-shelf sequential model, such as an RNN, trained in an end-to-end manner can serve as a strong baseline. However, the supporting evidence for this claim has been limited. The paper presents an extensive empirical investigation to address this gap. While RNNs can achieve strong performance in meta-RL, the study finds that the use of hypernetworks is crucial in maximizing their potential. Interestingly, when combined with hypernetworks, the simpler recurrent baselines outperform existing specialized methods and establish themselves as state-of-the-art (SOTA) on standard meta-RL benchmarks.

Strengths

S1: The paper is extremely clear S2: The authors make sure to tune each methods, which is something I do not see often enough S3: Coupling RNNs and hypernetworks is an idea that is novel in the field of meta-RL, and the attempts at understanding why they might be outperforming other baselines are interesting

Weaknesses

W1: While the hyperparameter tuning was fairly done in terms of computation budget, it is to me not clear this strategy is the most relevant. I can imagine that many of the complex methods have many more hyperparameters to tune than the simple methods that you propose, and an excessively small search grid could unfavorably disadvantage the former. W2: The empirical investigation is, as noted by the authors, quite limited. In that regard, more empirical evidence from different benchmarks would make the paper stronger.

Questions

Q1: Line 128, I could not understand why the predicted context is not directly used to condition the policy. From my understanding, the context should contain enough information to disambiguate the tasks, and the concatenation of the state and context should be Markovian. Could you provide an intuition why intermediate representations are used? Q2: The RNN-S is an interesting experiment. One additional hypothesis for why RNN+HN might work well is because it allows for some multiplicative interaction between the context and the state. What types of non-linearities were used in the network? Have you used activation units, such as Gated Linear Units (GLU) [1] which allows for multiplicative interactions between the neurons in the RNN-S? [1]: "Language Modeling with Gated Convolutional Networks", Dauphin 2016

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitations I can think of have been addressed by the authors.

Summary

This paper explores how to approach the problem of meta-reinforcement learning by using hypernetworks. In particular, the authors propose to employ an RL2-like scheme, where instead of producing a vector $\phi$, the recurrent model outputs a set of neural network weights. They dub the method RNN+HN. They introduce a set of task inference-based baselines with and without the usage of hypernetworks. Subsequently, they show that the simple RNN+HN approach outperforms more sophisticated baselines in gridworld environments and in Mujoco. Finally, they perform an ablation study by using RL2 with state-conditioning.

Strengths

Overall, I find the paper interesting and the results quite convincing. However, the authors' claims are very strong -- stating that the proposed method is state-of-the-art in the wide field of meta RL. Although the experiments do show that the method works quite well, I do not think it is enough to support this bold claim. As such, for now, I am going with "borderline reject", but I would be inclined to increase the score if the authors either extended the empirical evaluation or toned down the claims presented in the paper. Strengths: - The paper introduces a nice, conceptually simple idea that provides substantial improvements. Parameterizing the policy using a neural network with weights generated by a hypernetwork is an elegant idea. - The empirical results are in general quite good, the RNN+HN method outperforms most of the baselines in many cases. - The paper is well written. - The appendix includes additional ablation studies and more baselines.

Weaknesses

- The critical weakness of the paper is the mismatch between the claims and the empirical evaluation. Namely, the authors state multiple times that their method is state-of-the-art in the field of meta-RL, but the experiments do not support that claim sufficiently: - There are not enough external baseline methods. The baselines the authors use are inspired by existing methods but do not correspond exactly to what has been proposed in the literature previously (e.g. TI-naive, TI). Additionally, the authors omit existing established works such as MAML-based algorithms and PEARL. As they say, they "exclude the latter methods since estimation of a policy gradient in policy-gradient approaches requires more data than in our benchmarks". That is, in general, a reasonable assumption, but seems too strict when claiming state-of-the-art. - The authors support their claim by saying that their method outperforms previous SOTA [1, 2], but I'm not convinced that these previous works are still SOTA as of now. Some of the recently published works also show very good results [3, 4]. - Currently, the authors only consider two sets of environments: gridworlds and Mujoco continuous control. I think the empirical evaluation should be extended to include more complex environments such as Meta-World, RLBench, Atari, or DeepMind Alchemy. I think that environments with high-dimensional state spaces (e.g. images) would be interesting to test the limits of the proposed method. - There are some environments where RNN+HN falls behind (e.g. Walker, Cheetah-Vel). This is not a grave problem by itself, but it makes the problem of having relatively few benchmarks even more problematic -- how will this method scale up to other environments? - Why the RNN baseline only appears in a single environment in Figure 8 and Figure 9? I think it's an important baseline that should be included. [1] Zintgraf, Luisa, et al. "Varibad: A very good method for bayes-adaptive deep rl via meta-learning." arXiv preprint arXiv:1910.08348 (2019). \ [2] Beck, Jacob, et al. "Hypernetworks in meta-reinforcement learning." Conference on Robot Learning. PMLR, 2023. \ [3] Melo, Luckeciano C. "Transformers are meta-reinforcement learners." international conference on machine learning. PMLR, 2022. \ [4] Chalvidal, Mathieu, Thomas Serre, and Rufin VanRullen. "Meta-Reinforcement Learning with Self-Modifying Networks." Advances in Neural Information Processing Systems 35 (2022): 7838-7851.

Questions

My main suggestion related to the first point in the Weaknesses section above, i.e. extending the empirical evaluation or reducing the state-of-the-art claims.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The author discuss limitations sufficiently.

Summary

This paper investigates the performance of recurrent neural networks in meta-RL. They suggested that a current neural network can achieve strong performance with hypernetwork. They compared this method with numerous baselines on several meta-RL benchmarks and found that the recurrent baselines along with hypernetwork could achieve SOTA performance.

Strengths

This paper introduced a novel assertion that a recurrent method, when combined with a hypernetwork, can achieve SOTA performance in meta-reinforcement learning. The authors substantiated this claim with rigorous empirical experiments demonstrating superior performance. Furthermore, they conducted an analysis elucidating the reasons behind the method's impressive performance. The paper is well-written and clearly structured, which makes it easily comprehensible for the readers.

Weaknesses

1. The authors could enhance the comprehensiveness of the study by testing their algorithm on a wider range of environments. 2. The final analysis segment of the paper would benefit from further development to unequivocally ascertain why the proposed method can achieve state-of-the-art performance. 3. The paper lacks ablation studies for the different settings of hypernetwork component, leaving its individual contribution to the overall results unclear. 4. The format of the reference is wrong

Questions

1. Why in the Cheetah environment, RNN+HN is worse than RNN+S? 2. Please offer more ablation studies of the hypernetwork component.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have discussed the limitations of this paper in the last section.

Summary

The manuscript proposes to modify a common meta-RL baseline, in which a meta-learned RNN provides the policy network with an encoding of the trajectory. The proposed recurrent hypernetwork instead lets the RNN predict the weights and biases of the policy network. This is shown to be a strong model in comparison with several variations of RNN-based and task-inference meta-RL approaches.

Strengths

I think the proposed approach is novel and the idea is described in sufficient detail. The experimentation affords equal computational resources for hyperparameter optimization of all models. The method seems to perform better in 6 out of 7 considered environments. The ablation studies adequately address some questions that I thought of while reading the paper.

Weaknesses

1. The implementation of the RNN and RNN+HN models could've been described in a bit more detail. It is for example not clear what the meta parameters $\theta$ correspond to in the RNN baseline, since line 108 says that $f$ and $\pi$ use distinct parameters from $\theta$. Is this sentence wrong and $\theta$ are the RNN's parameters? I could also not find all hyperparameters of the architectures, such as the type and layer sizes of the RNNs. 1. For the strong claim that recurrent hypernetworks are SOTA in (all of) meta-RL, the presented experiments seem rather limited. What about experiments on RLBench or Meta-World? Also, there are many more categories of meta-RL methods to be considered than RNN-based and task-inference methods. The manuscript can benefit greatly from a clear justification of this strong claim. 1. I could not find an indication of how exactly the performance curves were generated, e.g. how many seeds per model, what exactly does the shaded area convey, etc.

Questions

1. References to figures such as "See Figures 3 and 4" in line 123, I'd put in parentheses directly in the preceding sentence or say something like "See Figure X for a diagram of Y". ## Acknowledgement of rebuttal I have read the rebuttal and reviews and engaged in discussion with the authors. The authors have mostly addressed my concerns by removing the general claims of the proposed method being state-of-the-art in meta-RL, by providing some clarification on the architecture and experimental setup, and by providing additional experimentation that support some of the design decisions. I have accordingly increased the score.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors mention that they cannot guarantee that the proposed method will improve over every baseline nor over every environment. This does not feel like a careful look at model details and how they might affect adaptation to different task variations. One general criticism of many meta-RL methods is that they are evaluated on very narrow task distributions. Maybe the authors could focus on that. The recent meta-RL survey by Beck et al. [1] might give some ideas for discussion. ## References 1. Beck, Jacob, et al. "A survey of meta-reinforcement learning." arXiv preprint arXiv:2301.08028 (2023).

Summary

This paper is an empirical investigation that tests the performance of different variants of three methods for meta-RL: RNN that provides task information implicitly from previous trajectories on the same MDP, TI that trains a task representation with VAE manner, and VI that is the baseline Varibad [1]. On mujoco environment, the paper finds that RNN with output being a hypernetwork that outputs the parameter for the policy works the best, and thus re-establish [2] RNN+hypernetwork as the state-of-the-art method for meta-RL. **References:** [1] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representation (ICLR), 2020. [2] Jacob Beck, Matthew Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta-reinforcement learning. In CoRL, 2022.

Strengths

1. Many variants are tested in the environment. There are 12 variants tested in the paper, which gives a thorough analysis on the effect of using hypernetwork for each method; in addition, the effect of "conditioning on the state twice" is also considered, which makes the conclusion that hypernetwork is crucial more rigorous. 2. The figures appended in the paper does a well job in conveying the architecture of each method.

Weaknesses

**1. The experiment results are not convincing enough.** a) The method proposed (line 184) in the paper, RNN+hypernetwork, is already proposed in prior work [1]. More specifically, RNN alone is equivalent to RL2 [3] as line 106 suggests, and RL2+hypernetwork is already tested in [1], which is inferior to Varibad [2]+hypernetwork. Thus, the key contribution of this paper is not proposing new method, but to overthrow the previous conclusion that Varibad+hypernetwork is better. However, this paper only tests gridworld and mujoco environments, and on both environments RNN+HN is only marginally better than VI+HN (which in [1] also has multiple methods that works similarly well, indicating that mujoco alone is not a strong benchmark). Most importantly, the metaworld (ML1 and ML10) environment, which is the decisive evidence that Varibad+hypernetwork is better than RL2+hypernetwork, is missing in this paper. With such result missing, it is hard for the readers to be convinced that the result is the other way around from that in [1]. b) The baselines, though with many variants, do not cover enough areas for the claim of the state-of-the-art. Reviewers of [1] have already pointed out that the state-of-the-art status of Varibad [2], even before the presence of [1], is questionable. Also, while it is true that the meta-RL method can be briefly summarized in policy gradient, implicit task representation and task inference as the paper suggests (line 61-63), in Table 2 of [4] there are many branches within each of the direction. For example, what if transformer instead of RNN is used [5]? Is policy gradient method necessarily worse than the other two branches? For a state-of-the-art method, those methods also need to be considered. **2. The delivery of the paper's idea can be improved.** a) There is no motivation stated about why hypernetwork is used. The paper only states that "we present the key insight that the use of a hypernetwork architecture is crucial ..." (line 36-37) and "hypernetwork has never been widely evaluated in meta-RL" (line 81-82). However, there is no intuitive explanation about why hypernetwork is useful for meta-RL and should be considered in the first place. While there is explanation in [1] about preventing degeneration of multi-tasking, the paper should be self-contained and inform the readers about why hypernetwork, the most important component in the finally proposed RNN+hypernetwork method, is considered for meta-RL. b) The method section can be modified to convey the idea more clearly. Currently, it is hard to identify which method is proposed (line 184) by the paper, and it is unclear for the first-time readers about why multiple methods are listed in the method section (and most of them is not proposed by this paper), instead of the usual paradigm where each key component is listed in a subsection. Since the methods are not novelly proposed by this paper, they can be put into experiment setup section (with more emphasis that this is an empirical study paper rather than conventional paper that proposes a new method, e.g., add a contribution summary at the end of introduction section); or, a summary could be added in the front of the method section, to tell the readers that all those methods listed below are tested in the experiment section, and in each section we discuss one type of method tested. A table summary would also be very helpful. c) There is no detail of the environment settings except gridworld. What is the cheetah-dir environment? What are the definitions of state, action and reward, and how are the MDPs in meta-learning differs? What reward can be considered expert-level performance? While they can be found in prior work, the paper should be self-contained and the settings should be attached in the appendix to help the reader form a better intuition about the environment. **3. Other minor problems:** a) Some experiment results are missing, for example, RNN in Fig. 8b, 8c, 9b, 9c and 9d; b) Fig. 10 is never referred to in the paper; reference to Fig. 10 should be added in "latent gradients" paragraph. c) The x-axis of walker environment is not aligned with other mujoco environments. d) The name, VariBad, should be mentioned in VI and VI+HN to more clearly show the connection of the method tested to existing baselines. **References:** [1] J. Beck et al. Hypernetworks in meta-reinforcement learning. In CoRL, 2022. [2] L. Zintgraf et al. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In ICLR, 2020. [3] Y. Duan et al. $RL^2$: Fast reinforcement learning via slow reinforcement learning. In arXiv, 2016. [4] J. Beck et al. A survey of meta-reinforcement learning. In arXiv, 2023. [5] L. Melo. Transformers are Meta-Reinforcement Learners. In ICML, 2022.

Questions

Besides those listed in the weakness section, I have one question: why the performance of RNN+HN in Fig. 9a and Fig. 10a is different? Below are my suggestions for this paper: 1. Modify the introduction and method section of the paper such that it is more clear that this is an empirical study instead of proposing a novel method, and add contribution summary at the end of the introduction. Add a table that summarizes the methods tested in the paper. 2. Add environment details of the mujoco environments in the appendix. 3. Compare RNN+HN to more methods summarized in [1]; 4. Compare RNN+HN with VI+HN (and other baselines) on metaworld and other more challenging environment; 5. Fix other minor issues mentioned in the weakness section. 6. I strongly advice the authors to open source their code upon acceptance / next submission. The author could also consider adding license statement in the appendix. **References:** [1] J. Beck et al. A survey of meta-reinforcement learning. In arXiv, 2023.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

There is an independent limitation section in the paper, which I think generally discusses an important limitation of the work. Though, as suggested in the weakness section, I think currently the limitation is not sufficiently mitigated. There is no negative societal impact discussed in the work, which I would encourage the readers to add; though the work is still far from application, automated control itself brings potential challenge, such as job loss, to the human society.

Response to the Rebuttal

Thanks for your detailed response, and I appreciate the efforts made by the authors to address my concerns. I think concerns such as delivery are well-addressed. Here are my follow-up responses: 1. While I agree that the supplementary minecraft environment is a stronger evidence than that shown in the original paper that RNN+HN is better than HN+VI, for the claim of "much better" in RL, I would expect at least one scenario where RNN+HN succeeds but HN+VI only reaches medium-level performance (e.g. 50-70% reward) or even completely fails. The performance difference on gridworld and mujoco only seem to be either an issue of convergence speed of ~10% performance difference. 2. Regarding the result for ML1/ML10 in [1]: a) I am referring to the Table 2 of [1] for result (on the 7th page on ArXiv version), where on ML10 task, the performance comparison between VariBAD+HN and RL2+HN is as follows: -|VariBAD|RL2 ---|---|--- HFI|**$28.4\pm 6.0$**| $7.1\pm 2.4$ Bias-HyperInit|**$23.9\pm 6.2$**|$14.2\pm 7.2$ b) in line 106, the authors write "**RNN** ... is equivalent to RL2"; in line 112, the authors write "we follow the initialization method for hypernetworks, Bias-HyperInit, suggested by Beck et al.". Thus, I assume that RNN+HN, the best method found in this paper, is equivalent to the RL2+Bias -HyperInit, which has the performance of $14.2\pm 7.2$ as opposed to $23.9\pm 6.2$ with VariBAD. When you plot them on a figure (which is the bottom-right figure of Figure 1 in the appendix, on the 13th page of the ArXiv version), even with large variance, the upper edge of the RNN+HN can barely touch the lower edge of VariBAD+HN, and the reward is almost doubled in VariBAD. Thus, I think this can be called significant despite the large variant. 3. Another thing that worries me from giving a higher score is that the change of claim is a major revision to the paper's contribution and organization. To sum up, I tend to keep my scores for now and see how the discussion goes (both ours and the discussion between the author and the other reviewers). **References:** [1] J. Beck et al. Hypernetworks in meta-reinforcement learning. In CoRL, 2022.

New Meta-World Experiments

**TLDR:** New Meta-World experiments show RNN+HN matches performance of VI+HN on ML10, which should address the last major concern of the reviews. Some reviewers have requested ML10 (from Meta-World), noting that Beck et al., 2022 demonstrated RNN+HN to underperform VI+HN on ML10. (This is the only experiment ever run with RNN+HN in meta-RL prior to our paper.) We noted that Beck et al., 2022 actually did run "two-tailed t-tests with p = 0.05 to determine significance" and found NO results on ML10 to be significantly different, which makes the result a bit misleading. The fact that that singular results carries so much weight is one of the reasons that the much stronger countervailing evidence in our paper is so necessary to the conversation. However, to allay the concerns of reviewers, we have conducted this experiment. While we are unable to upload an additional PDF, we will describe the results here and include the plots with the final paper. First, we find that the learning curves for RNN+HN and VI+HN overlap consistently throughout training. For instance, on training curves that run from 0 to 7e7 frames, the curves intersect near training steps 2e7, 4e7, 5e7, and 7e7. Second, we find that the variance of RNN+HN is significantly less than that of VI+HN. For nearly all of training, the confidence interval of RNN+HN falls within that of VI+HN. And finally, the final return of RNN+HN is higher (thought not significantly higher) than VI+HN. RNN+HN has a final average test success percent of 17.22, compared to 13.87 for VI+HN. While these results are not sufficient to show superiority on ML10, we do show superiority on other environments in the paper. Additionally, even matching existing task-inference baselines would still be of significant interest to the field. Given that it has widely been assumed in meta-RL that complicated task-inference methods are necessary to be competitive, this paper is the first to present strong evidence in the other direction. Finally, to ensure that the difference from Beck et al., 2022 is legitimate, we acquired code from that paper and re-ran that experiment (which runs marginally longer and with a smaller latent encoding size). We only had sufficient compute to re-run VI+HN. Still, using different seeds from the original paper resulted in worse performance, indicating that the results truly were insignificant due to variance. Specifically, the final mean return is 18.35 for VI+HN, with is an average over three seeds, with final returns 32.80, 3.32, 18.93, respectively. In contrast, the original paper reports a mean return of 23.9 for VI+HN. Clearly, with such high variance the comparison between these two methods on ML10 is not useful, hence our reluctance to use ML10 and our additional MineCraft experiments. We hope that this additional evidence on the requested benchmark puts all questions of ML10 to rest.