While reinforcement learning has achieved remarkable successes in several domains, its real-world application is limited due to many methods failing to generalise to unfamiliar conditions. In this work, we consider the problem of generalising to new transition dynamics, corresponding to cases in which the environment's response to the agent's actions differs. For example, the gravitational force exerted on a robot depends on its mass and changes the robot's mobility. Consequently, in such cases, it is necessary to condition an agent's actions on extrinsic state information and pertinent contextual information reflecting how the environment responds. While the need for context-sensitive policies has been established, the manner in which context is incorporated architecturally has received less attention. Thus, in this work, we present an investigation into how context information should be incorporated into behaviour learning to improve generalisation. To this end, we introduce a neural network architecture, the Decision Adapter, which generates the weights of an adapter module and conditions the behaviour of an agent on the context information. We show that the Decision Adapter is a useful generalisation of a previously proposed architecture and empirically demonstrate that it results in superior generalisation performance compared to previous approaches in several environments. Beyond this, the Decision Adapter is more robust to irrelevant distractor variables than several alternative methods.
Paper
Similar papers
Peer review
Summary
The paper proposes a novel way of learning how to adapt policies in contextual RL. The work proposes to use "adapter" modules whose parameters are determined by a hypernetwork that solely focuses on the context variable.
Strengths
The paper is generally easy to follow and well written. Illustrations are well placed and utilized to communicate the content of the paper and to emphasize the concepts that are being discussed. The work also addresses an important issue for RL, namely generalization, by combining existing ideas in clever ways to tackle this issue. I particularly commend the authors for the discussion of the limitations and the experiments that are conducted to explore the limitations. Overall, the presented method "adapter" seems well suited to be used for training more general RL agents. As such I believe the work is of high significance for the RL community and should inspire future works. Overall I think this paper is a clear fit for the NeurIPS (RL) community
Weaknesses
The presentation could be improved a bit. For example: * The title does not seem to fully fit with the work, or rather, it is overly general and gives the impression that a novel cRL formulation will be presented. * Similarly, the abstract only spends ~1/3 of the space discussing the "Decision Adapter" method * The wording in line 45 makes it seem like context could not be observed by a sensor. However, the robot mass example would be a sensable context. * The restriction on only changing dynamics and not changing rewards seems not really needed. * In line 115, Procgen is cited for examples where context unaware policies can work well, but I'd argue that the context is embedded in the visual state representation * Theorem 4.5 can only be understood when reading the Appendix. This is the case due to $\mathcal{C}\_{far}$ and $\mathcal{C}_{close}$ never getting defined in Section 4 but only in the appendix. * Similarly, its easy to overlook why the inequalities in Definition 4.3 make sense, since it depends on the reward definition. * The theoretical results from Section 4 seem a bit arbitrary since they heavily depend on the construction of the example. It might make sense to move this part to the appendix and move some more of the empirical evaluation to the main paper. * The AER metric is introduced but "AER" does never show up again in the paper. Is it maybe labeled as "Mean Eval. Reward" in the plots? * It is not clear why AER can not be defined over a context set that is distinct from the training set * The citation to Benjamins is by now a TMLR publication There is one more niche paper using cRL for learning dynamic configuration of AI planning algorithms that should be of interest to you: * Biedenkapp et al. (2022) https://rlplab.com/papers/biedenkapp-et-al-icaps2022wsprl.pdf (follow up to Speck et al. 2021 https://ojs.aaai.org/index.php/ICAPS/article/view/16008) This work discusses how to treat state and context information separate and also discusses different architectural choices that one can make in this regard. I don't think this would form a reasonable baseline to compare against, however it feels like an appropriate reference. In particular, it feels like a good reference since the work considers 300+ noisy context features and how to learn with them, compared to the relatively few context features considered in the present work. While I listed many bullet points for potential improvement, I am very happy with the state of the paper and I think it should be accepted. I am even willing to increase my score if code will be released.
Questions
* Will the code be made available? * Why is the pre-training phase (as done in NLP) not used for adapter? I would imagine that the hypernetwork would benefit from that and only need to "fine-tune" the generated weights to work well with the observed reward signals. * With many references to the work of Benjamins et al. 2022, why are not more environments from the proposed CARL benchmark used? * Intuitively it makes more sense to me to describe the Theoretical foundations in Section 4, using the optimality gap as defined by Benjamins et al. (2022). What is the advantage of using an Optimality ratio instead? * Are the same context distributions used in the cartpole example as done by Benjamins et al. (2022)?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
Limitations are discussed thoroughly and experiments are provided that support the discussion on limitations.
Summary
This paper addresses the problem of learning policies that can generalize to multiple contexts. The problem is formulated as a contextual MDP (CMDP). The paper starts by analyzing analytically the performance of policies that are not aware of the context in two different scenarios of a particular instance of CMDP. After that, a network architecture (decision adapter), which incorporates contextual information through a hypernetwork, is introduced. The architecture is empirically evaluated in three environments and its performance is compared against 4 baselines.
Strengths
The paper tackles an important problem in RL. The idea of using a hypernetwork to incorporate contextual information is sound and the empirical results are promising. The paper is very clear and easy to follow.
Weaknesses
I believe that the paper's novelty is somewhat limited. While I found the solution of using a hypernetwork quite neat, I was slightly disappointed to discover that a similar architecture called cGate had already been proposed to address the same problem. The cGate architecture is mentioned for the first time in Section 5 and is not introduced as a related work. The authors argue that the adapter architecture is a more powerful generalization of cGate as it can also capture non-linear relationships between states and contexts. This is confirmed by the empirical results obtained in the ODE domain. However, the results of cGate on the other two environments (CartPole and Mujoco Ant) are not reported. Moreover, the ODE domain seems to be specially designed to make cGate fail, as the dynamics are given by an nth-degree polynomial (equation Line 229). Finally, I found the example provided in Section 4 to be very helpful for understanding. However, in my view, framing the analysis o as a theorem is not appropriate in this case, since the applicability of the results seems limited to this specific instance of the problem. Typically, a theorem should offer broader insights into the overarching problem at hand.
Questions
1. Could the authors clarify why the results of cGate in CartPole and Mujoco Ant are not provided? 2. The paper attributes cGate to [13]. However, I did not find any reference to cGate in [13]. Could the authors clarify this? 3. Line 193 says *"For this and the next section, we consider only the Concat and Unaware models as baslines"*. However, the results for the unaware policy are not reported in Figures 4 and 5.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The limitations of the approach are discussed extensively in Section 8 and Appendix G.
Summary
The paper studies how to better incorporate context (such as MDP-specific parameters) in context-conditioned robot policies. Traditionally, this is done simply by concatenating the context vector to the state vector and processing both inputs simultaneously with the policy network. This work proposes a context-conditioned hypernetwork to output the weights for an adapter module, which adds a residual value to the policy’s intermediate representation. Along with other proof-of-concept tasks, the paper evaluates on cartpole (varying pole length) and mujoco ant (varying mass) and finds that the proposed method is more robust to distractor dimensions but does not necessarily perform better than naive concatenation methods.
Strengths
The authors study a problem that is important in multitask learning/generalization such as with respect to language/goal-conditioned policies. The paper is well-written and proves some interesting theoretical results. Empirically, experiments test generalization in a meaningful way. For instance, the evaluated mass ranges are not within the training ranges in the ant environment, which is a challenging problem in robotics.
Weaknesses
(1) Section 4 talks about performance guarantees for a context-unware policy. Why is this important to the rest of the paper, which is about how to best incorporate context into a policy for dynamics generalization? Furthermore, I don’t see the link between Lines 154-163 (talking about how context-dependent rewards are equivalent to context-dependent dynamics in a context of matrix rotation example) to the rest of the section or the rest of the paper. (2) The abstract promises to “establish conditions for when it is necessary to incorporate context,” but when reading the text, I did not see any section of the paper that brought this out explicitly. (3) The work would really benefit from architectural ablations, especially when the abstract said the paper would “examine how to best leverage this external context information to improve generalization.” Hypernetworks are not a common architectural component in context-conditioning, so were alternatives not involving hypernetworks considered? For instance, it would be good to compare with the following architectural results: (i) The proposed approach, with no hypernetwork, where the adapter module (a simple MLP) only takes the context as input. (ii) Something like cGate, but with an elementwise product with the context embeddings at every layer, instead of just at one layer. (4) Figures 4 and 5, which contain the most difficult domains examined by the paper (ant and cartpole), show that in the 0 distractor dimension case, naive concatenation performs roughly the same as the proposed method (adapter). This means that in this case, naive concatenation is preferable due to its simplicity and training speed relative to the proposed adapter. It is only in the differential equations domain, a non-control task, where adapter clearly outperforms naive concatenation.
Questions
(1) How might this framework be improved to incorporate multiple contexts at the same time? For instance, both a goal image and language instruction for a given task. If it is known that some context modality is more beneficial on average to inducing strong task performance than some other context modality, how might we handle that with adapters and hypernetworks? (2) I don’t understand how Theorem 4.5 relates to the rest of the paper, or even how it guided the experimentation toward the proposed adapter + hypernetwork architecture. (3) How would the proposed method perform in image-based domains where the main policy network is a CNN or transformer? It seems like the main results are working with an MLP operating on robot state. My main concerns are that this work may be useful primarily for simpler, older architectures and might not prove as helpful for transformer-based robotics policies that are moving toward cross-attentional context conditioning (such as VIMA [Jiang et al 2023]). (4) Perhaps include a citation of FiLM (Perez et al 2017), since it is a widely used context conditioning method and does a sort of context-conditioned “adapter” with a residual connection as well, which is similar to the proposed method but does not have a hypernetwork. A comparison would be helpful too. (5) Most of the crux of the experimental section hinges on the argument that the adapter is a better context conditioning architecture because it is more robust to distractor context variables. However, the distractor variables were set to some pretty contrived values--all to 1 during training and 0 during testing. How were these values chosen? Why not just gaussian noise during both training and testing, which seems more likely for a model to encounter, in which case concat will probably be as good as an adapter at filtering out the distractor dimensions?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Authors were upfront of the limitations of the adapter to noisy contexts and its tendency to overfit when trained on narrow context ranges and had a very detailed limitations section.
Summary
The paper presents an architecture that uses a hypernetwork to learn network parameters for a context-dependent policy adapter. The authors discuss an illustrative example that emphasizes in what cases a universal (context-dependent) policy is necessary, and in which cases a non-context-conditional ("unaware") policy is sufficient. In the experiments, they compare their adapter approach with a context-unaware policy, a policy that concatenates context information with state and processes them equally, and other approaches using a linear policy adapter (FLAP and cGate).
Strengths
The idea to hyper-learn a nonlinear context-dependent adapter as the final few layers of a context-general policy is original and makes sense. The paper is well written, and experiments are quite thorough. The relatively detailed ablation experiments, some of which are found in the appendix, are quite interesting and a strength of the paper, for example the study of nuisance context parameters in section 7.2. I also think that the distinction between extrapolating and interpolating generalization is important and rarely seen (presumably because this distinction is usually hard to make in more complex environments), and interesting enough to maybe move it to the main part. General policies are an important and active research field, and the paper makes a significant contribution.
Weaknesses
I appreciate the illustrative example for unaware vs aware policies in section 4, but I feel like this is quite intuitive for many readers, and could potentially be shortened a bit in favor of more experimental results in section 5. Also, I was a little confused by the section title being "Theoretical Foundations". I think that the related work section should probably discuss recent advances in foundation-model-style generalist agents (e.g. https://arxiv.org/abs/2203.11931, https://arxiv.org/abs/2305.10912, https://arxiv.org/abs/2301.09816, https://arxiv.org/pdf/2306.11706.pdf) as well. The authors briefly touch on "learning some unsupervised representation of the problem and inferring a latent context from a sequence of environment observations", which is similar in spirit to this line of work, but then only cite one reference. Considering that this field has been quite active lately, it probably would make sense to discuss this briefly. Generally, I think the paper would benefit from experiments in a larger and more diverse number of experiments. However, I think this is at least partly compensated by the fact that the authors report relatively detailed results for the two (or three) environments they consider. Minor comments: - The shaded areas in Fig 3 and elsewhere are very hard to read
Questions
- In 6.1: Would it not be possible to calculate the average evaluation reward only on an entirely unseen set?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors discuss limitations in quite some detail in section 8, and underpin this with additional experiments in the appendix. This is a strength of the paper.
Rebuttal Response (Increased Score)
Thank you for the thorough response! My concerns have been sufficiently addressed and I appreciate the new figures that the authors made available. I agree with the authors that they struck a good balance between thorough breadth and depth of experimentation and am not asking or further experiments on CARL environments. I was simply curious why exactly those environments were chosen. I had a look at the code and, as stated in the original review, *I have increased my score*. I am very happy with the paper and am of the opinion it should be accepted. I believe it is a valuable contribution to NeurIPS. I do have additional questions after reading the other reviews. In particular, my questions are along the lines of Reviewer sPQr's Question 5. How do you expect the proposed method would handle the environments of Biedenkapp et al. (2022) (also in comparison to cGate)? Do you believe a hypernetwork would make it easier to deal with such noisy/partially incomplete features? Lastly, would it be a problem for the hypernetwork to have vastly more context features than state features?
Response to Reviewer's Additional Questions
We thank the reviewer for their kind comments, and time spent considering our work. Regarding the questions: > How do you expect the proposed method would handle the environments of Biedenkapp et al. (2022) (also in comparison to cGate)? That work is a particularly interesting application of RL. We believe there is no fundamental reason why our approach would not work in this case. Indeed, since some of the features are noisy or not fully informative, our distractor results suggest that we would perform better than either a concatenation-based approach or cGate. That being said, in our we focused our efforts on the continuous-control setting, which is distinct from the discrete policy selection discussed in Biedenkapp et al. (2022) and its related works. Investigating this more in-depth in future work would therefore be promising. > Do you believe a hypernetwork would make it easier to deal with such noisy/partially incomplete features? As mentioned, we believe that the hypernetwork is particularly well-suited to dealing with these cases where the context variables are potentially distracting. However, as noted in our limitations section (particularly in Appendix G.1), if the context carries more noise than signal, the adapter model's performance suffers. Therefore, while the adapter is quite robust to distractor dimensions, future work can investigate ways to improve its robustness to very noisy contexts. In particular, one promising avenue would be to more deeply investigate the benefits and potential drawbacks of training in the presence of context noise. This could be expanded to consider a curriculum of noise, starting with a small amount of noise and progressively increasing this as the model learns. In addition, training on a broader set of contexts and associated dynamics (e.g., via domain randomisation approaches) could also improve robustness to noisy contexts and would merit study as well. Then, regarding incomplete features, where it may be that the context does not fully describe the environment, or where we cannot perfectly distinguish between separate settings using only the context information: This case is effectively on a continuum between two extremes: (a) A completely context-unaware setting, where we have different dynamics but no way to distinguish between them; and (b) the case we primarily consider in our work, where context does provide perfect information. This perspective suggests that the hypernetwork-based adapter module may be promising, as it processes the state features in a similar way to an unaware model, but modulates its behaviour based on the context. Therefore, ideally, if the context provided no information, it would reduce to an unaware model (by effectively implementing an identity function in the predicted adapter weights). Our intuition is that in cases where the context is more useful, but not perfect, the hypernetwork can still partially modulate the behaviour of the agent to take this information into account. Future work could consider this in more depth; one way would be to have a large number of context features (that are all relevant and collectively provide perfect information), and investigate the effect of systematically hiding more and more of these from the agent. > Lastly, would it be a problem for the hypernetwork to have vastly more context features than state features? This is one aspect that makes the work of Biedenkapp et al. (2022) quite interesting; there are only 5 dynamic features (i.e., state) per open list but 305 instance features (i.e., context). While we have not explicitly considered a case like this, we believe the hypernetwork will perform well for the following reasons: 1. Empirically, in the 1D ODE domain, our distractor results (which are very similar to the other two domains) considered up to 100 distractor dimensions; totalling 1 informative context dimension, 1 state dimension and 100 uninformative context dimensions (resulting in a similar ratio to Biedenkapp et al. (2022)). The adapter performed admirably and could successfully solve the task, and generalise to unseen instances. 2. Due to the adapter's architecture, where it explicitly separates the state and context features, we believe that it would better deal with cases where there are significantly more context features. This is because the state-based features are not immediately combined with the context-based ones (potentially running the risk of the model focusing on the many context features as opposed to the few state features). One potential problem in this case is overfitting, especially if we only have access to a small set of environments, and therefore very sparse coverage of the (large) context space. However, this problem is not unique to hypernetworks and would likely affect many different models, including concat and cGate. We believe these points merit discussion in the revised paper and we will make an effort to include them.
Response to authors
I would like to thank the authors for addressing my concerns. After reading the rebuttal I still have a few questions: - *Authors: "cGate is able to learn on the trained contexts and the appropriate function is in the hypothesis space of the cGate model".* **Response:** If the underlying function is in cGate's hypothesis class, how do you explain the poor performance of cGate on ODE? - *Authors: "We hope that we have clarified the relationship between cGate and our hypernetwork, as the hypernetwork is more than just cGate with a nonlinearity."* **Response:** Line 196 in the paper says: *"Finally, we note that our adapter architecture here is a more powerful generalisation of another network architecture introduced by prior work, cGate [13]. Intuitively, our model uses a nonlinear neural network to process the state based on the context, whereas cGate uses only a single, linear, elementwise product."* Could the authors clarify in what way the use of a hypernetwork is beneficial other than by being nonlinear? I kindly ask the authors to be concise in their response. The rebuttal was quite difficult to parse.
We thank the reviewer for their time spent considering our work and their engagement in the review process. While endeavouring to respect the reviewer's request for concision, we acknowledge that there are several helpful perspectives to take to answer these questions. We will attempt to be brief and urge the reviewer to continue this discussion if in our brevity something remains unclear. > If the underlying function is in cGate's hypothesis class, how do you explain the poor performance of cGate on ODE? Figure 3 shows the overall performance of the agent across the entire context range. Importantly, this contains contexts that the model did not train on, as the goal of our work is to improve context generalisation. Thus, the poor performance of cGate in Figure 3 (left) is due to its failure to generalise and *not* because it cannot fit the seen contexts. To support this and explain the poor performance we next direct the reviewer's attention to Figure 10 in the appendix. This shows the same information as Figure 3 (left), but separately considers the models' performance on training, interpolation, and extrapolation contexts, respectively. Here, we note that cGate performs perfectly on the contexts it trains on - confirming that the ODE domain is within the hypothesis space of cGate - but fails to extrapolate. Interpreting these results and the improvement afforded by a hypernetwork, cGate's inductive bias may be responsible for this poor generalisation. > Could the authors clarify in what way the use of a hypernetwork is beneficial other than by being nonlinear? We emphasise that cGate is **not** a hypernetwork. Rather it is an implementable architecture **by** a hypernetwork. This is an important but subtle point which we contribute to the literature. Thus, while also allowing for the addition of non-linearity, hypernetworks are also likely to have different inductive biases and training dynamics compared to cGate, which could also be the source of the improved generalisation. Put simply, while our model could implement the same function as cGate, it does not appear to. Furthermore: - As we have empirically demonstrated in this work, our model displays superior generalisation performance and improved robustness to distractor variables compared to cGate. This alone indicates that the model has meaningfully different behaviour which cannot be merely attributed to the presence of non-linearities (not without extensive future works). - Indeed, other fields such as NLP [1,2,3] and Computer Vision [4] have also identified and begun to explore the benefits of hypernetworks and adapter layers, which further grounds our approach. Yet much work remains in understanding neural network generalisation broadly, let alone the incorporation of something as dynamic as a hypernetwork. Further to this, our more general architecture could easily be modified to incorporate more sophisticated architectural designs with different inductive biases, such as convolutions. We hope this has clarified these aspects, and we are committed to engaging further in this discussion if the reviewer has any additional questions. References [1] Neil Houlsby et al. Parameter-efficient transfer learning for NLP. ICML, 2019. [2] Rabeeh Karimi Mahabadi et al. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. ACL/IJCNLP, 2021. [3] Haoxiang Shi, Rongsheng Zhang, Jiaan Wang, Cen Wang, Yinhe Zheng, and Tetsuya Sakai. Layerconnect: Hypernetwork-assisted inter-layer connector to enhance parameter efficiency. COLING, 2022. [4] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. NeurIPS, 2017
Completing discussion on this paper
Dear Reviewers, Thank you all for your reviews and also for those of you who have engaged in the discussion already. If you haven't already, please take a few minutes to read the author responses and let them know if your concerns are addressed or you would like further discussion. All discussion has to be wrapped up by Monday. Thanks, Your AC
Response to Rebuttal
Thanks very much for your effort in your response. I find your response to weakness 3 and 4 to be great, and thanks for running those additional experiments. I also find your answers to questions 1 and 3 to be reasonable approaches. My main remaining concerns are about linking the theoretical part to the experimental part of your paper in a much better way. It's a bit clearer to me after reading your rebuttals, but for example, lines 153-162 are still really hard for me to connect to the thesis of the paper even after I have read it several times again after your rebuttal. I appreciate that you are willing to improve this in the revision. For the distractor experiments, I still think it's not great to be using 0's and 1's for distractor variable values. I take your point about wanting the distractors to have different values between training and testing, but such well-structured binary values for distractor variables may cause weird things to happen in continuous-valued neural nets, as they basically render parts of the network weights useless when multiplying by the weight matrix $W$ in $XW+b$. 1. Could you look into perhaps sampling from $N(0, \sigma)$ during training and $N(1, \sigma)$ during testing for the distractor variable values, where $\sigma = 0.2$ for example? That would create two (practically) separate distributions for the distractors during training and testing. 2. Do the advantages of the adapter hold over the concat method if there are 20 distractor variables and each of them is sampled from $N(0, \sigma)$ during both training and testing? I know I am pressing on this--it's just in my mind, I interpret 'distractor variable' as something that is irrelevant to the task context, not necessarily something that must be different during training vs testing AND is also irrelevant to task context AND has fixed values. I think having the answers to these two questions would be important as it would show to me clearly when a hypernetwork is important for context vs simple concat. Thank you very much for engaging in a good discussion.
We thank the reviewer for their engagement in the discussion process, and praising our responses to several of the weaknesses and questions raised in the original review. Regarding the theoretical linking, the core idea behind those lines is the following: 1. The theoretical section considers a case where the dynamics of the environment is consistent (regardless of context) but the reward changes depending on the context. 2. The experimental section considers the dynamics being affected by the context, and reward remaining consistent across context (noted in lines 79-80). 3. Therefore, we need a way to link the theoretical section (consistent dynamics and varying reward) to the experimental section (varying dynamics and consistent reward). Lines 154-163 formally make this link, noting that there is an equivalence between (1) and (2) and making it clear that the theoretical results apply to the experimental setup considered. As mentioned in the general rebuttal, we will move this to the appendix for readers that still seek such rigour, but we agree that it should not be in the main paper. We will also make it clear why this discussion is relevant, along the three points mentioned above. However, we reiterate that the theoretical section is necessary as it discusses the case for *when* incorporating context is necessary. Please see the general rebuttal for a longer comment on the necessity of this section. However, we hope the elaboration above further addresses the reviewer's concern and makes the link between sections clearer. > My main remaining concerns are about linking the theoretical part to the experimental part of your paper in a much better way On this point, we would be eager to discuss any suggestion on how to link the sections better before the end of the discussion period. Does the reviewer have any proposed improvements - both compared to our original version as well as to the proposed changes mentioned above and in our original rebuttal? We will gladly run the requested experiments where, instead of being set to 0/1, the distractor variables are sampled from a Gaussian distribution. However, since the discussion period ends tomorrow, we are unsure if they will be finished before then. Regardless, we can add these experiments to the revised paper. To clarify, for these new experiments, we sample new distractor values only once per episode. This is because we require the context to be consistent within a single episode. This strategy is in contrast to sampling new distractor context variables at each step, which would conflict with our original distinction between state and context variables. Having noted this, we still believe the distractor experiments we have in the originally submitted paper are useful. The fact that these variables have different values between training and testing is a good showcase of how robust models are. Furthermore, we believe this choice is very important, since we measure the ability of agents to **generalise** to new settings. It is quite feasible if the agent is in a scenario where the relevant context variables change, that the irrelevant ones may also be different to what it experienced during training. Then, regarding having fixed values, we believe this also makes sense in certain cases. For instance, if we consider a robotics task, all training could be done in specific rooms in one particular building where each room has different properties - such as the floor friction. In this case, irrelevant context features (e.g. humidity, air pressure, etc.) would correspond to our distractor variables and would remain fixed during the entirety of training (as training is only done within this one building). Then, if testing is in a completely different area, these irrelevant variables may be different from what was encountered during training. This is increasingly likely when we consider the life cycle of a robot where conditions will change over time. Thus, we still believe our experiments are demonstrative of very important real-world settings and use cases that are not contrived. We hope this further clarifies these points. Unfortunately, we may be unable to get all of the results in one day - however, we will endeavour to give the reviewer as much information on these experiments as possible by the end of the discussion. If the reviewer has any more concerns in the meantime we would be happy to respond further as the discussion period draws to a close.
Re: New distractor experiments
Thanks again for your prompt response! I know that the timeline is very tight, and I'm sorry that I should have responded to your rebuttal much earlier--I had not expected the implementation time to be more than a few hours for my proposed distractor experiments, but forgot that I do not know what your training time is like. No worries if the training doesn't completely finish in time by Monday; if you can show any preliminary results (like the first x timesteps, if that is usually a signal of how the methods perform relative to each other at convergence), that would be appreciated. (I do want to note for the record that I did propose this experiment at a higher level in my very first review in Question 5: > However, the distractor variables were set to some pretty contrived values--all to 1 during training and 0 during testing. How were these values chosen? *Why not just gaussian noise during both training and testing, which seems more likely for a model to encounter,* in which case concat will probably be as good as an adapter at filtering out the distractor dimensions? Re: fixed values, I agree that it does make sense in certain cases with env attributes that you mentioned, like friction. I was specifically dubious about the choice to use 0's and 1's specifically. Point taken that your method is better at generalizing to these 0/1-valued dimensions during testing, but it's unclear if this will be the case with other fixed values or other random-noise values. Thanks again for doing your best to address my concerns and answer my questions. I just wanted to get this comment out, and in my next comment (which I'll try to finish on 8/21), I can list out a few more concrete points for connecting the theoretical to the experimental part better after thinking a bit more about what you wrote above.
Additional Distractor Results
We again thank the reviewer for their engagement in the discussion process. We also thank them for considering concrete improvements to the linking; we will be sure to take these into account when preparing the revised paper. As a follow-up to yesterday's message, we have managed to finish running the first set of experiments, with Gaussian distractor variables. We summarise the results below, but these new results do not change our conclusions; in fact, they strengthen our claims that the adapter is more robust to distractor variables than the concat model. We thank the reviewer for suggesting these additional experiments. In particular: - We ran these experiments on the ODE and CartPole domains, but we can run these on Ant for the revised paper as well. - We trained for 300k steps, but the reward curves started to plateau around 100-200k steps, so the models had sufficient training time. - To retain parity with the results in the main paper (where the distractor during training was $1$ and during testing it was $0$), we set the training context to be $\mathcal{N}(1, \sigma)$ and the testing contexts to be $\mathcal{N}(0, \sigma)$; however, we can run the opposite as well for the revised paper if so desired. - We used $\sigma=0.2$ as requested. - We ran over 8 seeds instead of the usual 16 (due to computational/time constraints), but the error bars are not much larger than those in Figure 4. For the revised paper all experiments will be run over 16 seeds. While we have contacted the AC to determine if we are allowed to share the figures at this point, here we briefly summarise the results: ### ODE - The concat model's plots look similar to Figure 4 (left) in the main paper. In particular: - The concat model with 0 distractor variables performs comparably to 1 distractor variable, achieving a final AER (the metric defined in section 6.1, the same one used for all of the other plots) of around 140-150. - The concat model with 20 or 100 distractor variables performs much worse, with around 50 AER at the end of training. - The adapter model's plots look similar to Figure 4 (right) in the main paper. In particular: - The adapter performs comparably with 0, 1, 20 or 100 distractor variables. The AER at the end of training for 0 or 1 distractors is around 190. The AER at the end for 20 or 100 variables is slightly lower at around 160-170. ### CartPole For CartPole, we also have very similar results. In particular, as we increase the number of distractor variables we have, the concat model performs worse. - At 0 distractors, the concat model has AER=~450 - At 1 distractor, it has AER=~360 - For 20 distractors, AER=~280 - For 100, AER=~220 The adapter, however, displays much less of a dropoff. Each setting (i.e., 0, 1, 20 or 100 distractor variables) has a similar AER between 450 and 350. ### Summary Therefore, these additional results further strengthen the claim that adapters are more robust to distractor variables, even if these are sampled from a Gaussian. We hope these results alleviate the reviewer's concerns regarding our distractor variable experiments.
Additional results where the training and testing distractor distributions are the same
Dear Reviewer sPQr, We have contacted the AC regarding how best to share the figures for these and the previous new results, since we are not allowed to share links according to the NeurIPS guidelines. We hope to be able to share these soon but for the moment we will report our findings. We have run the second set of experiments, in particular, the noise is sampled from $\mathcal{N}(1, \sigma)$ both during training and testing. This is to be consistent with the experiments we had in the previous message, where training was done on $\mathcal{N}(1, \sigma)$. We again use $\sigma=0.2$. ### Results The results for both ODE and CartPole in this case are similar. Particularly, for both the concat and adapter models, there is not a significant difference between having 0 distractors and 20 distractors. For completeness, we also train on $\mathcal{N}(0, \sigma)$, as requested. In this case, for the ODE, we observe a similar result compared to $\mathcal{N}(1, \sigma)$ The adapter's performance with 20 distractor dimensions is similar to its performance without any distractors. The concat model seems to learn slightly slower when it has 20 distractor variables; its performance lags slightly behind the model without these variables. On CartPole, the concat model's performance with 20 distractors again lags a bit behind the results with 0. For the adapter, as noted in the main paper, an overly large signal-to-noise ratio may negatively impact the adapter's training (see e.g. Figure 12(c) in the appendix). In this particular experiment (CartPole, training and evaluating on $\mathcal{N}(0, \sigma)$), we notice a similar effect, i.e., that the variance is somewhat high for the adapter when training on 20 distractor dimensions. However, if we consider the generalisation gap, i.e., the difference between training and testing performance (which we are interested in, as our paper focuses on improving generalisation), the conclusions are similar to the ODE results, in that having distractor variables does not negatively impact the performance. ### Summary In summary, we find that if the distractor distributions are kept constant between testing and training, both the adapter and concat models perform similarly to having no distractors. However, we wish to note that this experiment does not test generalisation performance with respect to distractor variables. In particular, if the concat model suffers from overfitting (as seems to be the case in all of our distractor variable experiments), the overfitting is not demonstrated when we evaluate on in-distribution data (similarly to supervised learning, overfitting is only visible when evaluating generalisation performance). For these reasons, we believe these results could be added to the revised paper's appendix, and discussed to contextualise the distractor results - The adapter is significantly less susceptible to overfitting to irrelevant distractors compared to the concat model (this conclusion from the original paper remains unchanged, and is further strengthened by the results described in the previous message). However, if these distractors remain the same during training and testing (which may not be particularly likely), it does not matter very much whether the model overfits or not. We again thank the reviewer for their engagement, overall kind words in the responses, and suggestions to improve our paper.
Response to Author's additional Distractor experiments, Updated Score, Theoretical-Empirical Linking, and Further Suggestions
**Re: New distractor experiments that I requested.** I really appreciate the authors sprinting to get the gaussian noise distractors experiments in. I am glad that the proposed hypernetwork method is robust to $\mathcal{N}(1, \sigma)$ distractors during training and $\mathcal{N}(0, \sigma)$ during testing and believe this is a stronger and more impressive result than the fixed value setting in the paper. I have decided to update my score from 4 (weak reject) to 6 (weak accept) because most of the weaknesses and questions I listed in my original review were addressed to my satisfaction. I also reread portions of the paper and found lines 78-79: “Our goal is to train on a particular set of contexts, such that we can generalise well to another set of evaluation contexts.” It now makes sense to me why the authors really wanted distractor contexts that were different. However, I think that even if the distractor dimensions are of the same distribution but the relevant context dimensions are different between train and test, this still falls under the desired generalization problem setting. Thus, I still think that the train-test-identically-distributed set of distractors that the authors ran in the comment above is relevant for the paper. The authors' finding above, that concat and the proposed hypernetwork method perform roughly the same over increasing distractor dimensions, makes sense to me, and would help readers understand when to use concat and when to use hypernetworks, given hypernetworks are probably more expensive to train. **Re: Connecting Theoretical and Empirical parts of the paper better** The abstract of this paper addresses when to use context and how to architecturally do so. The theoretical section talks about when to use context, and the empirical section presents the best architectural method to use context, assuming that one decides to do context-conditioning. There’s a fairly strong separation between the two sections, unless this was intended. 1. For instance, the key result of the theoretical section was Theorem 4.5, mathematically stating in a toy MDP the performance bounds on how well a context-conditioned policy does compared to an unaware policy. Does anything in the experiments depend on Theorem 4.5? As far as I can tell, no, in part because it is so specific to the toy MDP (as reviewer DXBg has brought up separately). 2. In the experiments, there is also no attempt to try to quantify the $\alpha^{\pi}(s)$ ratio, which undergirds the theoretical section. This could be one potential way to connect the theory to the experiments better. 3. In my opinion, the paper’s biggest contribution is showing that a hypernetwork conditioning architecture can help policies generalize better in multitask settings. The paper’s experiments and rebuttal results showed robustness and generalization well, but neither of these two properties were brought up in the theoretical section. (I think this goes into the discussion of lines 154-163 again, as it seems to mention dynamics generalization, but as I said earlier, I still don’t understand these lines of the paper and trust the authors will clarify it in the final version.) This leads me to conclude that the paper may be as effective at conveying the main point (hypernetworks are a robust way to do context-conditioning with good generalization properties) even if the authors drop the theoretical section. **Finally, some visual suggestions for improving the paper:** - I would appreciate a better color scheme for the rebuttal graphs; three of the lines looked really similar to me: adapter, cGate, and cGateEveryLayer. (Especially people without perfect color vision, avoiding having red/green/brown in the same graph, where the lines are close to each other, would help a lot.) - Is evaluation time-consuming? Graphs would look better if they didn’t only have 6-7 points per line total. - Polish up figure 1 to make it more aesthetic.
Thank you, I maintain my positive score
I thank the authors for their detailed response. My initial concerns were only minor, and I thank the authors for adding the improvements they suggested. I maintain my positive score.
Thank you
We thank the reviewer once again for their consideration, assistance in improving the paper and overall endorsement of our work.
Decision
Accept (poster)