Self-Supervised Reinforcement Learning that Transfers using Random Features

Summary

The authors propose a self-supervised method to learn approximate multi-step Q-estimates by leveraging random features as bases for a reward function in the target task. This approach addresses some limitations of model-based and model-free RL. In particular, model-based RL typically suffers from compounding error of predicting future states from predicted states. On the other hand, model-free RL tends to suffer from instability when transferring knowledge. The proposed method mitigates these problems by implicitly learning transition dynamics (without explicitly modeling future trajectories) from offline data in a self-supervised manner based on the prediction of random features.

Strengths

The methodology is interesting. Using random features in a self-supervised manner to form a Q-basis such that one can find a linear combination of these random features for the reward function of a downstream task using approximate multi-step returns makes sense and seems novel.

Weaknesses

The main weakness of this approach lies in the experiments and poor framing of how this work fits in the context of meta-RL. meta-RL is not mentioned, but it entails this same problem of having multiple tasks with shared dynamics such that the reward function changes from one task to another. As such, the authors should be comparing against meta-RL benchmarks, not standard RL. It is interesting because the authors are running their experiments on a popular meta-RL benchmark called Meta-World and yet they do not compare their approach against meta-RL approaches. One of the reasons why this is currently an unfair comparison is the need for the competing approaches to learn their value functions, etc. from scratch, whereas the proposed approach had the benefit of learning from an offline dataset befrorehand and adapting to the target task.

Questions

1. I am confused on your statement that the Q-function for multi-step returns is independent of the policy when H = \tauThe lim. You still have \tilde{a}_1, \tilde{a}_2, ..., \tilde{a}_\tau. From what policy do these actions come from? 2. Why do you not compare against meta-RL methods? The successor features approach you compare against may count as meta-RL (it is not called that in their original paper though), but it is a method from 2017.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The limitations have been addressed.

Summary

The paper introduces RaMP, an approach for fast adaptation to unseen reward functions given offline data collected with arbitrary behavior policies under the same dynamics. RaMP learns a set of basis multi-step Q-functions, each corresponding to a random reward defined as the accumulation of random state action features. For online adaption to new rewards, the new reward/Q function is then identified using linear regression given the basis functions and used for control via MPC. **Edit After Rebuttal**: Raised score from 3 to 5, see comment below for details

Strengths

RaMPs' key idea, learning "random feature rewards" which are later combined, is novel and interesting as it allows offline pertaining without "reward labels" and provides an easy and efficient adaptation mechanism. The paper addresses a significant problem, the generalization to novel rewards under little to no assumptions on the knowledge of the current scenario and previously seen rewards, and is mostly clearly written and easy to follow.

Weaknesses

The paper's main weakness is its experimental evaluation, which I think is in its current form insufficient to validate the paper's claims or allow assessment of the method's potential for several reasons: - The algorithm described in the main paper seems to be improved by several "implementation details" (c.f. Appendix B). While the paper states that "RaMP’s performance already surpasses the baselines even without them", I believe a rigorous analysis of their impact would be necessary to assess the method's potential and workings. - Baselines: none of the considered model-based RL methods (PeTS, MBPO, Dreamer) was designed to work with dynamics models trained on **offline** data. For a fairer comparison, methods designed to work with such dynamics models (e.g. MOPO[1]) should be considered. - While RaMP adapts very fast in the considered environments, its final performance is often lower than that of MBPO and/or PeTS, in particular for the Hopper and D'Claw experiments. Further discussion and analysis in the paper would allow a better understanding of RaMPs limitations. - Compared with most recent work in RL and/or offline RL the paper only uses a small set (8) of relatively simple environments - the maximal state dimension is 39 (meta world, where only a subset of these is relevant for the considered tasks) and maximal action dimension is 9. - Higher dimensional observations (pixels) are used for some experiments in the supplement but the results are again inconclusive or seem to favor the baseline (Dreamer-V1, which is to the best of my knowledge not SOTA on pixel metaworld [2]) - See questions below. While the majority of the paper is clearly written and understandable, this is, in my opinion, not the case for section 3.4. Here the assumptions, their realisticness, and practical consequences of theorem 3.1 could be stated much more clearly [1] MOPO: Model-based Offline Policy Optimization, Yu et al 2020 [2] Masked World Models for Visual Control, Seo et al 2022

Questions

- I am a bit puzzled by the meta-world return curves: If I recall correctly successful execution in most of these tasks corresponds to an episode return of ~ 4,000. However, all presented algorithms achieve max. 1,000 (on "Reach wall") and considerably less in the 3 other tasks. Is there some normalization here? I would prefer to (additionally) see the success rates for these tasks to improve the interpretability of the results. - What exactly is indicated by the error bars? How statistically significant are these results given that it's only 4 seeds? - Are the "implementation details" (Appendix B) also used for the baselines? In particular: Additional value function for infinite horizon (B.1) for PeTS, online adaptation (B.2, but adapting the dynamics) for PeTS and MBPO, Planning With Uncertainty (B.3) for PeTS and MBPO, and MPPI (B.4) for PeTS. - What are the standard "coverage and sampling assumptions" in theorem 3.1? How realistic are those in praxis? What does the theorem add, compared with standard random feature regression results?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

I believe all limitations have been addressed, although some only in the supplement (infinite horizon problematic with multi-step q function), and I believe the paper would benefit from moving this also to the main part.

Summary

The paper proposes an approach for unsupervised pre-training of RL agents, ie pre-training on offline agent experience without rewards. The approach generates a set of random reward functions and then learns a successor representation of the state-action space for predicting cumulatives of these rewards on fixed-horizon trajectories from the pre-training data. At test time, it uses a small reward-annotated dataset to learn a linear mapping from the pre-trained successor representation to the target reward. Then it extracts a policy by greedily maximizing the resulting estimates of future cumulative rewards. The paper demonstrates that this allows for faster finetuning than model-based RL since only a linear mapping needs to be learned on the target task data instead of the full Q-function. In contrast to prior work on successor features it does not assume access to given features or a reward-annotated training set to learn features, but instead uses random reward projections.

Strengths

- the problem formulation is impactful: pre-training with reward-free data has great potential for generalist RL agents - the paper is well-written and easy to follow, it clearly outlines the problem formulation, explains the method and provides theoretical justifications - the experimental evaluation seems comprehensive, with only minor experiments missing (see below): the paper compares to a representative set of baselines on multiple environments, including image-based control (though there with limited success) and also performs ablations of the key elements of the method

Weaknesses

(A) **Relation to prior work not sufficiently explained**: the proposed approach heavily builds on top of prior work on success features for RL, yet the current submission only mentions this relation in a half-paragraph in the main paper and adds a slightly more detailed discussion at the very end of the appendix. It would be helpful to more clearly introduce this most relevant prior work and explain the deltas, so it is easier for readers to understand the main novel components. Concretely, one could add a preliminaries section that summarizes the idea of successor features and then point out the two main novelties in the proposed approach: random reward projections to be independent of training tasks and action chunking to be independent of training policies. (B) **Random reward features don't scale well to images**: the main difference to prior successor feature works is that the proposed approach avoids the assumption of pre-defined features or given pre-training task distributions by using random reward projections. The downside of this is that the algorithm cannot "discard" any information and struggles on high-dimensional inputs like images, where predicting random rewards over pixel inputs is complex and lacks semantic meaning. I agree with the authors that this can potentially be mitigated with pre-trained representations, but since removing the assumption of pre-defined features is one of the main selling points of this work, this tradeoff should be discussed more prominently in the main paper (currently only in appendix C.3) and the corresponding image-based results should be included in the main text. It would also be helpful to show the successor feature baseline results on the image-based domain. (C) **Missing Ablation**: The other introduced novelty is the use of action chunking, ie conditioning the Q-function on a sequence of H actions instead of on a single action. However, this choice is not ablated. It would be good to include versions of the proposed approach without action chunking (ie H = 1) and with different horizon lengths to see the benefit of chunking. (D) **Missing baseline**: The paper compares to goal-conditioned offline RL (CQL) but with a pre-specified selection of goals. Prior work has instead proposed to use goal-conditioned offline RL on randomly sampled goal states from an offline dataset for pre-training [1]. This has two benefits: (1) it does not require pre-defined goals / tasks, fully matching the assumptions of the proposed approach, (2) because of that it may be more robust to test time adaptation to new tasks. Thus, it would be good to add comparison to Actionable Models on all environments. **Minor Point**: (E) **MPC is misnomer**: The paper refers to the downstream policy extraction as "model predictive control" and "planning" (see eg Fig 2). However it does not perform prediction or use a model. Instead it performs greedy policy extraction by maximizing the Q-function, but uses action chunking (ie N-step action inputs). Thus I would call this step "greedy policy extraction with action chunking" instead. [1] Actionable Models, Chebotar et al. 2021 ## Summary of Review Overall I like the paper and am happy to accept it if the authors can more clearly mention the connections to prior work on successor features, discuss tradeoffs for image-based domains and add the suggested ablation + baselines. The idea of using random reward projections for successor feature learning is novel to my knowledge, but I am not 100% certain and will thus assign lower confidence to my review.

Questions

N/A

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Limitations on image-based environments are demonstrated in the appendix but should be discussed more prominently in the main paper (see weaknesses (B)).

Summary

The paper tackles the problem of In the training phase, they create $H$ randomly initialized neural networks that serve as reward functions during the purpose of pretraining. The Q-function learned online is a linear combination of these random features: $w^T\sum_{h\in [H]}\phi(s_h,a_h)$.

Strengths

This is a beautifully written paper. The motivation is crisp and the proposed method is very well grounded theoretically. Results across a broad range of tasks are presented. On originality, clarity, and method quality, I rate this paper highly.

Weaknesses

The paper would benefit from more baselines: for example doing something like IQL with a reward of 1 at the end of each trajectory during offline training. Less importantly, I also think IQL would be a more natural oracle than CQL. It performs better than D4RL and directly evaluates online fine-tuning. In fact, the IQL paper shows that CQL can sometimes collapse during online fine-tuning. I also don't quite understand the choice of offline dataset: from the appendix, it sounds like the policies that generate the offline dataset are quite "expert." It would be interesting to see how well the model performs if the offline dataset was instead modified from the D4RL benchmark, which includes different levels of expert policies. I expect high quality data to benefit any self-supervised method, but the claims of the paper are not limited to offline learning on expert demonstrations.

Questions

I'm a little confused by Figure 2. The way that the random features and Q-basis are drawn makes it look like each $\phi_h$ is one of the basis vectors that together compose $Q-basis$. Everywhere in the text implies that each index of Q-basis is a different function parameterized by a different $\theta_k$ and together the $\theta_k$ make up the basis vector. Is that correct? If so I would recommend modifying Figure 2 left to include a reference to k. The number of environment steps in the online phase seems really small. Why was 3000 chosen and about how many trajectories is it? E.g., in the IQL fine-tuning experiments they use ~1M environment steps. Why did the authors not include D4RL data in their experimental evaluations?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

N/A

Summary

The paper proposes a Random Features for Model-Free Planning (RaMP) algorithm to solve the problem of learning generalist agents that are able to transfer across platform where the environment dynamics are shared, but reward function is changing. The proposed algorithm leverages diverse unlabeled offline data to learn models of long horizon dynamics behavior, while being able to naturally transfer across tasks with different reward function. The authors evaluated RaMP on number of simulation based robotic manipulation and locomotions tasks.

Strengths

- The problem addressed in the paper is very important problem for RL in robotics/robot learning real-life tasks. - The proposed method is very interesting and seems relevant to the research community.

Weaknesses

- The paper is not well-written and very hard to follow. For example, Line 13-15 and Line 59-61 is very hard to follow. - The paper attempts to solve an interesting problem and has strong results in simulation based robotics task. Since, the motivation of the paper draws from real-robotic tasks, it would be good to see how this method performs on real-robotic tasks. I understand authors have mentioned in future work but to understand complete effectiveness of the proposed method, it seems critical. -

Questions

Please address concerns mentioned in the weakness section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

mentioned in the paper.

Thank you for your rebuttal!

Thanks for providing the rebuttal. I am generally happy with the response and appreciate the authors willingness to incorporate the feedback. I also understand that the short rebuttal window can make it hard to implement new ablations / baselines on the spot, particularly if many seeds are required. I trust that the authors will follow through on their promised changes. Please also add comparison to baselines on image-based domains as mentioned in my review. I have also looked over the reviews of other reviewers specifically to understand the low ratings of reviewers 8qBK and gqpD. After reading through the reviews and rebuttals I find the criticisms rather surface-level and don’t fully agree with them: - compare to meta-RL (R-8qBK): agreed with the authors that meta-RL requires training tasks / rewards, which their method does not. There is work on unsupervised meta-RL though that could allow for clean comparison (Gupta et al 2018, https://arxiv.org/abs/1806.04640) - compare to IQL pre-training (R-gqpD): I don’t think IQL w/ final reward 1 for value pre-training is ideal on non-expert data — the proposed projection after random reward pre-training seems more general I believe the authors adequately addressed the concerns of these reviews so I stand by my accept recommendation and am willing to defend it to the AC. I will also increase my confidence given that no other reviewer raised concerns about too closely related work.

Updated experiments

Thank you for the updated experiments on the pdf. I have raised my score. Should the paper be accepted, please include these results and explanation of meta-RL and how you differ.

Thank you

Thank you for taking the time to address these concerns. Based on the response, I raised my recommendation to Accept.