Accelerating Exploration with Unlabeled Prior Data

Summary

This work presents an off policy RL algorithm that use prior unlabeled data in a novel manner. The mechanism use an "optimistic" reward labeled based on a UCB estimate, that it is practically an iterative RND full update of the unlabeled dataset based on a stepwise exploration update of the RND functions. This reward of the unlabeled data smoothly decay making the agent explore better in comparison to other methods.

Strengths

The paper is clear on explaining all the methods, algorithms and theory used. Has a in depth citation of the literature. Compare the methods to relevant methods. It also characterize well were the method stand with respect to related efforts. The results clearly surpass comparative methods in sample efficiency. It also re-use state of the art results (RLPD and ICVF ) in the project. It also clearly and unambiguously answer it three research questions. Novel method for reward labeling prior unlabeled data.

Weaknesses

The re-initialization of the reward model after 20k for the relocate task is puzzling, which imply that this algorithm need a more robust extension to deal with this situation. The situation is not clearly understood. Why the reset are really needed? How can the process automatic identify the needs for reset or change the interpolation dynamics of the algorithm too adapt to this situation. Seems that something is missing. It seems that this approach it is inefficient when processing or bad demonstration or subpar-demosntration in the sense of what we can derive from not able to learn from high dimensional data, and the need for ICVF to learn a compact state. Which is reasonable but also imply that it would been nice to have this analyzed in the work. The analysis of the quality and quantity of the unlabeled data, for example, as was done in (How to Leverage Unlabeled Data in Offline Reinforcement Learning. Yu et al 22) This might indicate that the method needs a certain type of unlabeled data to be successful.

Questions

Given the concern of the previous section, the research questions might have been extended to understand the quality of the unlabeled data. Can the authors help me understand if this is valid concern? Other comments were done in the previous section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

None.

Summary

The paper proposes a simple method to leverage reward-free data in order to accelerate exploration in sparse reward tasks. This is done by learning a reward model and an uncertainty model during online learning, labeling the unlabeled data with an upper confidence bound based on these models, and sampling from both the online data and the pseudo-labeled offline data during learning. It results in fast achievement of sparse-reward tasks in a number of domains.

Strengths

- The presentation is very well done. - The method is extremely simple and clear to follow, simple enough to implement just from the paper with some experience in the field. - The coverage results in Figure 2 are very strong (if not so surprising). - The simple descriptions of baselines in 4.2 are helpful. - The results against baselines are very strong, even when compared to a topline oracle (Figure 3, 4)

Weaknesses

- One of the core assumptions of the method is somewhat obscured, namely that the offline data must contain good coverage in order for learning to be accelerated, otherwise there would be a significant gap between the frontier of the offline data, and the reward that must eventually be achieved, which would still result in slow exploration. As a result, I'm not sure I agree with the phrasing in section 4.5 that this method has better exploration, thus it achieves higher reward. Good exploration in the offline data is a prerequisite rather in order for the method to work well. If that exploration were not achievable in the first place, we could not use the existing method to accelerate learning. In some sense, we must have a working policy in order to learn a working policy. - Related work comes at the end, in general this makes it difficult to contextualize the work done in the paper against prior methods, though details on the baselines help somewhat.

Questions

With respect to the first weakness above, it might be nice to have an ablation experiment that limits the frontier of the offline data so as to examine the performance in that setting . In particular this is easy with AntMaze, one can only keep transitions in the first few quadrants of the maze.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The discussion section (6) mentions limitations in the form of quirks in reward optimization on Adroit domain, as well as a hypothetical limitation due to stability in learning (which does not seem to manifest). But, as discussed in weaknesses, I think there is a large limitation in that this method as demonstrated requires coverage in the demonstrations in order to be feasible.

Summary

The paper propose a method exploiting unlabeled prior data (i.e. trajectories without the reward signal) for more efficient online learning. To this end, a reward model is trained (based on the online data) and the so-labeled offline data are utilised jointly with the offline data.

Strengths

The paper tackles an important problem of using prior data towards accelerating online learning. The experimental setup covers three different domains (AntMaze, Adroit, COG) and $7$ baseline methods, both of which make a solid impression. The presentation of the method is very clean, and the paper is easy to follow.

Weaknesses

I find the paper's analysis somewhat shallow with respect to the design choices and analysis. Below, I list a tentative questions and suggestions: - the paper does not study much how does the offline data distribution affects the training. I acknowledge that different domains have different characteristics with this respect, however, it is hard to conclude something meaningful here. - The conventional knowledge that the offline data should be treated with pessimism, here we assume optimism? How does these two relate? - Is the solution specific to the selected offline RL method? I understand that this can be done easily, but would it work? I'd be great to have some experiments or at least some arguments. - I'd be great to know more about the uncertainty modelling. How accurate is it? What is the best balance of the uncertainty bonus and the learned reward signal. I am aware of practical constraints, and I do not expect the authors to include all suggestion listed above. Nevertheless, I hope that including some of them would enrich the paper. ### Small - The number of seeds (4-5) is rather small for the RL experiments, but is perhaps excusable by number of environments and baselines. - I am not fully convinced by Sec 4.5. It states that the method improves the coverage (a proxy of the exploration performance). However, it is not completely clear, that this yields benefits to the online training. That is, I could imagine that there is some spurious correlation here. I admit, that I do not have any particular better idea, though. - some hacks are needed (e.g. resetting reward model) - I find Sec 4.4 (representation learning) a little bit detached from the main topic of the paper.

Questions

(see above)

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are mostly missing. In the discussion section, the authors admit that the presented approach is the 'initial instantiation' and least some problems. However, it is not clear if the presented list is comprehensive.

Summary

This paper proposes a method to leverage unlabelled offline data to expedite online exploration. The idea is to train a reward model and random network distillation models on online data and use them to construct a UCB estimate for offline data. By training an off-policy RL algorithm on both online and relabelled offline data, the agent is encouraged to visit novel states represented in the offline data that are deemed to have high rewards. This is in contrast with most online exploration methods which only expand the exploration frontier. The authors validate their method on state-based and visual domains and demonstrate that their method achieves more efficient exploration compared to baselines.

Strengths

- The method is simple and effective, providing an appealing means to leverage unlabeled offline datasets to facilitate online learning. - The experiments are substantiative, demonstrating the superiority of UCB rewards compared to pure online exploration as well as other exploration methods that utilize unlabelled offline data. It is nice to know that in addition to state-based domains, the method is also compatible with visual domains through pre-trained visual representation, which further expands its applicability. Moreover, the analytical experiments in Section 4.5 offer clear evidence that the reason behind improved online learning is efficient exploration. - The overall presentation of the paper is coherent.

Weaknesses

- The problem setting is a bit awkward. Since the offline and online data have to be from the same domain, this method does not benefit from Internet-scale unlabeled data. It makes sense if the offline data is collected from a diverse set of tasks with the same dynamics but different underlying reward functions, and the goal is to show fast adaptation to new rewards via efficient exploration. But in that case, it would have to compare with the unsupervised skill-learning line of work, e.g. DIAYN [1], RaMP [2], and I'm not sure if exploration can outperform adaptation in terms of sample efficiency. - The quality of the offline dataset is important. Not only would the data need to be from the same domain, but it would also need to cover interesting trajectories for the task to really facilitate exploration. If the data is from a completely orthogonal task then it might even hinder exploration, depending on the balance between the reward estimate and the uncertainty quantification. [1] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine. Diversity Is All You Need: Learning Skills Without A Reward Function. ICLR 2019. [2] Boyuan Chen, Chuning Zhu, Pulkit Agrawal, Kaiqing Zhang, Abhishek Gupta. Self-Supervised Reinforcement Learning that Transfers using Random Features, ArXiv Preprint.

Questions

- How does the quality of offline data affect performance? What happens when the offline data doesn't cover high-return trajectories? What happens if it contains orthogonal trajectories (e.g. the task is to go left, but the data goes right)? It would be helpful to see an ablation experiment with different dataset qualities vs. exploration efficiency. - It would be nice to include a comparison to an unsupervised skill-learning baseline. I expect UCB exploration to achieve better asymptotic performance than unsupervised skill-learning when the dataset does not cover optimal trajectories.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors adequately addressed the limitations of their method in the paper. They mention that sometimes naively fitting the reward to online exploration can lead to overfitting. Moreover, the UCB estimates can be volatile at the beginning of training.