Summary
Broadly, the paper addresses the challenge of learning multi-stage robotic navigation and manipulation tasks in simulation.
The paper frames several benchmark tasks as goal-conditioned RL (GCRL), and they present a novel method ("HUGE") for leveraging human preferences collected during learning. In particular, the preferences are used to train a goal-selector model that guides (biases) exploration. For policy-learning, they use goal-conditioned supervised learning on the collected replay buffer, without requiring a handcrafted task-specific reward.
Compared to [1], HUGE also uses hindsight relabeling to learn from a replay buffer despite a sparse reward, but it differs in that it biases sampling from the replay buffer based on human preferences. Compared to [11], HUGE also collects human preferences iteratively (intertwined with learning). However, HUGE doesn't use its collected human feedback to learn a reward or otherwise directly bias the policy. Instead, the feedback is only used to bias exploration. Compared to [17], HUGE also has an exploration phase that builds an archive of trajectories. However, HUGE chooses promising goal states and generates trajectories using its goal-condition policy, while [17] chooses promising start states based on novelty and can generate trajectories with various exploration policies (random, epsilon-greedy, novelty-based).
The authors evaluate on four simulated manipulation tasks, two simulated navigation tasks, and two real-robot tasks. They compare to several baselines including the related work mentioned above as well as PPO (with dense and sparse reward variants).
Strengths
Their method doesn't require a handcrafted reward and it's able to leverage noisy, infrequent human feedback. These strengths are relevant and valuable to this domain.
Evaluation on a variety of simulated and real tasks.
Impressive crowdsourcing infrastructure and diversity of annotators.
Interesting, well-explained adaption of real-world tasks to fit their use of goal-conditioned RL.
Weaknesses
Lack of clear, fair numeric results on task performance across HUGE and the baselines:
= In Figure 5, I can only compare task performance after a given number of steps. A table with a performance number after training each approach to convergence would be more clear and would better support the claim from your text "HUGE beats prior work on the proposed long horizon tasks in simulation".
= It's not clear why some tasks were trained for millions of steps and others not.
= The paper text conveys that Figure 5's various approaches are using a poorly-tuned reward function ("we did not do any reward finetuning"). In particular, this makes the PPO results questionable.
= This section also uses synthetic human preferences derived from your reward function, instead of real human preferences. This confuses me and contradicts the earlier premise of HUGE. HUGE with this synthetic source of preferences would appear to be another RL method that utilizes a handcrafted reward function and not human feedback. This muddies your comparison with other approaches, especially [11] which is intended to be guided by real (not synthetic) human feedback.
= It seems like other baselines might benefit from the addition of behavior-cloning pretraining, as you did for "BC + Ours".
Questions
Can you improve your comparison to baselines based on the feedback I gave above? In particular, I am most interested in a robust, fair comparison to [11] ("Human Preferences"), because I consider that work the most directly comparable to yours: it doesn't require a handcrafted reward signal but it does require collection of human preferences intertwined with learning.
I hope you'll address the weaknesses in your PPO baseline I mentioned earlier: train to convergence with a fine-tuned reward function. Figure 5 currently shows that HUGE (with synthetic preferences derived from a reward function) outperforms PPO using the same reward function. If this result holds after you improve the PPO baseline, can you analyze this? Would it mean the synthetic variant of HUGE is fundamentally a better optimization algorithm compared to PPO?
I notice your two real-world tasks are fully-observable. How would you apply your approach to a real-world task that is only partially-observable, such as room-scale manipulation with visual perception?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
No concerns here