Do's and Don'ts: Learning Desirable Skills with Instruction Videos

Summary

The paper introduces "DoDont", an instruction-based skill discovery algorithm designed to learn desirable behaviors and avoid undesirable ones through unsupervised skill discovery (USD). The method uses instruction videos to train an instruction network that distinguishes between desirable (Do’s) and undesirable (Don’ts) behaviors. This network adjusts the reward function of the skill discovery algorithm to encourage desired behaviors. The authors validate their approach through experiments in complex continuous control tasks, demonstrating that DoDont can effectively learn desirable behaviors with minimal instruction videos.

Strengths

- The integration of instructional videos into the USD framework is innovative and addresses the challenge of learning desirable behaviors without predefined reward signals. - This paper stands out for its practical value, offering a USD algorithm that effectively learns meaningful and complex behaviors rather than merely generating skills that are variations of simple actions/jittering. - The paper provides thorough experimental validation on three tasks, showing that DoDont outperforms state-of-the-art methods in learning complex and desirable behaviors. - The presentation, writing and clarity of the paper are great.

Weaknesses

- The instruction network is trained using in-domain video data, which might not always be readily available in real-world scenarios, but can be fairly easy to obtain.

Questions

- What are the specific characteristics of the instruction videos required for effective training? For example, do they need to be of a certain length, resolution, or context? - How does the quality and clarity of the instructional videos impact the performance of the DoDont algorithm? - In Section 2.2, alongside DOMiNO, I think you should cite [1] and [2] that also balance a trade-off between intrinsic reward and task reward using constrained optimization. - I am not clear on the state space for each tasks. Do you use exclusively visual pixel inputs or do you combine the visual input stream with proprioceptive data? For example, the labeled consecutive states $(s_t, s_{t+1})$ are only pixel values or combined with proprioceptive data? [1] Skill-Conditioned Policy Optimization with Successor Features Representations\ [2] Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics

Rating

7

Confidence

3

Soundness

4

Presentation

4

Contribution

2

Limitations

Yes, the authors adequately addressed the limitations.

Summary

Unsupervised skill discovery is an RL task to learn interesting behaviors without rewards from environments. However, since there is no specification of desired behavior either, a lot of learning is wasted on acquiring skills that people may not be interested in eventually. The paper studies a setting where a few demonstration video are provided as a minimal specification of desirable skills. Then the proposed method can train a GAN like loss as a distance measure for unsupervised skill discovery, so the skill learned is more desirable. Experiments on dmlab are shown with interesting analysis.

Strengths

1. The paper is well written 2. The setup is well-motivated. It's indeed interesting to see something between complete unsupervised RL & imitation learning. 3. A lot of the ablation gives the readers good insights about the proposed method, which I find enjoyable to learn.

Weaknesses

1. The idea of combining unsupervised RL with some form of task specification is not new. In the very early days of unsupervised RL, people already add intrinsic reward and extrinsic reward together. I hope the authors could justify their novelty 2. While the authors discusses USD, I feel a lot of works in RL exploration are not mentioned. e.g. Curiosity [1]. There are other works about using very high level information as guidance, such as DeepMimic [2]. I am curious to see whether the proposed method can be combined with general USD methods? [1] Curiosity-driven Exploration by Self-supervised Prediction, Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell [2] DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills

Questions

1. What's the camera angle for "Quadruped"? Clearly the acquired skill is more diverse than instruction video as shown in figure 3. I am wondering how, because intuitively those trajectories are equally out of distribution.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The paper discusses the limitation of scaling to in-the-wild videos. I think this is reasonable address. No ethics concerns.

Summary

This paper proposes a method, DoDont, to avoid hand-crafting reward functions in unsupervised skill discovery. DoDont first learns a reward function from labelled instruction videos that discriminates desired and undesired behaviors, and then use the reward function in unsupervised skill discovery. The authors evaluate the method in several experimental settings and find DoDont can learn more diverse and safer skills than the baselines.

Strengths

Clear motivation that hand-crafting rewards in unsupervised skill discovery is tedious. The proposed method makes sense and is presented fairly clearly.

Weaknesses

I think the major weakness here is using a implicit reward function instead of explicit hand-crafted rewards has been explored before while those baselines are missing in the paper. DoDont learn the reward on some labelled instruction videos. There are several other approaches of automatic reward design in the past, for example, using a LLM [1, 2] or a vision-language model [3], which the authors should compare DoDont to and explain the advantages of DoDont. [1] Ma, Yecheng Jason, et al. "Eureka: Human-level reward design via coding large language models." arXiv preprint arXiv:2310.12931 (2023). [2] Kwon, Minae, et al. "Reward design with language models." arXiv preprint arXiv:2303.00001 (2023). [3] Fan, Linxi, et al. "Minedojo: Building open-ended embodied agents with internet-scale knowledge." Advances in Neural Information Processing Systems 35 (2022): 18343-18362.

Questions

1. According to Appendix C.1, the Do videos for DMC tasks is collected from a policy that's trained on ground truth reward functions. Does this mean DoDont still needs hand-crafting "ground-truth" rewards for a new task? If so then DoDont is not solving the motivating problem of USD that it needs hand-crafted rewards. 2. From the same section, it seems the Don't videos are collected from random action rollouts. Will these be enough for learning a reward that can teach the model to avoid hazardous behaviors? For example, walking into a hole is a bad behavior, but random action rollouts probably won't touch this kind of trajectory because random actions won't make an agent walk in the first place. Therefore, the reward module hasn't seen such bad behaviors during training and probably can't learn to assign a low reward value to this behavior.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

See above.

Thank you for the response

I would like to thank the authors for the thoughtful response. I think my concerns about data are resolved. For the weakness, while the authors and Reviewer oJKL consider the related papers [1, 2, 3] having different objectives and thus not comparable, I still think at least one or two of them should serve as baselines to DoDont. The task DoDont is trying to solve is avoiding undesired and hazardous behaviors in unsupervised skill discovery and the main contribution of DoDont is avoiding hand-crafting reward functions in this task by introducing an implicit reward function that is learned on instruction videos. Since the main contribution of DoDont is removing hand-crafting reward function in this process and the listed papers also propose methods to replace hand-crafted rewards with automatic reward design, DoDont should compare to these methods to analyze what is the best practice for reward design in the area of unsupervised skill discovery. I see that the authors have some preliminary attempts on using LLM to generate rewards which I think is valuable and I encourage the authors to include more of such comparisons or ablations in a revised version of the paper. Given the additional comparison to LLM baselines and the concerns on data being addressed, I've increased my score to 5.

No opensources codes on website https://mynsng.github.io/dodont/

Dear Author, I read your paper and found it very interesting. However, when I visited the website (https://mynsng.github.io/dodont/), I couldn’t find any code related to the article. Might you have forgotten to upload the code? This seems inconsistent with the statement in the abstract on OpenReview and in the final version of the paper: “Code and videos are available at [https://mynsng.github.io/dodont/].” Do you plan to update the website and release the code in the near future?