Identifying Selections for Unsupervised Subtask Discovery

Summary

This paper addresses offline subtask discovery from a causal perspective by identifying subgoals as selections, targeting at solving long-horizon tasks and acquiring transferrable skills. The algorithm design is well-motivated and shows superior performance in offline subtask discovery.

Strengths

(a) The causal-graph-based algorithm design is well motivated and theoretically solid. (b) The presentation is elegant and easy to follow. (c) Superiority of the proposed algorithm is shown through both quantitative and qualitative study.

Weaknesses

(a) Kitchen is a challenging benchmark, but each of its tasks consists of a sequence of subtasks. Consequently, subtask discovery from this well-structured offline data is relatively straightforward. It would be beneficial to demonstrate the subtask discovery capability of the proposed algorithm in tasks that lack clearly defined, semantically obvious subtask structures. (b) This work also focuses on multi-task learning, but it is confined to scenarios where the tasks in the set are merely different compositions of the same set of subtasks. More generalized multi-task learning can be future directions.

Questions

Please see the weakness part.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper studies the subtask decomposing problem. The paper proposes a formal definition of subtasks as the outcome of selections. The proposed seq-NMF is introduced to learn the subgoals and extract subtasks, conforming with the proposed theory. The experimental results show strong results on transferring to new tasks.

Strengths

- The paper is well-written and easy to follow. - The paper offers a formal definition of subtasks, to address the challenge of interpretability that has not been discussed in the prior works. - The insight of the definition of segmented subtasks should be consistent with the true generating process is novel and interesting. - The paper validates the proposed theory through the unseen long-horizon manipulation tasks and the results present a strong performance of the proposed method.

Weaknesses

- The paper only considers state sequences to extract the subgoals and subtasks. In real-world settings, the states are not available. So it is not clear if the proposed theory is also applicable to the real world with various modalities, e.g., visual perception. It would be nice to explore and discuss the possibility. - The generalization of the proposed definition of the subtask is rather specific and the shift of task distribution is not significant. So the generalization of the proposed method might be less convincing.

Questions

See weaknesses.

Rating

8

Confidence

2

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes, the limitations are discussed in the paper.

Summary

This paper studies the problem of decomposing expert trajectories (in the context of imitation learning) into sub-trajectories corresponding to subtasks. First, the authors introduce a causal framework to understand and explain what subtasks mean in this context. Subtasks are then defined to be variables that reduce uncertainty in the observed expert actions (or selections in the language of causal analysis). Motivated by this definition, a matrix factorization based task decomposition algorithm is presented. Experiments on multiple environments demonstrate that the algorithm is effective at discovering subtasks from expert trajectories.

Strengths

- There are many novel aspects to the paper. The authors provide an argument to view subtasks as selection variables and this insight is used to develop the task decomposition algorithm presented in the paper. The task decomposition algorithm also seems novel and the use of matrix factorization here is very apt. - Experiments presented in the paper indicate that the proposed approach is effective at discovering useful subtasks. The experiments also show that the subtask decomposition algorithm enables learning policies (in the context of imitation learning) that generalize well to new and unseen tasks. In this context of transfer learning, the proposed approach outperforms many existing state-of-the-art methods. - I believe that the paper offers interesting insights relevant to the NeurIPS community and RL researchers working on hierarchical and compositional RL. These insights have the potential to inspire new directions of research.

Weaknesses

One primary weakness is that some parts of the paper lack clarity. For instance, the mathematical objective in Equation (1) is not very clear although the overall idea and intuitive definition is clear from the text. Defining the notations early on in the paper would make it much easier to read and understand.

Questions

- In the second line of Equation (1), should the "forall" and "exists" terms be switched? Should it read as "for all sub-trajectories, there is exists a subtask..." ? - The use of convolution in the matrix factorization algorithm suggests that it allows multiple subtasks to be "active" at a given step. Could you provide some intuition behind this?

Rating

8

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

I don't think there are any major limitations that need to be discussed in the paper.

Response for the authors

I appreciate the authors for their efforts in additional experiments and explanations that address my concern. - The authors clarify the gap from real-world setup and provide a potential solution. I am convinced that the paper can be a first step in applying its theoretical framework for subtask discovery and further inspire work toward real-world subtask discovery. - The authors strengthen the verification of generalization by testing the longer horizon of tasks. After reading all the reviews, which I believe have been well addressed by the response from the authors, I am willing to raise my score to strong accept.