SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions

Summary

The authors present SkiLD, an unsupervised RL method that learns skills by augmenting DIAYN reward with a graph-state dependency reward to induce meaningful changes in object interactions.

Strengths

**Experiments:** Experiments are performed on a reasonably comprehensive set of 10 downstream tasks in 2 domains, and demonstrate that SkiLD is significantly better than all baselines. **Novelty and Idea:** The idea of using explicit factors as skill learning reward is interesting and novel. At a high-level, it’s also pretty intuitive.

Weaknesses

**Clarity: Quite a few clarity issues** - In the main paper, the method could be presented more clearly at a higher level. For example, introducing the Graph-Selection policy first (in 3.1 instead of 3.2) could help with understanding, at a higher level, what the method is trying to do. Currently the paper first talks about the skill selection policy but, by the time the reader gets to 3.1, it’s not fully clear yet (it’s mentioned at the top of section 3 but then gets into details a bit too suddenly in 3.1) that there is an initial unsupervised skill learning phase and what that phase is trying to do. Going from high level idea → high level policy → low level policy would make this more clear. - More explicit details are needed. How is the dynamics model trained? What’s the high-level overview of the algorithm? When and how are diversity rewards applied? What exactly are the input/outputs features of the graph selection policy? All of this could be more clear if the authors provide some pseudocode linked to from the main paper and beef up the appendix. - High-level details about environment assumptions (learned/given dynamics, what is given in the graph factorization, etc.) in section 4.1 should be given, even if details are already in the appendix. **Experiments:** - Are 10M timesteps really needed even in Mini-behavior domain? This seems extremely sample-inefficient. Same with requiring up to 5M timesteps to learn downstream tasks reasonably in these environments. Perhaps some details about why all methods need so many timesteps would be helpful. - How are hparams selected and tuned? And for the baselines? How do we know this is a fair comparison? **Minor issues:** - L163: missing end of sentence period - Fig 4: CaSk?

Questions

One limitation of all unsupervised skill learning methods is that a significant portion of the learned skills are meaningless. Qualitatively, what proportion of the skills ended up being meaningful? Was it more with SKilD than baselines like DIAYN? This perhaps could be visualized by randomly sampling ~50 skills and visualizing them to see which ones are actually meaningful.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Adequately addressed.

Summary

In the presented paper the authors propose a novel skill discovery method called Skill Discovery from Local Dependencies (SkiLD). The method utilizes the concept of local dependencies to incorporate the interaction of factors in a factorized state space. A novel intrinsic reward signal is introduced to guide the skill discovery. In the skill learning phases, different skill-conditioned policies are learned for the skills and in the task learning phase these (frozen) skills are used to learn a task policy solving more complex downstream tasks. The method is evaluated in two simulation environments and compared to several baseline methods. --- **Post-Rebuttal/Discussion**: I appreciate the discussion with the authors and their effort in clarifying open questions, my main criticism wrt presentation and evaluation are not solved though, and I remain my score of 5.

Strengths

The paper tackles an interesting, challenging and important research direction: equipping agents to acquire skills autonomously and in the absence of a concrete task that can then be used in given tasks. The main idea behind the method – utilizing local dependencies/interactions (realized with dependency graphs) and diversity – is well motivated and introduced. Overall the paper is written well and understandable, notations are solid, and the figures are of good quality (especially Figure 2 gives a good overview of the method). The explicit formulation of the two main research questions is appreciated and gives a good structure and focus for the evaluation. Similar, the explanations of baselines and especially what the respective comparison highlight is a very nice. (Explanations of differences to the different related work sections as well). Additional ablations show the effect of the main idea.

Weaknesses

My main concern is related to the use of overexaggerations and mismatch in some claims and with respect to the experimental results. For example, ‘…resulting in a robust set of transferable skills’, there is no evaluation or similar regarding robustness (in fact that word only occurs in that claim). Or more prominent, the domains are introduced with ‘…a LARGE number of state factors.’, while the state factors range from only 3 to 6 (Section 4.1). While mentioning the research questions is good, the chosen evaluations do not fit these questions very well. Q1 is about the diversity of the interactions, but Figure 4 then only shows a comparison of some ‘particular’ (how are they chosen?) interactions. For a diversity comparison a general metric comparing all found/used interactions should be compared. The number of skills is also not shown/discussed or evaluated. Similar, Q2 is about more efficient downstream learning. Efficiency in terms of what exactly? The according evaluation in Figure 5 (and Section 4.4) measures the performance. Is performance used as a proxy for efficiency here? As there are many details, modules and methods, it is not completely clear what is assumed to be given/known, or how certain quantities are inferred/used. For example, the events of interest (every factor in just the next_state of the s,a,s’ tuple?) or the target dependency graph (a crucial parameter, and lines 157ff mentions HER for that, but not for all cases?). What exactly does the diversity parameter do? How is the exact flow of the skill learning phase (getting the target graph, the intrinsic reward, learning the policies, and multiple policies can be learned for each skill?)? Some additional effort in clarifying such details would greatly benefit the paper. Some critical details, like that for some setups the ground truth dependencies were used instead of the learned, are only provided and thus somehow hidden in the Appendix. Additional comments are given in the Questions. While computational demanding and, hence, understandable, 5 seeds/runs are quite low for comparing RL based algorithms (known problem in the community, although often neglected/ignored). Recent frameworks and metrics have been proposed tackled at these problems related to low number of runs and comparing overall algorithms performance [1]. It would be beneficial to add such metrics (e.g., performance profiles and IQM/IQR) to the paper for a better comparison of the proposed algorithm. ----------- [1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021)

Questions

How can the approach deal with varying number of objects? The policies depend on the states and the state-specific graph? Does the chosen representation allow for ‘infinite’ number of skills? How does the approach scale with increasing skill number? How does it effect the learned policies (skill and task policy)? How are the two learning phases (skill and task) orchestrated? Is each phase done once? Or do they take turns, if so, how? In the factored MDP, is the action space A defined over the full S, or does it take the factorization into account? L168: ‘…we utilize a factorized lower-level policy, where there is a separate policy for each factor.’ So each skill is realized with multiple policies? How is this modelled and especially used? How are skills then chosen, does this affect the action set of the task policy? Notation in Figure 4 (and same in text) is unclear, what does the ‘x,y -> x’ notation represent? Figure 1 seems to be never referenced? How are the skill policies modelled? Table 1 only give learning details, is it a neural network? What are its details? Same for other methods, the presented parameters are not complete. What does ‘state-specific’ dependency graph mean? Figure 6 misses the plot description (is it mean and std as well?).

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Limitations are discussed, focusing on the assumption of known state factorization and accurate detection of local dependencies. An additional (potential) limitation is related to the scaling of the approach wrt. the training (data, time, ..), the number of objects and skills (rather low currently), or the transfer of skills to novel objects.

Summary

The paper introduces SkiLD (Skill Discovery from Local Dependencies), an unsupervised skill discovery method. Unlike existing methods that focus on state diversity, SkiLD leverages state factorization to guide skill learning by inducing diverse interactions between state factors. This method is designed to be more effective in complex environments, such as household settings with numerous objects. SkiLD uses local dependencies to model interactions and introduces a novel intrinsic reward mechanism. The method is evaluated in several domains, including a realistic household robot simulation, demonstrating superior performance compared to existing methods.

Strengths

- The clarity, presentation and writing of the paper are great. - The problem of unsupervised skill discovery is an important one. - The paper presents strong empirical results, demonstrating that SkiLD outperforms other unsupervised reinforcement learning methods in various challenging tasks. - The experimental setup is properly designed (i.e., right choice of baselines and domains).

Weaknesses

Minor weaknesses: - The method's reliance on accurately detecting and modeling local dependencies adds a layer of complexity that may limit its applicability. - The effectiveness of SkiLD hinges on the availability of a factored state space, which may not always be available or easily obtainable.

Questions

The two points raised within the weaknesses limit the applicability and thus the impact of the paper. Do you have insights on this? Especially, how the method can be made more general? Nitpicking: > L88: Formally, for an event of interest $Y$ and its potential causes $X = (X^1, ..., X^N )$, given the value of $X = x$, local dependencies focus on which $X$ is are the state-specific cause of the outcome event $Y = y$. I think the vocabulary used in this sentence is wrong. - $Y$ is not an event but a random variable. - $Y = y$ is an event. > L108: In this work, for a transition $(\mathcal{S} = s, \mathcal{A} = a, \mathcal{S}^\prime = s^\prime)$... If I understand correctly the notations, this shouldn't be $\mathcal{S}$ and $\mathcal{A}$ but $S$ and $A$, because $\mathcal{S}$ and $\mathcal{A}$ are the state space and action space and are not random variables. Plus, you said in the Background section that "In this paper, we use uppercase letters to denote random variables and lowercase for their specific values...". Same remark for L149-150. > In this section, we describe SkiLD, which enhances the expressivity of skills using local dependencies. SkiLD represents local dependencies as state-specific dependency graphs, defined in Sec. 2.2. State-specific dependency graphs are actually not defined in Section 2.2, there is just a sentence at the end of the section. It would be useful to better explain what are these graphs.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations.

Summary

The paper introduces SkiLD, a novel method leveraging state factorization to guide skill learning in unsupervised reinforcement learning. SkiLD emphasizes learning skills that induce diverse interactions between state factors, which are crucial for solving downstream tasks. The authors demonstrate that SkiLD outperforms existing unsupervised RL methods through empirical validation.

Strengths

1. **Novel Approach**: SkiLD introduces a unique approach to skill discovery by focusing on local dependencies and interactions between state factors, addressing the limitations of state diversity methods. 2. **Empirical Validation**: The effectiveness of SkiLD is demonstrated through experiments in various environments, showcasing its superior performance compared to baseline methods.

Weaknesses

1. **Assumption of Factored State Space**: SkiLD assumes access to a factored state space, which may not always be available or easy to obtain in real-world applications. 2. **Evaluation on Limited Domains**: The evaluation domains are somewhat restricted. Broader evaluation across more commonly used environments could emphasize the method's performance. Evaluating environments like Crafter, 2D Minecraft, or manipulation environments with multiple objects would better illustrate the benefits of focusing on local dependencies. 3. **Scalability**: There is a need to explore how SkiLD scales with an increasing number of state factors, which is crucial for practical applications.

Questions

Please refer to Weakness

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to Weakness