Combining Behaviors with the Successor Features Keyboard

Summary

The author(s) present a novel and efficient approach to knowledge transfer in long-horizon tasks, aiming to minimize interactions with the environment (less frames required in training). Their proposed method leverages Categorical Successor Feature Approximator (CSFA) within an enhanced deep OK network, resulting in several tasks (e.g., Find, Place).

Strengths

- Proposed a dynamic transfer query that enables sharing among task-encoding and SF-approximators; - Implemented this methodology in long-horizon tasks.

Weaknesses

1. The authors did not clearly position their contribution in relation to other approaches commonly used in long-horizon tasks, such as hierarchical approaches, hindsight approaches and meta-learning approach [1-3]. 2. While the overall results of the experiments are significant (e.g., Figure 3 improve 0.2 ~ 0.8 in success rate), the description and documentation of some experiments might be unclear or insufficient. 1) USFA or MSFA might require fewer computational resources compared to MSFA, as their calculations appear to be simpler (line 246). However, without further detailed information, it is difficult to make a definitive judgment. I suggest that the authors include memory and computational load comparisons among these methods. 2) The author's comparisons are focused on MTRL, but there is a lack of comparison with other approaches mentioned in references [1-3]. It would be beneficial to include comparisons with other relevant approaches mentioned in reference [3]. 3. From the paper, it is unclear why MSFA is considered sharable or suitable for fitting different tasks. How it compare with other pre-training encodings or other method representations. 4. Some typos, e.g., "Transfer tasks are conjunctions of known known tasks", line (289) [1] Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors, NeurIPS 2020 [2] Learning to Reach Goals via Iterated Supervised Learning, NeurIPS 2020 [3] Skill-based Meta-Reinforcement Learning, ICLR 2022

Questions

see weakness and limitations

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

- This paper could benefit from an expanded evaluation of the transfer learning quality beyond just the success rate metric, e.g. adding visualizations of the baseline task encoding and dynamic query encoding to show the benefit of dynamic query is scalable. - This study lacks of discussion about the upper bound of the number of tasks it can effectively handle during sharing coding. The author primarily tests the approach on two main tasks ("find" and "place") with several sub-tasks or steps required to complete each main task. However, it remains unclear what the upper limit number of main task and sub-tasks?

Summary

This paper focuses on the Option Keyboard (OK) task which was recently proposed as a method for transferring behavioral knowledge across tasks. They propose a new method of Deep Option Keyboard (Deep OK), which enables transfer with discovered state features and task encodings. To enable discovery, they further propose the Categorical Successor Feature Approximator (CSFA), a novel learning algorithm for estimating SFs while jointly discovering state features and task encodings. With Deep OK and CSFA, this paper achieves the first demonstration of transfer with SFs in a challenging 3D environment where all the necessary representations are discovered. The experiment results show that CSFA can discover SFs compatible with SFGPI and Deep OK achieves strong jumpstart performance for transfer to combinations of training tasks.

Strengths

1. Transfering with SFGPI in a large-scale multi-task setting, discovering cumulants and task encodings, while sharing a task encoder and SF approximator across tasks gives some insights to the community and seems novel. 2. The paper’s writing is generally clear and the figures were helpful for understanding. 3. The problem formulation is clear and the approach is well-reasoned. 4. The experiments are sufficient and convincing to some extent.

Weaknesses

1. Though SFGPI is dynamic and can share a task encoder and SF approximator across tasks, its generalization ability is not clear and no more discussion can be found in the paper. Maybe a more general method that sampling queries from the task encoding space that is more fairly concentrated could be designed and at least provide some discussions. 2. In Figure 3, CSFA discovers representations compatible with GPI across short and long task horizons. The longer horizon is defined by just one “Place Near” task. Will CSFA outperform MSFA and USFA on other general tasks? 3. Deep Ok doesn’t reach an optimal performance but uses far fewer frames compared with Distral.

Questions

In Line 364, the author claims that “This may enable other methods that leverage SFs in a multi-task setting”. Since Deep OK can’t get the optimal performance, how can leverage SFs for discovering representations and get the optimal performance simultaneously？

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See the weakness and question sections.

Summary

This work focuses on attempting something which no previous work has done before; learning task encodings, cumulants and successor features (SFs). The author(s) proposed a solution based on learning these three functions jointly. They considered a model that learns the SFs as probability mass function using the two-hot representations inspired from MuZero. They named this model as the Categorical Successor Feature Approximator (CSFA). With the learned SFs, the authors will then be able to perform Generalised Policy Iteration (GPI) to obtain the policy for the relevant task at evaluation phase. In order to find a better policy for this task, the authors also introduced a method named Deep Option Keyboard (DOK), which relies on a parameterised function to output coefficients which are then used as preference vectors. The models are evaluated in a 3D environment, using pixel observations. The tasks included going to certain locations and collecting and then dropping objects at different locations. In particular, DOK is used to study the effects of transfer in sparse reward and long horizon tasks. The proposed model is able to perform better than the baselines such as Universal Successor Features and Modular Successor Feature Approximators.

Strengths

I find the work interesting as this is an attempt at a challenging problem of learning task encodings, cumulants and successor features jointly. The figures, in particular, figure 2, is clear in aiding the reader to understand the architecture of their model. The writing overall was mostly clear and easy to read. The experiment's descriptions are concise and helped me to understand the setup a lot even though I am not familiar with the playroom environment. The authors also performed ablation studies which showed that the key to their model is learning the SFs using a probability mass function, rather than the canonical expected values.

Weaknesses

Although there are no major weaknesses in my opinion, I do have some points for thought. Firstly,I am not sure it is a good idea to call the mechanism for learning the preferences, “Deep Option Keyboard” when it seems that there are no options involved. Secondly, it would also perhaps be a good idea to see how this model compares with the standard Q-learning algorithm, such as Rainbow or even just a distributional RL model since the proposed model uses a probability mass function to get an idea how much of the transfer benefits one could achieve from the utilization of the successor features. Thirdly, there was some mention of static vs dynamic query. Perhaps it will be helpful to give a clear definition of what it means by static and dynamic query.

Questions

Q1. The authors discretize the cumulant returns values using bins. Is there a study done to show how the different number of bins impact the performance of the agent? Q2. I understand that Bernoulli distribution was used as the parameterized probability function. What are the other distributions that were considered? Why did Bernoulli distribution do better than the others? Do you have any insight on this? Q3. On line 273, the authors mentioned that sharing an estimator across cumulants is necessary. What do you mean by keeping categorical representation? Is this successor features or state function or something related to cumulant network (with reference to figure 2). (Potential typos): Line 289: Known known tasks? Caption of figure 5: Distral and MTRL learn an ??? about the same speed. My guess is “an optimal or policy”?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

None

Summary

First, I must emphasize that I was new to the RL community before this NeurIPS. This is the first RL paper I have reviewed. I have been struggling to understand this paper. I tried my best. I hope the AC, reviewers, and authors may think me helpful. This paper studies how to transfer a pre-learned RL system to new tasks using successor features and the optional keyboard. Compared with previous methods, new things are as follows: 1) the Q-value is no longer linear about the task encoding, here, the authors use a network $g_{\theta}$ to predict the weight of SF features based of task encoding and state representation. 2) The SR approximation is modeled as the expectation of a discrete distribution, the authors claim this can help stabilize the training. 3) The learning objective is modified. The new objective consists of three parts a) a term to decrease the prediction error of reward using $\phi$, b) a term to decrease the prediction error of reward using the difference of $\psi$, and c) a term to maximize the possibility of current $\psi(a_t,s_t,w)$ to occur using the twohot operation. 4) A new transfer scheme that learns a new task encoder and state function at transfer time, where the task encoding is the weighted sum of a Bernoulli random variable and is aware of the time. The proposed method achieves better performance than USFA and MSFA in 3D playroom dataset, showing a better success rate.

Strengths

# Novelty Above all, I may not be able to judge the novelty of an RL paper. So a comment based on my experience in my familiar domain. 1. The new assumption of SF and Q-Value: It is good to introduce $g_{\theta}$, but the Q-value is still a linear combination of the SF. So I think this is still somewhat narrow. 2. New SF approximation: In my opinion, the key advantage is that the distribution distributes the difference between $\psi(a_t,s_t,w)$ and $\psi(a_{t+1},s_{t+1},w)$ to $m$ different points. So each point only needs to change modestly when the time progress, and the similar features among different times can be better stored by those points of high density among times. Correct me if I am wrong here. This seems a good point, but the authors lack enough explanation here. 3. New Learning Objective: I don't understand why the authors omit $\mathcal{L}_{\psi}$ in training, the new objective seems to have no term to align the difference of $\psi$ and the cumulant $\phi$. The authors also do not provide enough explanation to why this objective is designed, nor any ablations to explain this.I have a hard time understanding this part. 4. New transfer scheme: This seems a novel point. The authors again use distributions to model the task encoding. This form may be of stronger approximation ability. # Clarity 1. I like the Summary of Challenges in Fig.8, it helps quickly figure out the clues of this work. 2. The authors give enough reference to key concepts, this helps me a lot in reviewing this paper. # Significance Good in the showed case, but narrow in the scenario.

Weaknesses

# Clarity: One of my major concerns is the clarity of this work. It seems like it was done in a hurry. The notations are used carelessly, usually not unified. I can see the same symbols with different superscripts or subscripts, or arguments. The components of the proposed method lack enough explanation of why they are helpful. Those make me restless, can't sleep overnights and recall them in dreams. I suspect several places use wrong notations, correct me if I am wrong. Some issues of clarity are as follows: 1. Line 112 you miss an "of". 2. It is $r_t^{\tau}$ or $r_t$ in the second eq. of Eq (4)? 3. I recommend to write Eq. (7) as $\mathrm{Twohot}(y_t^{\psi_k})\log p_{\psi_k}^T$ for clarity. 4. Not every fig of this paper has an error bar. 5. Notations are a bit confusing and frequently unexplained. I have a hard time understanding them. 6. What does $\theta^o$ mean in Eq. (6\&7)? 7. Righthand side of Eq. (7) seems not a scalar? 8. Line 213, what does the loss function mean? Why do you design it? Also, is the state function also trained by it? # Significance 1. The improvements are not significant. 2. Environment is a bit narrow. Why only in 3D cases? 3. The authors only study one dataset, making the result not strong enough. 4. Limited ablation studies. No ablation about the loss, the new task encoding scheme, or the new SF approximation. The showed two ablations, in my opinion, are about minor issues.

Questions

I have some puzzles about the paper, plz help me clarify them. 1. Eq. (5), what is the SF dimension? Is $B$ a random variable and what does it stand for? How $b_k$ take values? 2. I don't understand why you can write SF $\psi_{\theta}^k$ as the Expectation of some distribution. Do you mean that $b_i$ should be $\gamma^i$ and $\mathrm{Exp}(l_{\theta}(s_t,w,e_k))$ should be $\phi_{t+1+I}$? 3. Is there any difference between saying $\psi_{\theta}^k=B^T l_{\theta}^{\psi}$ is an Linear combination of all $\mathrm{Exp}(l_{\theta}(s_t,w,e_k))$ and saying it is an expectation of $p_{\psi^k}? Why do you prefer the latter? 4. What does Line 144 mean? What is the difference between $\tilde{\psi}^{\pi_k},\psi^{\pi_k},\tilde{\Psi}^{w}$? I look into the "Universal Successor Feature Approximator" paper. It seems the tilde stands for approximation. What confuses me is why it only approximates the neural network $\psi_{\theta}$? Shouldn't they be the same thing? 5. Do you lose a $t+1$ in $\phi_{\theta}(s_t,a_t)$ in Eq. (4)?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Limited scenarios. The computation burden seems to be increased compared with previous methods. The networks can be harder to train due to complicated designs. Computing approximations from sampling of distributions can also heavy the burden of computation both in training and inference.

Reviewer z7ra

Dear Reviewer z7ra, Could you please read the author's rebuttal and other reviews, and indicate whether it has changed your opinion? You can also engage further with the authors. Thank you. Best, AC