Efficient Multi-Task Reinforcement Learning with Cross-Task Policy Guidance

Summary

The paper presents Cross-Task Policy Guidance (CTPG), a novel framework designed to improve multi-task reinforcement learning (MTRL) by leveraging cross-task policy similarities. CTPG trains a guide policy for each task to select the most suitable behavior policy from a pool of all tasks' control policies. This method aims to generate better training trajectories and enhance learning efficiency. The authors propose two gating mechanisms: a policy-filter gate to filter out non-beneficial control policies and a guide-block gate to block unnecessary guidance for mastered tasks. Empirical evaluations on manipulation and locomotion benchmarks demonstrate that integrating CTPG with existing parameter sharing approaches significantly enhances performance.

Strengths

### S1. Innovative Approach The paper introduces a novel method for leveraging cross-task similarities in MTRL. By training a guide policy to select behavior policies from a pool of tasks, the approach provides a direct and efficient way to exploit shared skills, which is an underexplored area in MTRL. ### S2. Empirical Validation The proposed framework is validated through extensive experiments on various benchmarks, including MetaWorld and HalfCheetah. The results demonstrate significant improvements in learning efficiency and performance when CTPG is integrated with existing MTRL approaches. ### S3. Clear Presentation The paper is well-structured and clearly explains the CTPG framework, including detailed descriptions of the guide policy and gating mechanisms. Figures and tables effectively illustrate the benefits of the proposed method, and the provided pseudocode enhances reproducibility.

Weaknesses

### W1. Scalability Concerns The scalability of CTPG to more complex and large-scale environments is not thoroughly discussed. While the method shows promising results in the tested benchmarks, a broader analysis of its scalability and practical utility in more complex scenarios may improve the significance of this work. ### W2. Computational Complexity The computational complexity of training and deploying CTPG, particularly the guide policy and the gating mechanisms, is not explicitly addressed. Understanding the computational requirements and potential limitations in terms of resources and execution time would provide a more comprehensive evaluation of its applicability.

Questions

### Q1. Scalability to Complex Environments How does the CTPG framework scale to more complex, large-scale environments? Are there any specific challenges or limitations that need to be addressed for practical deployment in such scenarios? ### Q2. Computational Requirements What are the computational requirements for training and deploying CTPG? How does the method perform in terms of execution time and resource consumption compared to existing MTRL methods?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors acknowledge the limitations related to the predetermined guide step K and the reliance on specific benchmarks. However, a more detailed discussion on potential negative societal impacts and strategies to mitigate them would be beneficial.

Summary

This paper tackles the problem of multi-task reinforcement learning. Previous works have primarily focused on MTRL through specialized network structures or methods to resolve conflicting gradients. This paper considers the problem from an orthogonal perspective, by actively sharing control policies for each task adaptively, in the hope that sometimes the policy of one task can guide the exploration of other tasks. Specifically, similar to in hierarchical RL, a high-level policy is used to decide which “task policy” to call at each particular state, where “bad” policies are masked out through a learned value function. Evaluated on Metaworld and HalfCheetah MTRL benchmark, the method shows improved performance compared to previous works.

Strengths

- The idea of exploring with cross-task policies to facilitate multi-task reinforcement learning is interesting and novel. - The method is thoroughly evaluated on a set of test domains, and show superior performance compared to previous methods - This paper is well written, and the method is easy to follow.

Weaknesses

- The proposed method contains many floating pieces that seem to add burden both to implementation and hyperparameter tuning. Some of the pieces seem rather like hack than a principled solution. For example: - For the hindsight off-policy correction, the action of the guide policy is relabelled to the one that is most likely to generate the sequence of actions. However, it should be likely that none of the control policies can generate the old sequence of actions with high probability, in which case relabeling will not help at all. - The action space masking is essentially masking out control policies with low Q values. Shouldn’t this already be reflected in the policy through policy gradient updates? Why do we need additional masking? - Using the temperature coefficient of SAC as an indication of the policy performance is very empirical and may not always be true. This also makes the proposed method specific to adaptive temperature SAC. - It seems that the hindsight off-policy correction would require much more forward passes than a regular SAC. Can the author show the performance of the proposed method with x axis being wall-clock time? - For the proposed method to be effective, we need the policy of some other tasks to have better performance than the policy that is trained for the current task. One scenario I can think of is when the multiple tasks secretly form a curriculum (e.g. one task is the prerequisite for another task), which might be the case in Metaworld and halfcheetah. But I’m not sure how realistic such an assumption is in the real world. Minors: - Line 207: dose → does - “someone who can ride a bicycle can quickly learn to ride a motorcycle by referring to related skills” → I don’t think that’s actually the case…

Questions

See weaknesses

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations are adequately addressed.

Summary

This paper addresses the problem of multi-task reinforcement learning (MTRL). To this end, the paper proposes a method that selectively shares behaviors from the policies learning to solve other tasks. The experiments conducted in a locomotion domain (multi-task Half-Cheetah) and a robot arm manipulation domain (Meta-World) verify that the proposed method can improve the performance of the learned policies. Ablation studies justify the effectiveness of many proposed components, including the policy-filter gate, the guide-block gate, and the hindsight correct. Yet, I am not convinced that the proposed method could improve the sample efficiency, which is the key goal of MTRL. Also, it seems that the proposed method is incremental and lacks novelty. Therefore, I am leaning toward rejecting this work in its current form.

Strengths

**Motivation and intuition** - The motivation for sharing behaviors for MTRL is convincing. **Experimental results** - The main experimental results show that the proposed method achieves better converged performance compared to the baselines in MT Half-Cheetah and Meta-World. **Ablation study** - The ablation studies justify the effectiveness of the policy filter gate, guide block gate, and correct hindsight. **Reproducibility** - The code is provided, which helps understand the details of the proposed method.

Weaknesses

**Main results (Table 1)** - Table 1 and Section 5.2 present quantitative results of converged performance. However, as far as I am concerned, the comparisons of MTRL should focus on sample efficiency, i.e., how fast each method can learn, instead of converged performance. Therefore, I am not convinced by this evaluation. **Comparison to k-step QMP** - The paper states that the proposed method outperforms QMP because "guide policy learns long-term policy guidance." I am not entirely convinced. One can simply use QMP to select from k-step behavior proposals and execute the selected proposal, which can also achieve this long-term policy guidance. Including this variant of QMP would be necessary to show that the performance gain comes from other designs of the proposed method. **Guide step K** - The best guide step K in the MT Half-Cheetah and Meta-World are different. It is unclear, given a new MTRL domain, how we should choose K. It seems that we could only tune this hyperparameter via trial and error. **Backbone RL algorithms** - This work adopts SAC as the backbone RL algorithm. Is it possible to use other RL algorithms, such as TD3 or PPO? **Clarity** - Section 4 is difficult to follow. Sufficiently describing the intuitions before introducing each component of the proposed method would significantly improve the readability of this section. **Novelty** - The proposed method seems incremental given that the QMP paper explores this idea of behavior sharing. Despite the high similarity between this work and the QMP paper, the authors seem to deliberately hide this by avoiding discussing QMP in the introduction.

Questions

See above

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes

Summary

This paper proposes a method called CTPG to enable policies trained on different tasks to learn from each others' generated trajectories. CTPG operates by learning a guideline policy that determines which control policy in a given set should best generate trajectories to enable agents to learn a particular task. Then, the authors also designed additional mechanisms that (i) deal with how each policy whose trajectories are sampled may change from their learning process, (ii) prevent negative transfer from using trajectories from irrelevant policies for learning, and (iii) promote the convergence towards good policies by preventing any transfer of experience once a policy is already proficient at its intended task. Using environments from multi-task reinforcement learning benchmarks, the authors then investigated their method's performance when (i) learning a set of tasks in parallel and learning a new task that is not from the set of previously learned tasks. Furthermore, the authors also conducted further analysis to investigate (i) which component within their learning algorithm is most responsible for the method's performance and (ii) whether the guideline policy learned a sensible experience selection policy.

Strengths

**Major Strength - Clarity** Except for a few minor clarification details provided below, I find the paper to be well-written. I especially appreciate the clear outlining of (i) the motivation behind the proposed method, (ii) the questions being investigated by the authors, and (iii) the experiment design. I hope that future iterations of the manuscript will keep this same level of clarity. **Major Strength - Method Soundness** From a high-level perspective, it seems that the learned transfer learning mechanism proposed in this paper seems reasonable. I especially find the authors' design of various mechanisms to address different problems (i.e., policy selection, negative transfer, variable rates of learning between tasks) to be great choices that should improve learning performance in multi-task reinforcement learning scenarios. **Major Strength - Experiments and Analysis** I also think that the authors did a great job of thoroughly investigating the efficacy of their method. The five questions in the experiment section provided interesting insights into the method. At the same time, the experiments done to answer each of those questions were well-designed and showed the method's positive performance. **Minor Strength - Significance** While allowing agents to learn from each others' experience to improve their performance at achieving their respective objectives is not exactly new, to my knowledge, the authors' proposed method for learning in multi-task reinforcement learning (and addressing its various associated issues) is novel. Even if I am wrong on the novelty aspect of this method, I still believe the thorough analysis provided by the authors would contribute to different insights that may be useful for the multi-task RL community.

Weaknesses

**Minor Weaknesses - Additional Comparisons** Perhaps another baseline that could be compared is the simple strategy from [1], where one simply learns from another policy with the highest performance at the task. Using this baseline should elucidate the effects of switching between different policies to transfer from as opposed to just transferring from a policy that seems to perform the best. **Minor Weaknesses - Hindsight Off-Policy Correction** From Section 4.1 alone, I also do not find how the hindsight off-policy correction mechanism affects the remainder of the learning process. While I did check Algorithm 2 in the appendix, I am uncertain why $j^{'}_{t}$ (the behavioral policy chosen in hindsight) only affects SAC's critic updates and not also the policy updates. References: [1] Disentangling Transfer in Continual Reinforcement Learning. Wolczyk et al. NeurIPS 2022.

Questions

1. How would CTPG perform against a simple transfer learning strategy where experience is only transferred from the policy that has the best performance? 2. Does the behavioral policy chosen in hindsight affect the actor-network training? If it does, how would it do so?

Rating

7

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have sufficiently outlined their method's limitations and interesting directions for future work in the last paragraph of the manuscript.

Re: Rebuttal by Authors

Thank you for the rebuttal, including the additional TD3 results and clarifications. I am increasing my score to 5 to reflect what's addressed by the author's rebuttal.

Response to authors rebuttal

I appreciate the authors for the detailed response. Most of my concerns are addressed therefore I'm raising my assessment.

Rebuttal Summary and Appreciation

Dear Reviewers, ACs, SACs, and PCs: As the discussion phase comes to a close, we would like to sincerely express our gratitude for your time and effort throughout the review and rebuttal process. We deeply appreciate the constructive and insightful comments provided by the reviewers after carefully reading our manuscript, which have significantly improved the quality of our work. Here, we would like to summarize the key points of our rebuttal and discussions: 1. Based on (Reviewer xTD7)'s feedback, we conducted ***additional experiments comparing CTPG with BPT***. These experiments more clearly demonstrate the advantages of CTPG in learning a mixture policy by combining different control policies. 2. In response to (Reviewer 5DoG)’s concerns, we performed ***supplementary experiments using another base RL algorithm TD3***, which further demonstrates that CTPG is a generalized MTRL framework. 3. To address (Reviewers J1dv and k7nL)’s concerns, we provided a complementary ***analysis of the computational complexity*** of CTPG. 4. We are grateful for all the reviewers’ comments on the revisions, which have made our paper more complete, including ***a detailed explanation of the hindsight mechanism's impact on SAC*** (xTD7), ***additions to the experimental hyperparameters setup*** (5DoG), ***a more relevant realistic example with CTPG*** (J1dv), and corrections of some typos. ***During the discussion, all the reviewers provide appreciations and claims for his/her concerns addressed,*** and we were ***especially encouraged*** by (Reviewer xTD7)’s conclusion: ***"Given the authors' responses to the points raised by each reviewer, I am increasing my score because I believe this is a good paper worthy of acceptance."***. We appreciate that we receive ***all positive*** feedbacks from every reviewer. We believe that our work would offer valuable insights to the MTRL community and make a positive contribution to the `NeurIPS 2024` conference. We would like to thank you once again for your time and dedication throughout the review process. Best regards, Authors