Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning

Summary

This paper introduces KIAN, a method for leveraging multiple sub-optimal policies to solve reinforcement learning tasks. The paper starts out by introducing a new Knowledge-Grounded MDP, which adds a set of external knowledge policies to the traditional MDP framework. To leverage these external knowledge policies, KIAN learns an embedding for each knowledge policy (including an inner knowledge policy for the current agent). Then at each step, the current state is mapped to a query embedding that can be used to find the policy that is best equipped to take an action at that timestep.

Strengths

* The paper is structured well and is easy to read. * The authors demonstrated great attention to detail by motivating the theorems and equations with intuition before defining them concretely. * The paper is very technically sound.

Weaknesses

* > In such a case, whenever the knowledge set is updated by adding or replacing policies, prior methods require relearning the entire multi-policy fusion process, even if the current task is similar to the previous one. This is because their designs of knowledge representations are intertwined with the knowledge-fusing mechanism, which restricts the number of policies in the knowledge set from being changed. * It would be good to point to specific prior works that suffer from this problem so the reader can build intuition. * There are no visualizations of the tasks used for evaluation. Adding pictures of the environments would help readers understand what the agent needs to do.

Questions

* I don't fully understand the incremental property. In the introduction. the authors state that "Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones." But then Definition 3.5 states that an incremental agent has two properties: 1. The agent can use different knowledge policies to solve a single KGMDP 2. Given a sequence of KGMDPs, the agent can solve them with different knowledge sets. * In its current form, definition 3.5 does not specify the case where an agent adds or removes a specific policy from its knowledge set. I think a condition should be added to highlight how an incremental agent should be able to leverage a sequence of knowledge sets that differ by one or more policies. * How do the initial external knowledge policies perform on the given tasks?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors discussed broader impact and future work in Appendix D, but no limitations were addressed. Could the authors please add limitations to either Appendix D or the conclusion?

Summary

The authors of this work introduce Knowledge-Grounded RL (KGRL), an RL framework that combines multiple knowledge policies to achieve human-like efficiency and flexibility in learning. They propose a new actor architecture called Knowledge-Inclusive Attention Network (KIAN) that enables arbitrary combinations and replacements of external policies through embedding-based attentive action prediction. KIAN also addresses entropy imbalance, a challenge in KGRL, addressing exploration in the environment.

Strengths

Firstly, the paper addresses a relevant and interesting problem in the field of reinforcement learning (RL) by improving sample efficiency from arbitrary external policies and enabling knowledge transfer skills. Furthermore, the paper is a well-written work that effectively conveys the ideas and findings in a clear and concise manner. The authors demonstrate excellent writing skills, employing appropriate terminology and organizing the content in a manner that enhances readability and comprehensibility. In addition, the paper effectively establishes the motivation and position of the study in the existing literature. The authors articulate the significance and relevance of the research problem, demonstrating a strong understanding of the field. They situate their work within the broader scholarly context, highlighting how their study fills a gap in knowledge or builds upon prior research. The methodology employed in the research is clearly described, allowing readers to understand and replicate the study. The authors provide a detailed and comprehensive explanation of the experimental and theoretical approach used, supported by well-designed figures and diagrams. These visual aids enhance the understanding of the methods employed and facilitate better comprehension of the research. Lastly, the authors provide solid mathematical understanding for their proposed method. Collectively, these strengths highlight the quality of the paper. ts focus on addressing a relevant and interesting problem, combined with its well-written content, clear methodology, positioning in the literature, and mathematical rigor, make it an intersting contribution.

Weaknesses

The paper has one notable weakness that should be addressed to enhance the overall quality. The evaluation presented in the paper is quite limited. The results obtained for the OpenAI robotics task only offer a slight support for the proposed method. The authors should consider expanding the evaluation to provide a more comprehensive assessment of the proposed method's performance. Additionally, the use of only 3 or 5 seeds in the experiments may be insufficient, especially when considering the large error bars observed in some models. The authors should consider increasing the number of seeds to improve the statistical robustness of the results. Moreover, the paper lacks clarity regarding the error bars shown in the plots. It is not explicitly stated what these error bars represent, which hinders the interpretation and understanding of the presented data. Lastly, the paper would greatly benefit from conducting ablation studies to investigate the effects of various factors on the proposed method's performance. Specifically, the authors could consider performing ablation studies on - the influence and distribution of attention/weights of actors, - the impact of random/irrelevant policies in $\mathcal{G}$, - the impact of the (near) optimal policy in $\mathcal{G}$, - the impact of a larger set of knowledge policies $\mathcal{G}$, - the effects of different types of knowledge policies, - and an investigation of the mentioned entropy balance issue, that the authors specifically address in their method. These ablation studies would provide valuable insights into the individual contributions and impacts of these factors on the overall approach. Overall, addressing these weaknesses would significantly improve the research paper, clarifying important aspects such as error bars, conducting relevant ablation studies, ensuring consistency in reporting variances, and strengthening the empirical evidence for the proposed method. ### Minor - I assume ori-KIAN is KIAN with the original $\mathcal{G}$, this should be mentioned in the text. - l.314 mentions less variance, but tables do not show variances for individual experiments

Questions

See the above mentioned ablation studies.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The limitations have not been discussed by the authors. The above mentioned ablation studies will probably help to identify the shortcomings of the proposed method.

Summary

Humans can learn by aggregating external knowledge from others’ policies of attempting a task. While prior studies in RL have incorporated external knowledge policies for sample efficiency, there still remains the generalization problem to be solved that agents have difficulties to perform arbitrary combinations and replacements of external policies. The authors propose a new actor architecture for Knowledge-Grounded RL (KGRL), Knowledge-Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding-based attentive action prediction. KIAN addresses entropy imbalance as well. The authors demonstrate in experiments that KIAN outperforms other methods incorporating external knowledge policies under different environmental setups.

Strengths

The authors clearly define the problem as how RL can be grounded on any given set of external knowledge policies to achieve knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental properties. The proposed method of KIAN is clearly described. They use knowledge keys and the query performs an attention operation to determine how an agent integrates all policies. The solution of the entropy imbalance issues when integrating external policies are proposed as well.

Weaknesses

Generally this work has many related works but is tackling the unique challenge problem of fusing knowledge policies with different state and action spaces. Limitations of the proposed method is not clear based on the experimental results.

Questions

More experiments directly connected with this challenging problem would more clearly support your claims.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Some insights and analysis about limitations based on the experimental environments more directly connected with the target problem are expected to be included.

Summary

This paper defines the Knowledge-Grounded RL setting, a general RL setting for integrating knowledge (in the form of policies) into a policy to learn new tasks efficiently. Essentially this setting is similar to the MDP setting except that the agent is also given a set of knowledge policies to utilize. The paper also introduces a system/architecture within this KGRL setting, called Knowledge-Inclusive Attention Network (KIAN). The aim is to improve RL that is grounded on external knowledge policies. The paper outlines five desirable human-like properties they desire in their agents: knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental. Moreover, they formally define these so that they are measurable within the KGRL setting (e.g., for evaluating algorithms on these dimensions). While previous methods typically intertwine knowledge representation and knowledge-fusion, thereby restricting their ability to adapt to numbers of policies, losing flexibility. KIAN is developed more flexibility, separating the knowledge representation and knowledge fusion. KIAN consists of three components: a policy that learns a strategy (similar to a normal policy) called the internal, embeddings that represent the given knowledge (or external) policies, a query that performs attentive action prediction to fuse the internal and external policies. KIAN also solves other issues that can occur in entropy-regularized KGRL. Entropy-regularized RL is common, but the authors show that in the KGRL setting issues can arise through entropy regularization where only a select few policies are selected, counterproductively reducing diversity in policy usage. The authors show that in the KGRL setting the agent will pay more attention to the policy with large entropy and in continuous control, will rely on the internal policy extensively. The paper introduces modifications so that this does not occur in KIAN. The authors show results on both MiniGrid and robotics tasks and demonstrate sample efficiency, generalizability, as well as compositional and incremental learning.

Strengths

The paper is mostly well-written and well-explained. The method makes sense, and is a well-thought out architecture. I like how the authors address the entropy imbalance problem. I like that the authors define and quantify the behaviors they would like in the agent. The results do seem to demonstrate their method is effective.

Weaknesses

While I think the experiments are good, with appropriate baselines and good environments to test the agent’s capabilities. I am concerned about statistical significance. In particular, only 5 seeds are run, and the performance benefit in many cases is minimal, which may quite possibly be attributed to randomness. While I do believe the method outperforms the baselines, I cannot say so with a lot of confidence to merit inclusion at NeurIPS. If the authors can run more seeds, especially given the large variance, it would dramatically improve their results. Qualms: KGRL is consistently referred to as an RL framework, which it is, but the connotation can be misconstrued as being a framework “for” RL, implying it is a solution method for RL problems. I would recommend calling it a “setting” rather than a framework. Indeed, I was confused temporarily as a reader, especially when KGRL is stated as being “introduced” by this paper (as opposed to “described” or “outlined”). Nits Typo Line 345: “border” should be: “broader”

Questions

In line 33, it is stated (for incremental): “Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones”. Do humans truly “remove” outdated strategies? In line 222, the paper states: “However, fusing multiple policies as equation (3) will make an agent biased toward a small set of knowledge policies when exploring the environment.” I am somewhat confused, equation (3) as is does not seem to have this issue. I thought this only occurs in the MaxEnt KGRL setting as introduced in the next section. Can the authors please clarify this? The authors state the entropy imbalance problem is a property of the maximum entropy KGRL setting. I want to clarify that this is only shown for the specific case of policies in the form of equation (3), correct? How could the policy embeddings/knowledge keys be learned? Is Gymnasium Robotics used for the experiments? Or the older OpenAI codebase?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

To me it is unclear how in more sophisticated settings how the knowledge keys (or policy embeddings) would be learned. This seems like a bottleneck to scalability that is not well-addressed.

Followup Question for Authors

What did the error bars in the original pdf represent?

Followup Response to the Authors

Thank you again for your response, and for addressing all my points. After reviewing the updated curves, I still maintain my score. For some environments, the results are statistically significantly better than the baselines. For most other environments, they are are not, though they do not appear to be worse. As such, I still believe the paper should be accepted. The evidence in support of the proposed method is convincing, but not overwhelming, hence why I am not giving a stronger accept.

Reviewer Responses Needed

Hello Reviewers, The authors have made efforts to address your comments on their work via the rebuttal. Part of the NeurIPS review process is participating meaningfully in the rebuttal phase to help ensure quality. Please read and respond to the author's comments today, latest tomorrow, to give everyone time to respond and reach proper conclusions. Thank you for your assistance in making NeurIPS a great conference for our community.

Response to Author Rebuttal

Thank you for addressing my concerns! I can't see Figures 6 and 10 in the supplementary material nor the latest paper revision.