Integrating Suboptimal Human Knowledge with Hierarchical Reinforcement Learning for Large-Scale Multiagent Systems

Summary

The authors proposed a new framework (hhk-MARL) that integrates human abstract knowledge with hierarchical reinforcement learning to address the learning challenges in large-scale multi-agent systems. The framework employs fuzzy logic to represent human knowledge and a graph-based group controller to enhance agent coordination. Experimental results in the StarCraft Multi-agent Challenge demonstrate that the proposed approach significantly accelerates the training process and improves the final performance of the agents, even with imperfect human prior knowledge.

Strengths

* Overall, the proposed hhk-MARL framework effectively integrates human knowledge with hierarchical reinforcement learning, providing a flexible and adaptive approach for multi-agent systems. * Methodologically, the authors use fuzzy logic and hypernetworks to dynamically generate weights based on agent observations allowing for the seamless combination of human and agent preferences. * In the experimental results, the comprehensive experiments demonstrate the framework's efficacy and robustness, showing significant improvements over baseline methods in various scenarios.

Weaknesses

While the paper is well-written overall, it lacks detailed comparisons with prior research and thorough explanations of the human knowledge used. Specifically, section 3.1 does not clearly justify the choice of fuzzy logic over other approaches, which could also handle uncertainty and abstract knowledge effectively. Additionally, the paper can explain how its approach differs from previous works such as KoGuN and Shi 2021, to clarify its unique contributions. In section 3.2, the new aspects of using hypernetworks for knowledge integration should be more distinctly highlighted in comparison to similar methodologies. For the details, see the following Questions.

Questions

1. The paper employs fuzzy logic in the hhk-MARL framework for integrating human knowledge, but it does not provide a clear explanation in section 3.1 for choosing fuzzy logic over other methods such as Bayesian networks, a combination of reinforcement learning and heuristics, or neural network-based knowledge representation. These alternative methods could also handle uncertainty and abstract knowledge effectively. A more detailed justification for selecting fuzzy logic would enhance the understanding of its advantages in this context. 2. The idea of incorporating fuzzy logic into hierarchical RL has been introduced in prior works such as KoGuN [20] and in Shi 2021 [a]. However, it is essential to explain the differences between these approaches and the proposed method in section 3.1. Highlighting these distinctions will clarify the unique contributions of this work and demonstrate how it advances beyond existing methodologies. 3. In section 3.2 on Knowledge Integration, it is important to highlight the novel aspects of the proposed method. The integration of human knowledge using hypernetworks to dynamically generate weights based on agent observations presents a unique approach. However, the methodology bears similarities to existing works such as HAMXCS [21] which also incorporates heuristic knowledge into reinforcement learning, Hierarchical RL for Self-Driving [19] which uses hierarchical reinforcement learning for decision-making, and KoGuN, which refines fuzzy logic rules with hypernetworks. A clearer distinction of the innovative elements in comparison to these related works would strengthen the contribution of this paper. 4. Appendix A.6 presents the suboptimal human knowledge rules, but it is unclear who determined these rules and to what extent they are incomplete or reasonable. Given that they are labeled as suboptimal, a clearer explanation of their limitations and adequacy is necessary, especially for readers unfamiliar with Starcraft. This information is crucial for understanding the potential applicability and adaptation of these rules to other tasks.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper integrates an human in the loop to provide knowledge that improves learning in marl. This is done through a hierarchical structure, but ultimately it is up to the agents the final decision of accepting the human suggestions (hierarchy comes from human knowledge to agents). Overall, there is an integration of human knowledge with what the agents learn.

Strengths

The paper is well organized and it is interesting. The idea of integrating human knowledge is this kind of tasks is interesting and it makes sense. Mostly, the paper is easy to follow and understand, with some exception that I outline below.

Weaknesses

While I find this paper interesting, I have some remarks, as noted below: * in line 131 it is stated that $M_L^I$ corresponds to a fuzzy set; however, it is not clear what $M$ is exactly, and how it relates to fuzzy logic in this context; it is not easy to understand how the relation between $O$ and $M$ is calculated and how is the membership function $\mu$ implemented * from my understanding, when integrating the proposed approach with IQL, there is not a mixer as in the other value function factorization methods like QMIX; this leads me to think that the lack of a mixer could be the reason behind the huge improvements seen in Figure 4 for IQL when combined with this method; can it mean that the mixer can be slightly detrimental in the overall framework?

Questions

In addition to the points above, I have some specific questions that I would like the authors to comment on: * what are the limitations of considering only a small set of human opinions, like the 8 ones defined in appendix 6? * if a certain human knowledge concerns only one specific agent, is it still given to the others as well? if yes, how does it affect learning? do the others get confused by that knowledge? * how do the agents decide if they accept a certain human suggestion? is it a joint decision? if yes, can they reason accurately if the decision concerns a specific agent, but has nothing to do with the others? * is the knowledge controller deterministic? or was it trained a priori with human knowledge? * it is stated in lines 179-180 that the group controller is used only during training to not violate CTDE; what about the knowledge integration, is it also used only during training? or also during execution? since it receives human knowledge from all agents too * the knowledge integration module seems to follow an interesting approach, since the weights of the second network are generated by an initial network based on the observations of the agents given as inputs; could the authors elaborate on the motivations for this approach? why are the weights generated from a first network and not only using one network? * i can see the authors focused in environments involving marines only; is there a specific reason for that? is it because of the availability of prior human knowledge? or how hard it would be to create a knowledge controller network for other cases?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

In this paper, the authors propose a novel method to tackle the multi-agent reinforcement learning problem. They do so by combining human abstract knowledge with hierarchical reinforcement learning. Specifically, human knowledge in the form of fuzzy logic rules is combined, at the top level, with each individual agent’s decisions, learned at the bottom level. Then, a graph-based group controller performs agent coordination to decide what the action at each step should be. The authors evaluate the proposed method on the StarCraft multi-agent Challenge, combined with three algorithms (IQL, QMIX, and Qatten). The results indicate that the proposed approach is capable of improving the overall performance.

Strengths

The authors tackle the multi-agent reinforcement learning problem in a creative and novel way, combining human feedback with hierarchical techniques. Moreover, it is a very interesting idea to provide a general algorithm that can be coupled with any existing MARL technique. In this way, it is possible to take advantage of the benefits of previously proposed algorithms while also incorporating new ideas to improve overall performance. The paper is well-written, and all the high-level ideas behind the different components are well-explained.

Weaknesses

The empirical evaluation is thorough, and it indicates that the claims are correct. However, only three samples are rarely enough to draw strong conclusions. I believe some crucial parts of the algorithm were not very detailed. In particular, what are the learning rules/loss functions for each of the components? That is, how is $\beta$ trained? How is the knowledge integration component trained? With the current description, I believe it would be very hard for someone to replicate the method.

Questions

For the ablation studies of the impact of each component on the final performance, it would also be interesting to see how much of the agent’s Q is used vs how much of the human knowledge Q is used for the final Q prediction. That is, what is the magnitude of the impact that the human knowledge component has over the Q predictions?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss the limitations of the work. No major concerns.