Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning

Summary

This paper presents a new approach to automatic curriculum learning designed specifically for multi-agent coordination problems.

Strengths

- The main strength of the paper, in my opinion, is the well-formulated approach to the curriculum learning problem. To the best of my knowledge of the related literature, the non-stationary contextual bandit as the teacher and the population-invariant skills for the students are both original and useful contributions to the literature.

Weaknesses

- My general problem with this paper is that I am finding it hard to evaluate the significance of the work in the automatic curriculum learning sphere without an adequate baseline provided for GRF. Whilst the authors do argue that VACL is not used on GRF due to requiring prior knowledge, it seems unreasonable to therefore provide no baselines that are actually designed for these larger settings. For example, if VACL was unusable, then I would have maybe liked to have seen a comparison to population-based approaches in MARL or any of the other automatic curriculum learning approaches mentioned in Sec. 4. Overall, it is hard to properly evaluate the gains from this automatic curriculum learning framework without seeing the performance of baselines in an environment that actually requires automatic curriculum learning (MPE does not need it according to line 297-298). I am happy to update my score if the authors can make a reasonable argument against the lack of other baselines in the work.

Questions

- Line 37-39, 'However,..., tasks matters' - I am very confused by this sentence, could the authors please clarify? - Upon inspection of the results, the final improved performance does not seem to be massively impacted by the hierarchical RL element of the framework. I was wondering if the authors could discuss a little more on this, in terms of its necessity in the framework and when they believe it would provide greater gains in performance?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors briefly make mention to the limitations of the work. I agree with the over-design of the framework for simple tasks, so would definitely like to see its performance in more difficult environments that it is designed for.

Summary

This paper presents Skilled Population Curriculum (SPC) which is a method for learning a curriculum to help a team of agents complete a complex task. SPC models the problem of choosing tasks for agents as a contextual bandit problem, and builds on top of the Exp3 algorithm to solve this bandit problem. SPC also uses an attention-based communication approach, and a hierarchical policy framework. Experiments are performed in the Multi-agent Particle Environment (MPE) and Google Research Football environment (GRF). While MPE does not seem to benefit much from SPC, in the more complex GRF domain the authors show using their SPC approach can accelerate training relative to MARL baselines.

Strengths

- The paper is clear: it is well-structured, and provides a good balance of intuition and detail. It is clearly motivated, and addresses an interesting problem. - The presented results show clear benefits to the authors' approach. - The authors use suitable baselines approaches, and suitable environments.

Weaknesses

- SPC seems to add significant computational complexity vs. baselines like IPPO. While the authors can justify focusing on sample complexity, for completeness they should also record information about the wall-clock time / computational resources needed to train their different baselines. - In their research question stated on lines 52–53, the authors highlight their desire to consider complex sparse-reward settings. However, the MPE domains are a sparse setting, though fairly simple, and the results here show little benefit to using SPC. On the other hand, the GRF experiments appear to have a somewhat dense reward (the GRF checkpoint reward, which while not necessarily active every timestep, it could be argued is 'somewhat dense'). It seems like absent the checkpoint reward, SPC would struggle because there would be little information in the returns for the teacher agent to use — and I expect this would be the case in most complex (very) sparse reward environments - Line 277 the authors state MADDPG/MAPPO would not be suitable in these experiments. This might be true in general, but for GRF specifically the critic input size actually would be the same across all tasks (as it pads observations if agents are absent). But since GRF is fully observable, MAPPO is equivalent to IPPO so this is not an issue for this work — though the authors may wish to revise their statement. - Though the GRF environment is complex and difficult to solve, its level of complexity is somewhat deceptive, as evidenced by the video on the project website. The rollouts show that the agents have learned a simple "force an offside and run in a straight at the goal line" strategy which exploits deficiencies in the GRF bots. This behaviour has been observed before by Song et. al (http://arxiv.org/abs/2305.09458). However, this is not a fault of the authors, and is more broadly an issue in the MARL research community. Because of this, it's not clear what skills the agents learn in the training tasks that are useful in the target task. It would be interesting to see a plot of training task performance throughout training. - It's unclear why the IPPO baselines have a sharp step change in performance around 80 and 90 million timesteps. The authors should investigate this, and perhaps make a comment (at least in the appendix) about why it occurs. In my experience, things like this sometimes occur when training runs stop unexpectedly before the full 100M timesteps, and so the remaining timesteps are aggregating over fewer seeds with lower performance. I would encourage the authors to produce plots reporting the interquartile mean of their results, and produce a plot showing the disaggregated training curves for each seed. These can go in the appendix. - The authors state (line 301): " InFig. 5b, we omit the curve of QMix as its mean score is low and affects the presentation of the figure". I don't expect QMix to perform worse than the presumably near-uniform policies at the start of training for the other agents. So it's not clear how including QMix would disrupt the graph. Is it the case that QMix has a worse average goal difference than -2? - Can the authors clarify: the target distribution for GRF is "100% 5vs5"? What is the target distribution for the MPE tasks? (I see now that these are mentioned later in the text: they should be mentioned when introducing the environments) - It doesn't seem like there's a pattern to the task distribution (Fig. 6a) beyond "academy_pass_and_shoot_with_keeper becomes less common". It would be good to see the same plot for other trials. This possibly explainable by academy_pass_and_shoot_with_keeper requiring coordinated passing and shooting, whereas the 5vs5 rollouts (see video on project website) show a very simple GRF-bot exploiting strategy which does not closely resemble the behaviour required in academy_pass_and_shoot_with_keeper. - Where the authors claim "For example, the proportions of 3vs1 and Empty-Goal tasks gradually drop as the student becomes proficient in these scenarios", it is difficult to support this by looking at Fig. 6a. - In my opinion this approach is over-engineered, but the authors do acknowledge this. Stripping some components (e.g the hierarchical RL) and focusing on deeply investigating the remaining components would improve this work - Minor writing fixes: - line 42/43 "more scores" → "more goals" - line 43: "4v11" → "4v1" (I assume) - line 305: "tons of" → "many" (more formal tone)

Questions

- Why did you decide to add the shooting reward for 5vs5? Does it make a big difference? - What causes the step-change drop in IPPO performance? - How does including QMix in 5b disrupt the graph?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

- The limitations section is quite limited, and limitations and assumptions could be more clearly stated throughout. - The authors recognise that their approach is complex and computationally intensive, so might not be applicable in simple environments. Testing in a complex environment like Google Research Football is a good choice, although due to issues with Google Research Football (such as the exploitability of the built-in AI) it is perhaps not as complex as the authors may hope, even though it has presented a challenge to past MARL research. However, this is a broader issue within the MARL community and the authors of this paper cannot fairly be singled out for this.

Summary

The paper introduces a new automatic curriculum learning framework, Skilled Population Curriculum (SPC), for multi-agent reinforcement learning. The algorithm includes three major components: (1) a contextual bandit conditioned by student-policies representation for automatic curriculum learning; (2) An attention-based communication architecture for policies to learn cooperation and behavior skills from distinct tasks with varying numbers of agents; (3) A hierarchical policy architecture to help agents to learn transferable skills between different tasks. The experiments are conducted in Google Research Football environment and Multi-agent Particle environments, which demonstrate the efficiency of the proposed method to IPPO and VACL.

Strengths

1. The proposed method is simple yet efficient in the complex Google Research Football environment. 2. The motivation of the components are also clear and make sence. 3. In exepriment, several ablation studies demonstrate the effectiveness of the proposed components; 4. Also, the paper is overall easy to follow to me. The key idea is easy to understand.

Weaknesses

This paper could benefit from further improvements in the following aspects: 1. It seems that the manuscript introduces various components. While each one appears to be intuitive and rational in isolation, I recommend that the authors should provide a unifying theme or framework to better connect these components. Presently, it appears as if these components are addressing three discrete issues: a) efficient curriculum learning, b) policy architecture development, and c) communication in varying agent scenarios. It is noteworthy that a paper does not necessarily need to devote substantial attention to the innovative aspects of each introduced components. In the case of this paper, the hierarchical structure, for instance, appears to be a standard approach with limited novelty. The authors can highlight how they design efficient automatic curriculum learning in the context of variable agent scenarios. 2. In the section discussing related work (Line 221), the authors mention various curriculum learning mechanisms without a detailed discussion. Could the authors provide an expanded explanation on how these works conduct curriculum learning and how they relate to or differ from the proposed methodology? 3. There is room for improvement in the experiments section. Specific recommendations are detailed in the questions section. 4. The paper could be further polished, for instance: - There are several instances where a capital letter follows a comma, such as in line 40: "For example, In the football environment, when we…" - The legend of Figure 6(b) lacks clarity. It would be beneficial if the authors could provide a detailed explanation of what the labels 0,1,2,3 represent.

Questions

1. Could the authors clarify why maximizing rewards should be the objective of a teacher in curriculum learning? Intuitively, it seems a teacher policy that aims to maximize performance given student policies would tend to recommend simpler tasks to learn, which is not what we would like to see. 2. Ignoring the previous question, could the authors explain why the Bandit model necessitates using the representation of students' policies as input of the teacher policy? It seems that providing an optimal course distribution based on the current student's behavior would be sufficient. Why is there a need to consider course distributions under the representations of other (mainly comes from the historical) student policies? Regarding the experimental section: 1. At Line 293, the authors mention that the SPC can switch to the largest population rapidly. Could the authors provide further explanation as to why this is possible and why it represents an advantage? 2. The second and third paragraphs of Section 5.3 are unclear; the authors could rephrase them for clarity. You could try presenting your point as follows: "From figure X (or the comparison of X and Y), we can observe XX, which indicates XX." 3. In Appendix C, a more challenging 11 vs. 11 experiment was introduced, with the authors claiming superior performance of the SPC, which is great to see. But this claim raises questions as there are no baselines for comparison. Could the authors consider adding some baselines to this task?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

NAN

Summary

This work introduces the Skilled Population Curriculum (SPC), an automated curriculum learning algorithm designed for Curriculum-enhanced Dec-POMDP. The goal of SPC is to enhance the student's performance on target tasks via a sequence of training tasks provided by the teacher. The SPC functions as a nested-HRL method, where the teacher serves as the upper-level policy and is modeled as a contextual multi-armed bandit. At each teacher timestep, the teacher selects a training task from the distribution of bandit actions, with the context derived from the student policy's hidden state. The teacher's bandit is optimized using the student policy's test reward. The lower-level policy, also known as the "student", is in itself a hierarchical policy. The high-level policy implements population-invariant communication using a self-attention communication channel to manage messages from a number of agents, and all students share the same low-level policy.

Strengths

- This paper is well-presented. Figure 1 is well-designed. I can get a good understanding of this paper's method just by reading this figure. - The algorithm is implemented with Ray RLlib, though the code is not currently available.

Weaknesses

- **This study seems to be an overcomplicated amalgamation of pre-existing methods.** SPC stacks three layers of hierarchical policies (teacher 1 + student 2), the teacher is modeled as a multi-arm bandit with a fixed output dimension (number of tasks), and the lower-level control policies of the students are shared. The intricacy of this pipeline leads me to question its generalizability and practical applicability. - **More rigorous comparison with current MARL algorithms, and need benchmark results on SMAC**, which is de facto the most standard benchmark for MARL algorithms. Please consider adding [MAPPO](https://github.com/marlbenchmark/on-policy), [HARL](https://github.com/PKU-MARL/HARL), and their multi-agent communication variant as your baselines. - Line 236-238, “However, current approaches that extend HRL to multi-agent systems or utilize communication are limited to a fixed number of agents and lack the ability to transfer to different agent counts”, this is an inaccurate claim because it has been done in the ICLR 2022 publication, [*ToM2C*](https://arxiv.org/pdf/2111.09189.pdf), which similarly uses the HRL with a population-invariant multi-agent communication mechanism. AFAIK this cannot be treated as "communication limited to a fixed number of agents". Please consider citing this work and changing your statement regarding the previous work.

Questions

- **I need more convincing results for proving "The Necessity of Curriculum Learning".** Why is an in-depth analysis of the teacher-student framework necessary? Despite its significantly increased implementation complexity compared to the original MAPPO algorithm, their performances appear roughly equivalent. Moreover, the MAPPO algorithm, provided sufficient exploration and large batch size, has already demonstrated state-of-the-art performance on both SMAC and GFR benchmarks. I presume an advantage of combining the ACL and the teacher-student framework lies in handling more challenging scenarios through incremental learning. To underscore the superiority of SPC over traditional multi-agent PPO methods, could you present performance data from the SMAC Super-Hard Map difficulty? This could include instances like 3s5z_vs_3s6z, where MAPPO previously underperformed significantly. - I am interested in understanding the implementation of multi-agent communication in Ray RLlib. It appears that the agents are exchanging messages before outputting their current actions. It's somewhat challenging for me to envision how this process is technically executed within the RLlib framework. I wish the code is available.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The limitations of this paper are only briefly mentioned in the last section.

Summary

This paper studies the multi-agent RL problem with sparse reward and a varying number of agents. The authors propose a novel automatic curriculum learning strategy to solve complex cooperation tasks in this setting. Their curriculum strategy involves a teacher component and a student component. The teacher component selects the sequence of training tasks for the student component using the contextual bandit algorithm with predictive representation of the student’s current policy as context. The student component is endowed with a hierarchical skill framework and population-invariant communication. They empirically investigate their proposed strategy in two environments (MPE and GRF).

Strengths

The paper is overall well-written, and the related work is extensively discussed. The theoretical results in this paper seem correct; I haven’t checked the details of the proofs. The population invariant communication module is an interesting contribution to dealing with the varying number of agents across tasks. It would be interesting to compare its effectiveness (on its own) against the existing methods to deal with varying numbers of agents [23, 24].

Weaknesses

I am unsure about the broader applicability of the contextual representation of the student policy using an online clustering algorithm. How much information will be lost in this process for a high-dimensional policy (e.g., that operates on image inputs)? Presented experimental results are not sufficient to validate the effectiveness of the proposed curriculum strategy (specifically the teacher component) in complex scenarios, given that in the MPE environment, the impact/necessity of curriculum is negligible.

Questions

Why EPC [9] is not used as a baseline in the experiments? The authors mention, "To ensure a fair comparison, we modify VACL by removing the centralized critic for MPE tasks.” Please explain why. In Figure 5, including the results for SPC w/o “both” HRL and COM would be good. Then, we can see the effectiveness of the teacher component (or curriculum). In Figure 3 (MPE environment), including the ablation study results (similar to Figure 5) would be good. It is also important to discuss/report the proposed strategy's computational cost/overhead (run time) compared to the baselines like VACL.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The paper is of an algorithmic nature and does not have any direct potential negative societal impact.

Thanks

Thanks for your rebuttal. I can see the amount of effort spent, and it addresses most of my questions, hereby raising my score. Please consider uploading the updated version of your manuscript asap.