Co-Learning Empirical Games and World Models

Summary

The paper addresses the problem of multi-agent RL by using learned world models over multiple potential opponent policies. The technique uses a Dyna-style algorithm to train the core policy with a combination of experiences generated through a world model and experiences playing against opponents. One evaluation demonstrates the world models benefits from training on data from multiple distinct policies. A second evaluation compares ways to use experiences generated from a world model to train a policy, showing pretraining on purely generated experiences is effective to warm-start a policy. An ablation study compares the proposed Dyna-PSRO model to vanilla PSRO on three MARL games, showing improvements over the PSRO model.

Strengths

## originality Modest. The paper extends existing lines of work on MARL and world models, specifically studying the question of the policy diversity for training the world model. ## quality Modest. The core results (figure 5) show clear improvements over PSRO. This is limited to a small number of games and the games themselves are relatively simple game domains. The paper does a good job of breaking down specific claims to isolated experiments. ## clarity Low. It was difficult to interpret many of the figures (questions and suggestions below). Generally the results of each experiment were hard to understand and would benefit from a single clear statement of the core outcomes in each section. ## significance Low. The core audience of this work is researchers in MARL and particularly those considering world models as a solution.

Weaknesses

The experimental results are promising, but would benefit from expansion. There are a few experiments that would help: 1. More games from MeltingPot. I hate asking simply for "more", but in this case it would help to show how well the agents perform on a wide variety of tasks. The results would help clarify where DynaPSRO benefits and may reveal limitations or areas for improvement. The wider set of results would give others confidence in the generality of the improvements gained by planning against diverse other agents. 1. More complex games. Consider more complex environments from PettingZoo (https://pettingzoo.farama.org/) or SMAC (https://github.com/oxwhirl/smac) that would highlight the potential of these algorithms in more compelx scenarios. This would help address the point that world models can become unstable and the value of strategic diversity in scenarios that support a much wider array of behaviors. 1. Scaling experiments. For example, when do prediction improvements level off when adding more policies? The experiments only examine adding 2 policies, which is a sparse sample of the space of strategies for most games. The evaluations would benefit from other baselines to compare. What other algorithms could be used aside PSRO? The full evaluation (last experiment) would benefit from a set of ablation studies. This could easily replace some of the planning experiments as the ablations would examine similar capabilities. I ask for ablations as these will be more convincing that the parts of Dyna being used add benefit over PSRO.

Questions

- [Q1] How do the results here scale? - This is a broad question, but for any RL algorithm it's important to understand the scaling and sample efficiency of the method. For example, how does increasing the amount of warm-start background training impact asymptotic performance or convergence rate? How does increasing the number of policies used to train the world model alter prediction performance? Reward obtained during training? - [Q2] Figure 2 - This figure is hard to understand and summarize. It may help to instead plot the average performances for each model combination and omit the heat maps. - [Q3] Figure 3 - What is "Plan: Model"? - Why does "Plan: Model" start strong then decay? (Opponents get better?) - Please define the legend (labels like "Plan: Model") as the names are not immediately obvious. - [Q4] Figure 4 - Is it correct to conclude that concurrent background planning has no benefit? - [Q5] What ablations can add more detail to Dyna-PSRO model? - What evidence is there that DynaPSRO was learning and responding to strategic diversity during these experiments? - [Q6] Section 3.2.2 DT planning - It seems like warm-start planning is sufficient to capture world model benefits. Why is that? - Consider adding an ablation: remove DT planning but retain BG:W and BG:C. This could clarify how much help the world model is for training the policy, independent of evaluating future states during execution. - [Q7] Figure 5 - Why do the DynaPSRO traces stop partway through the real experiences?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Yes. The work is focused on integrating world models into game playing agents and recognizes the preliminary nature of this work along with the potential risks introduced by using a simulation (world model) for decision making in real world applications.

Summary

This paper introduces a new approach to PSRO algorithms, where a world model of the environment is learned concurrently to the iterative PSRO strategy expansion.

Strengths

- The authors are right that the problem of having to re-learn policies from scratch is a large problem in the PSRO literature. Therefore, the idea to co-learn a world model alongside the expansion of the empirical game, in so taking advantage of the diversity of experience created by agents with slightly different best-response targets is an interesting approach to this problem. To the best of my knowledge this is also a novel solution to this problem. - I really like the presentation of the paper, and in general I think the authors do a very good job in terms of analysing the different moving parts of the framework in a reasonable manner.

Weaknesses

I have a few concerns with the paper, however none of these necessarily game-changing in my evaluation of the work. - I think the greatest misgiving I have with this work is that the related work seems to miss quite a large collection of PSRO papers that probably deserve mentioning. PSRO-style algorithms is a fairly small research area and I am surprised that the authors fail to make mention to many variants. In particular, as there is a section on strategic diversity itself in the paper, it seems odd that the authors have failed to comment on the line of works on diversity-based PSRO frameworks. For example, [1], [2], [3], [4], [5] are all diversity PSRO approaches. It also fails to place itself in the literature involving PSRO algorithms that attempt to speed up convergence times such as [6], [7]. - Furthermore, I was additionally surprised at the lack of comparison to NeuPL [8] which is another population-based framework attempting to similarly deal with the best ways to transfer information between agents in the population. - I do not necessarily believe that the authors need to benchmark against all of the approaches that I have listed. I do however believe the paper still needs work in terms of placing itself within the current literature on PSRO and other population-based frameworks. - Based on the above, my score is set at a borderline accept. However, I am willing to revise this upwards upon seeing a better framing of this work in the current literature. REFERENCES [1] Policy Space Diversity for Non-Transitive Games - Yao et al. 2023 [2] Open-ended learning in symmetric zero-sum games - Balduzzi et al. 2019 [3] Modelling behavioural diversity for learning in open-ended games - Perez-Nieves et al. 2021 [4] Towards unifying behavioural and response diversity for open-ended learning in zero-sum games - Liu et al. 2021 [5] A unified diversity measure for multi agent reinforcement learning - Liu et al. 2022 [6] Pipeline PSRO: A scalable approach for finding approximate Nash equilibria in large games - McAleer et al. 2020 [7] Neural auto-curricula in two-player zero-sum games - Feng et al. 2021 [8] NeuPL: Neural Population Learning - Liu et al. 2022

Questions

- Discussion starting line 208: I think this whole section on strategic diversity is very interesting. However, I am not entirely convinced as of yet that there does exist strategic diversity between the PSRO policies, especially as the changes in performance of Fig. 2 when the PSRO policies is only minor. My question to the authors is: was the strategic diversity of the PSRO policies actively checked? If so, am I missing some proof of that here? If not, why was it not checked? In my experience with PSRO policies, it is quite easy to end up with policies that act similarly even under the presence of different training data. - Discussion starting line 264: Could the authors comment a little further on the gap between the world model and the real game - do they foresee any scenarios where this gap may be far more damaging to performance than in the proposed experiments? Do they have any intuition for why it does not see to matter as much as one would expect in these experiments? - Why is this a better transfer learning method than NeuPL? I would be interested to hear the authors thoughts on this.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I think the authors actively engage with the limitations of the work.

Summary

The authors consider learning world models for deep reinforcement learning in combination with the construction of empirical games through PSRO. They first show that world models benefit from training on a diverse set of strategy profiles as can be generated through PSRO meta-game solvers. They then empirically show that PSRO best responses can enjoy sample efficiency benefits by training with simulated world model experience. Finally, they present Dyna-PSRO, in which PSRO best responses make use of a world model trained on all available experiences collected thus far in a run of the PSRO algorithm. Dyna-PSRO provides lower-regret solutions with higher sample efficiency than PSRO without a world model.

Strengths

- The paper is very well written and presented, and the experiments are well designed. - World models are seeing increased use in the RL community, and PSRO is one of the more practical and general methods currently available for finding approximate game solutions. This paper provides insights on how to properly combine the two and make improvements to PSRO's sample efficiency, which is one of its largest issues. - The proposed Dyna-PSRO method is sound. - While many implementation details are not present in the main paper, the appendix describes these details thoroughly.

Weaknesses

It would have been nice to see how current high-performing world model methods such as Dreamer, which employs latent state spaces [1,2] might perform with the same approach. It's not immediately clear if experiments like in section 3.3.2 would have had the same outcome. [1] Hafner, Danijar, et al. "Dream to Control: Learning Behaviors by Latent Imagination." International Conference on Learning Representations. 2019. [2] Hafner, Danijar, et al. "Mastering diverse domains through world models." arXiv preprint arXiv:2301.04104 (2023).

Questions

Was there a strong reason for adapting a Dyna-based method over other existing world model approaches?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

All limitations have been adequately addressed.

Summary

This paper describes combining two things: training a world model of a game, and doing Policy Space Response Oracles (PSRO) on the game. Doing PSRO involves getting a lot of episodes from the game (episodes are used to train the RL best-responses, and also to estimate the payoffs of the empirical game). The novel algorithm in this work (Dyna-PSRO) can be thought of as a modification of PSRO where those episodes are **also** used to train a world model (which is essentially a learned simulator of the game engine). Then, the world model is used to improve the training process of the best-response policies. Through experimental results, the authors show that this improved training process (based on Dyna) can cause the best-response learners to learn a stronger policy than the normal method when using the same amount of interactions with the real game environment. It does this by training the policy using trajectories from the world model (in addition to the usual trajectories from the real game environment), and by equipping the agents with one-step lookahead planning during training. This paper showcases experiments on the Dyna-PSRO algorithm in three games, and Dyna-PSRO outperforms PSRO in all three, as measured by an approximation of NashConv. The paper also shows experiments to measure the quality of the learned world model, to test the hypothesis that Dyna-PSRO results in a good world model. I think the paper has some flaws* in its current form, but the core work of the paper is good, the charts are beautiful, and the results are strong. --- *Edit: many of the abovementioned flaws were addressed during the rebuttal/discussion period.

Strengths

- Overall, the paper is well-executed. - It is well-written and polished. - There has clearly been a lot of time and effort put into the engineering and writing of this paper. There are 4 sets of thorough experiments in the main paper, and more in the appendix. - The figures are extremely readable. - The results concerning the performance of the best-response policies are impressive. - The effort will surely be helpful to future researchers: the research directions of (1) improving the efficiency of PSRO response calculations and (2) training better world models should continue to flourish, and the work presented in this paper contributes to both. - The research direction seems natural, especially in the direction of using world models to improve the performance of PSRO.

Weaknesses

I think the paper could be better in explaining or hypothesizing the "why" for a lot of things, even if just qualitatively. I think the paper states some conclusions too strongly. I think some things are not explained well enough and are confusing to the reader (at least, to me): - SumRegret metric: - It's really not clear from the main paper how SumRegret works (even though it is explained in the appendix). This could be clarified by defining the terms "method" and "combined game". - Also, I would feel a lot better if I saw results measured by an alternative metric, where the deviation set is the set of **all** policies. The $max_{\pi_i \in \bar{\Pi}_i}$ could then be approximated by just training one more response policy (as if doing one last epoch of PSRO). This seems like it would be a more accurate approximation of the Nash Conv. Is there any reason to use the metric in the paper instead of this? - Empirical Game Solution not described - Since the settings here are general-sum, it's probably important to specify what solution concept is used for the meta-strategies in the main paper (even though it is included in the appendix). - Experiments in Section 3.1 Strategic Diversity - Looking purely at Figure 2, the conclusion "Overall, these results support the claim that strategic diversity enhances the training of world models" does not ring true to me. For example, there are three world models which perform better on the metric (accuracy) used in Figure 2 than the most diverse one, for Observations. - Even if I look in the appendix at E.1, there doesn't seem to be significant evidence to support the conclusion: multiple world models have similar recall scores than the most diverse one, and the one trained without the random policy seems to have better scores. - I would be interested in seeing the cross-entropy loss instead of (or in addition to) the accuracy. - The discussion of the Decision-Time Planning results (3.2.2) seems incomplete: - "**The main outcome of these experiments is the observation that multi-faceted planning is unlikely to harm a response calculation,** and has a potentially large benefit when applied effectively. These results support the claim that world models offer the potential to improve response calculation through decision-time planning." (emphasis mine) - However, Figure 4 does show that decision-time planning causes the response to be *worse*: The solid blue line (top) has no decision time planning, and the dashed gray line (second from the top) has decision-time planning, and performs worse. - It would be nice if there was some discussion about this, perhaps an intuitive/qualitative reason why this is. - Dyna-PSRO results need more details (Figure 5): - For each experiment, how many policies (iterations of PSRO) were there? - Does each policy train for a fixed number of steps, or until some measure of convergence is reached? - Was "policy" vs. "strategy" ever defined like this before? In my opinion, we shouldn't define these terms like this, because they are usually considered synonymous. The terms I'm familiar with are "policy" or "strategy" for the former, and "meta-policy" or "meta-strategy" or "meta-strategy distribution" for the latter. Just my opinion! - I was very confused by the definition of World Model while reading the paper. - Even after reading it through entirely, I was under the impression that each player had their own world model, and that it implicitly modeled the actions of the opponent. - If one misses the bold notation of the definition of agent world model from line 137 to 141, it's easy to think that this is the case, especially since the phrasing is that "the agent learns and uses a ... world model" (instead of, say, "the agents learn and use a ... world model). - On one hand, the formal definition given for an "agent world model" is technically accurate, and I am just dumb. On the other hand, I suspect many of us are dumb, and will be similarly confused upon reading the paper. (Also, I'm not **that** dumb: it's really hard to tell that the O and A are bolded!) (Also, even if some of us are not dumb, we are likely lazy and will gloss over the explanation that boldface means joint.) This is all to say that I would suggest explicitly stating that the world model takes as input an observation and action from **each** player, and returns an observation and reward to each player. And that the world model does NOT model the actions of any player. - The bolding is nice, but it would be less confusing to **also** say "strategy profile" or "joint strategy" anytime this is meant instead of just "strategy" and something bold, as it's very easy to miss or forget what something bold means (plus, it seems incorrect to call a strategy profile a strategy). Also, maybe emphasize that sentence that explains what boldface means, so that readers don't miss it? - For example in line 774 and 775 of the appendix: - "This is typically not tractable, but instead draws are taken from a dataset generated from play of a behavioral strategy **σ**. And the performance of the world model is measured under a target strategy **σ∗**" - should be "strategy profile" and "target strategy profile" - and same in Line 159 and 160 of the main paper, and throughout section 3.1

Questions

- The world model is deterministic. Can it easily be made stochastic? - The Harvest game has stochasticity in the apple regeneration, right? Do you think the deterministic world model affected impacted anything in the experiments? - Section 3.1 Strategic Diversity - I'm guessing the world model implemented for the categorical harvest game outputs some logits that can be interpreted for each cell as a probability distribution over the possible entities in that cell. Then the world model takes the argmax of each cell and returns the observation? Or does it sample from the probability distribution? If not, why not? - What are the details of the weighted-average cross-entropy objective? - Did the empirical games often have multiple Nash equilibria with different values? If so, do the authors think it would change anything if specific Nash equilibria were sought -- e.g. welfare-maximizing? - Section 3.2.1 and 3.2.2 - Section 3.2.1: "The planner employs the x world model and the opponent plays the previously held-out policy." The opponent plays the previously held-out policy in all 3 cases, right? (Baseline, Plan: Real, Plan: Model) - It's stated that the planner switches from planned to real experience after 10,000 updates. Does this correspond to 1,280,000 experiences? - In Figure 3 and 4, I would be interested in knowing how the baseline experiments do when they are run with as much experience as the amount of real+planned experience that the planning models get. - On Figure 3, if the planning learners continue to outperform the baseline ones even on the real+planned exp x-axis, then what is the reason for that? Does the baseline policy settle into some local maximum that it can't escape, while the pretraining on the world model experience makes the planner walk into some better part of the policy space? - Is there any explanation for why the baseline returns are so variable and spiky compared to the planner's? e.g. in Figure 3. - Section 4: Is the decision-time planning only used to train? Do the final learned agents use any decision-time planning? - Seems like a lot of problems were the result of a subpar World Model - Could the performance of the world model be improved? - In the experiments of 3.1 - Are the world models trained to convergence? Do they achieve near-0 training loss? What do the training curves look like? - Did you try training the world model where one or more of the policies was a PSRO policy that played epsilon-random? - Out of curiosity, it would be nice to know what the results look like with wall-time on the x-axis. - Or, how long does it take to query the world model? How long does it take to query the real game environment? - Will the code be open-sourced?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations addressed

Response to Rebuttal

1. SumRegret metric. - > Regarding alternative metrics, we understand the suggested metric to be equivalent to evaluating our regret results without the final datapoint from each graph. Combined-game regret should provide at worst an equivalent approximation of regret as we are measuring each method's performance against an additional set of policies (those generated by the other method). We do agree with the general sentiment of needing stronger ways to approximate regret. - Oh, yeah, I can see how that's equivalent. In an ideal world, a stronger way to approximate nashconv might be to train response policies for longer so that they more closely approximate the exact best-responses (if they don't already near convergence by the end of training?). 2. Experiments on Strategic Diversity. - Thanks for the clarifications and the additional charts. It's interesting that the most diverse world model has the lowest observation cross-entropy, and it makes me feel more confident about the thesis that diversity makes better world models. I am also happy to hear that the claim in the paper was softened. 3. Discussion of Decision-Time Planning Results - hm, yeah I guess I missed the emphasis on *multifaceted* planning in my review. Seems reasonable. Is the claim that an agent with DT + BG is unlikely to be worse than one with neither? Or is it that DT+BG is unlikely to be worse than an agent with DT, and unlikely to be worse than an agent with BG planning? Seems like the natural interpretation is the latter. If so, it seems useful to have a chart comparing DT+BG to BG (but I appreciate that it's probably hard to figure out how/where to put such a chart). 4. Dyna-PSRO Results - Thanks for the details. To reiterate, I feel strongly that these details are important and should be included (in the appendix at the very least). 5. Was "policy" vs. "strategy" ... - > We employ "policy" as typical in the RL literature, to refer to the object generated in a BR calculation. We use "strategy" as typical in GT in the context when GT reasoning is more salient than BR (RL). A "meta-strategy" as used in some PSRO literature is just a "strategy" in the empirical game. - Yes, I agree it's not wrong to say that the strategy in the empirical game is a strategy. But I still think it's confusing because without knowing the way the terms are defined in the paper, a "strategy" could be interpreted as a strategy in the empirical game, or as a strategy in the original game. Using the term "meta-strategy" avoids this ambiguity. But maybe that's just me! 5. Q: Section 3.2.1 and 3.2.2. - Thanks for the new chart in the pdf. I agree that it's surprising that it's converged even given more training. Do the authors understand or have a hypothesis for why this is? This seems pretty counterintuitive to me, and makes me wonder if the explanation could be something like a software bug. 6. Q: Walltime of algorithms? - Thanks for the walltime measurements and explanations. I think it would be useful to others to put those in the appendix somewhere, including the explanations. ### summary Thanks for the thorough rebuttal. I appreciate the clarifications that alleviated some of my concerns about the sumregret metric and strategic diversity and DTP claims, the additional details about Dyna-PSRO experiments and strategic diversity experiments, the loosening of some claims, and that my suggestions for reducing confusion for the world model definitions and "joint" definitions were taken. I'm much more confident in recommending this paper for acceptance now.