Policy Space Diversity for Non-Transitive Games

Policy-Space Response Oracles (PSRO) is an influential algorithm framework for approximating a Nash Equilibrium (NE) in multi-agent non-transitive games. Many previous studies have been trying to promote policy diversity in PSRO. A major weakness in existing diversity metrics is that a more diverse (according to their diversity metrics) population does not necessarily mean (as we proved in the paper) a better approximation to a NE. To alleviate this problem, we propose a new diversity metric, the improvement of which guarantees a better approximation to a NE. Meanwhile, we develop a practical and well-justified method to optimize our diversity metric using only state-action samples. By incorporating our diversity regularization into the best response solving in PSRO, we obtain a new PSRO variant, Policy Space Diversity PSRO (PSD-PSRO). We present the convergence property of PSD-PSRO. Empirically, extensive experiments on various games demonstrate that PSD-PSRO is more effective in producing significantly less exploitable policies than state-of-the-art PSRO variants.

Paper

References (56)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer Mw947/10 · confidence 4/52023-06-16

Summary

This paper investigates the strategy exploration problem faced in learning-based game-solving algorithms. Their posed solution is to expand the empirical game with policies that are both approximate best responses to the current solution and maximally expands the policy hull. This method is motivated by an analysis on the exploitability of the population captured by the empirical game. They show a variety of performance benefits across a suite of games.

Strengths

Overall, this paper is well-rounded. The framing is clear and the related work very nicely characterizes a variety of the preceding work. While the contributed ideas of population performance and exploitability are lightly different than previous work, I think the interpretation brought forward by the small difference is rather nice. Moreover, this work treats the soft and difficult topic of diversity formally, which is a nice change of pace compared to preceding work focused on heuristics. They provide both an exact formulation of their metric, and a practical version that can be used in gradient optimization.

Weaknesses

The major weakness of this work is the confusion of empirical games and what they refer to as the "meta game". The crucial difference being that empirical games have only estimates of the true payoffs. PSRO uses an empirical game, but meta games may be useful for best-case convergence guarantees. Unfortunately this confuses and invalidates a variety of claims throughout the paper that mix up their usage. This is easy to correct, and I am very supportive of the paper post this correction. Another general point of weakness is the recurring claim of the problems with gamescapes. They are right, as they showed, that each iteration isn't gauranteed to lead to a monotonically improving game solution. However, this does not mean that the gamescape analysis wouldn't have long-term benefits. I suggest lightening the critique of gamescapes, and instead focus criticism only on their specific application of per-iteration policy exploration. I would have liked to see some analysis related to diversity and/or settings of their algorithm in the results.

Questions

- Convex Hull Enlargement and Occupancy Measure Mismatching appear to be functionally different from the other diversity measures because they focus on the impact of including another policy. I was wondering if you thought about how your methods or the others might be alternatively used between these two perspectives? - Population exploitability appears to be similar to the metrics defined in Lanctot et al.. "Population-based Evaluation in Repeated Rock-Paper-Scissors as a Benchmark for Multiagent Reinforcement Learning." Might be worth discussing as concurrent/previous work. - L159-160: There's nothing ill-posed with the idea of enlarging a gamescape generally. I would recommend refining this claim to focus on your point: for diversity. - L165-166: Exploitability increasing in the short-term might be OK if it facilitates better long-term exploration. I would suggest revising this claim to focus on when future iterations occurring are uncertain---I believe AnytimePSRO makes an argument like this. - L197: If you're defining the concept of fine policy representation could you make it more clear please? - L218: There is something potentially confusing about your definition of the sequence-form. Shouldn't it either include all possible trajectories to (s,a) or be only the probability chain of actions since perfect recall makes all paths unique? A quick comment about this in the paper I think would be very nice. - L245: This method requires a design choice of which opponent distribution diversity will be measured against. Could the authors comment on different settings they tried? It might be also useful to discuss how this is both a limit and potential advantage of this method when compared to other diversity metrics. You are both potentially more limited in your measure with a bad choice of opponent, or able to better measure by including behaviors outside of the empirical game. - For the AlphaStar888 experiments could you explain how you computed the diversity metrics given only the empirical game? The metrics are all dependent on the policies that are not directly available. - L323: I would be interested in having the details of what makes BD&RD-PSRO to tune and anything you might have learned from the attempted reimplementation in the appendix. - For Goofspiel, if you can easily get them, could you please also provide approximate regret curves? This can be estimated using the combined empirical game across methods. - For Goofspiel, are you using the final solutions produced by the algorithms to compare or regret? It's also not clear how long this experiment has been run and the respective population sizes.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

There is no focused discussion on either the limitations or broader impact of this work within the paper. The last paragraph, presumably their limitation sections, quickly discuss the role of trading-off diversity in game solving. I would have preferred to see discussions related: - Setting the opponent policy for policy comparisons - How the selection of the aforementioned opponent biases the measure of diversity - How changes in the opponent policy will require much larger computational costs - Results are partially on games that are not exactly solved, and the true relative performance of methods may differ - etc..

Reviewer YW2F5/10 · confidence 4/52023-06-28

Summary

This paper considers PSRO framework for approximating a NE in multi-agent non-stransitive games. The authors points out that the existing diversity matrics cannot guarantee that a more diverse policy space leads to a better approximation of the NE. To solve this issue, this paper proposes a new diversity metric and a corresponding algorithm, PSD-PSRO, which enjoys better convergence property.

Strengths

The paper is very well-organized and easy to follow. It provides a nice overview of related literature and states its contribution clearly.

Weaknesses

My major concern is the correctness of the result. In specific, I think the derivation of the gradient caculation in equation (15) might be incorrect. The derivation from line 484 to 485 in the supplementary material might need some reconsideration. I particular I don't think the first equation actually holds because the $\pi_i^{\min}$ should be a function of $\pi_i$. It might even be questionable whether this function is differentiable. I'm also curious about the computational complexity of the algorithm proposed in the paper. It seems that compared with other diversity metrics, the new metrics proposed is harder to compute because it needs to solve a minimization problem. What are the computational time for the numerical examples? And how does it compare with other methods? In Section D.3, the authors only provides the percentage, but it would be better to have a comparison among different methods.

Questions

See the Weakness section

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

3 good

Contribution

2 fair

Limitations

See the Weakness section

Reviewer jhEH5/10 · confidence 3/52023-07-03

Summary

In this work, the authors sought to justify the benefit of diversity by investigating properties of an expanding policy hull (PH), as supported by a set of policies in the context of population learning using Policy Space Response Oracle (PSRO). The authors discussed the limitations of expanding gamescape as a proxy for exploration in the policy space and reviewed several works on inducing diversity in prior works. Finally, the authors proposed their own diversity regularisation techniques and showed evidence that this approach leads to faster convergence to full game NE compared to baselines.

Strengths

* originality / significance I find the discussion on what to focus on when it comes to defining policy space diversity interesting, thorough and well motivated. There have been several works on this topics in the literature (as surveyed by the authors) but prior works presented different definitions and motivations for the need for policy space diversity which this work attempts to address. Having a unified language to discuss diversity with the goal of game-solving is sorely needed by the community. * quality / clarity Empirical results of PSD-PSRO appear thorough, though I have a few clarifying questions on the results. I would encourage the authors to revise the presentation to make the experimental protocol more clear and convincing. The theoretical justification for policy hull expansion appears sound.

Weaknesses

* Clarity: the authors presented prior works from the angle of NE game-solving. However I believe this is not necessarily the stated goal in all prior works? For instance, PSRO_rN was proposed as a way to expand gamescape (rather than a method that solves for a full game NE at faster convergence rate). This should be clear from their discussion on the disc game where focusing on NE game solving would lead to a contracted gamescape, i.e. the final solution would not contain pure strategies that would be interesting to represent as they optimally exploit certain opponents (as compared to the NE mixed strategy). I would encourage the authors to revisit the stated motivation for these prior works and re-state their motivations more precisely. * Scope: the scope of this work could be stated more precisely? Several prior works on population diversity explicitly focused on symmetric two-player zero-sum games (i.e. gamescape was defined in this regime). The authors appear to state their results generally (e.g. L103, $\Pi_i$, $\Pi_{-i}$), do you aim to make statements about n-player general-sum games in this paper? If so, how should one interpret e.g. Equation 7? Should I interpret $\sigma_{-i}$ as the joint distribution across all coplayers of player $i$? What would be the dimensions of the relative population performance in that case? * Preciseness: in several places the authors seem to imply that vanilla PSRO would not converge to a full game NE while there are convergence guarantees when using suitable best-response solvers. Perhaps revisit these statements to make it clear that the intention is to offer faster convergence rate? E.g. L266 and L174.

Questions

* Eq 14: how is the min operator implemented as there are infinite such mixed policies? If each such sample corresponds to a mixture-policy mixed at the start of an episode, would you draw multiple samples from the same distribution in the population simplex? * Policy Hull: is the definition of Policy Hull the same as Population Simplex described in https://arxiv.org/pdf/2205.15879.pdf? I.e. they both correspond to all convex mixtures supported by a set of policies? If so, would simplex-NeuPL be a useful representation scheme when searching for diverse BR policies? If not, could you clarify their differences? * L269: ... PSD-PSRO over PSRO in approximating a NE. Is this statement empirical or do you expect better convergence rate in theory which can be stated more explicitly? * Fig 1 (Left): nit: could you specify finer-grained y-axis for these figure? * L302: instances of Goofspiel as implemented in OpenSpiel could be solved exactly, could the authors clarify which game setting is used? Could similar analysis be done in more restricted setting such that we can show exact convergence in Goofspiel? * AlphaStar888: the dataset presents the payoff between 888 RL agents, but the diversity regularisation term seem to suggest needing access to state action in the underlying game. Could you clarify how diversities between these RL policies are computed? * Fig 3: exploitability in Leduc Poker seems to be reducing still (at 0.2 currently), would it reduce further to lower level? Are there computational bottleneck that would prevent further training? How are best-responses solved in this experiment? By RL? Or by exact solvers from OpenSpiel? My reservation in accepting this work is primarily due to my confusion as stated above, I would encourage the authors to further clarify their empirical results and experimental settings.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

This work is methodological and does not present immediate potential for negative societal impact.

Reviewer 6jtZ7/10 · confidence 3/52023-07-06

Summary

This paper studies how to promote policy diversity in two-player zero-sum games to better converge to a Nash Equilibrium (NE) under the algorithm framework of policy-space response oracles (PSRO). While previous works propose various diversity metrics to enlarge the gamescape, this paper shows that the enlargement of gamescape does not necessarily contribute to a better approximation of NE. Therefore, the authors propose a new diversity metric in policy space, namely KL divergence, to alleviate this issue. Theoretical results show that the proposed algorithm, PSD-PSRO, enjoys good convergence properties to NE. Experiment results in several common benchmarks for PSRO demonstrate the strong empirical performance of PSD-PSRO.

Strengths

+ The main logic and presentation of this paper is very clear. Personally, I like the writing paradigm of first analyzing drawbacks of previous approaches, motivating the algorithm, and then showing strong theoretical and empirical results with the proposed algorithm. + This paper points out an important issue of the main-stream diversity-motivated PSRO algorithms, which in my opinion is a solid contribution to the community. To resolve this issue, the proposed algorithm is quite simple (with KL divergence) but effective.

Weaknesses

+ The defined diversity metric in Eq.14 is simple in form but very complex in derivation (from line215 to line234). The fancy derivation process essentially prevents readers from obtaining the algorithm insights. I will ask detailed questions in the next section.

Questions

+ I'm not quite familiar with the notation in two-player zero-sum games. In Eq.11, xi seems to be the occupancy measure over the entire state-action space. Why is the state transition probability omitted here? + In line221 the authors say that Eq.11 is independent of environment dynamics, but why? Eq.11 involves the trajectory \tau. + From my point of view, the derivation process from line215 to line234 tells the reader that KL divergence essentially measures the differernce of policies w.r.t. the sequence representation, but in my opinion this is not that straightforward. If this is generally true in RL (or two-player zero-sum games), I think there should be previous works illustrating this point as a theorem. In line230, why \beta_s can depend on the opponent policy? Depending on the opponnet policy is intuitively true, but I didn't see any theoretical evidences that it should be this way. Why does setting \beta_s in this way not changing the property that Eq.13 still measures policy differences in terms of sequence representation?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have addressed several limitations in Sec.7. I didn't see any additional major limiatations.

Reviewer jhEH2023-08-11

Thank you for your explanation. I'm happy with most answers except the empirical results on exploitability in e.g. Leduc Poker (and also goofspiel) still seem relatively high (at ~0.2) if I understood correctly. Especially if exact BR solver is used. See https://arxiv.org/pdf/2201.07700.pdf Figure 4 (middle) where exploitability in this game using Double Oracle with exact BR solver is much more clearly approaching 0. While I do not expect to see faster convergence to an NE of the fully game, I do expect the algorithm to converge to a full game NE eventually at a reasonable rate; is there something I'm missing from the experimental results?

Authorsrebuttal2023-08-11

Many thanks for your quick reply.

Thanks for your quick reply! We appologize that we misunderstood the BR you mean in **Q10**. In fact, there are two "BRs". - One is the BR to the meta policy of the opponent at each iteration in PSD-PSRO training, where we use RL algorithm (PPO) to train against the opponent’s meta policy. - The other is the BR we use to evaluate the exploitability of policies. To calculate exploitability, we need a BR solver. In the definition of exploitability (Eq 1 in main paper), we have $\arg \max_{\pi'}u_i(\pi_i,\pi_{-i})=BR(\pi_{-i})$. We use an exact BR solver (available in OpenSpiel) to evaluate policies' exploitability. Hope we make it clear this time. Also, we believe the results of ~0.2 exploitability for Leduc (when approximate BRs are used) are competitively strong: - The results of SOTA methods demonstrated in Fig 3 by OpenSpeil (OpenSpiel: A Framework for Reinforcement Learning in Games, https://arxiv.org/pdf/1908.09453.pdf) use more than $10^7$ episodes to reach the average performance of ~0.3. In comparison, PSD-PSRO use less than $3*10^6$ episodes to reach around 0.2. - Also, for the comparison to Anytime PSRO ( https://arxiv.org/pdf/2201.07700.pdf ), we should actually compared with Figure 6(a), where we believe PSD-PSRO is no worse than Anytime PSRO.

Authorsrebuttal2023-08-16

May we ask have we clarified all your confusion?

Dear Reviewer jhEH, you mentioned in your official review that "My reservation in accepting this work is primarily due to my confusion as stated above". After our rebuttal, you followed with "I'm happy with most answers except the empirical results on exploitability" As the author-reviewer discussion phase is coming to an end, we would like to know whether our latest reply on the issue of "empirical results on exploitability" clarify your last concern. We would be happy to discuss if there is still anything unclear. Many thanks again for your time and insightful review. We really appreciate it.

Reviewer jhEH2023-08-16

Thank you for the explanation and I agree that the empirical results are as expected. I will increase my initial rating accordingly.

Reviewer Mw942023-08-11

Thank you for addressing some of my questions. I would like to follow-up on several of the questions that I feel are still unresolved: - Q3: In the question about Convex Hull Enlargement and Occupancy Measure Mismatching I did not mean to request you defend the novelty or superiority of proposed methods. My question is specifically that these metrics "focus on the impact of including another policy" as opposed to measuring diversity of say a fixed set or between two policies. I was curious if you had any _thoughts_ or _speculations_ about the role this might play in their performance? Moreover, have you _thought_ about how other diversity metrics might be recast to focus on "the impact of including another policy", or how your methods might be recast in other perspectives? - Q6: I agree with the authors intention, which is to say that adding a single policy can reduce performance. However, the point I'm making ontop of that is this not necessarily a bad thing. If you know you're going to continue the game solving process and add more policies in the future, taking a loss in the short term for larger gains in the long term (exploration v exploitation) could be very advantageous. I am suggesting that the discussion in L164-166 consider this instead of implying that this is a strictly bad result. Does this make sense? - Q10: Could the authors link where the acquired the AlphaStar888 data? I've tried to track it down and cannot easily find it. It looks like the github you linked might have the data, but how did they end up with it? They're different authors from the StarCraft team. It would be nice to have traceability of this data and confirmation that this is the version of the data you're using. I urge the authors to cite that this is the source of the data and mention it explicitly in their work. - Q11: If you did not attempt at all to replicate their work then I would modify L322-323 in the paper, because I believe its phrased in a misleading way. "BD&RD-PSRO is hard to tune (the code is not available)" to me suggested that you had _tried_ to tune the algorithm. I would have preferred if the authors had attempted to replicate it to the best of their ability, as it is a very pertinent baseline, and then provided discussion about any difficulties associated with it in the text. As of now the claim in the paper "is hard to tune" is strictly speculation---and frankly, most deep reinforcement learning methods are hard to tune, so I don't think this is particularly unique. I realize at this point in the process it's unreasonable to run this experiment, so at least the language in the paper needs to be corrected. I would also strongly urge the authors to actually attempt to run this baseline, especially for future versions of this paper.

Authorsrebuttal2023-08-12

Many thanks for your reply

We provide responses to your remaining questions. We would be happy to discuss if there is still anything unclear. **A3**: Thanks for your clarification. Within the context of PSRO, where new policies are iteratively added to the population, we believe it is more straightforward to have a diversity (for the purpose of NE solving) that evaluates the impact of a new added policy, as opposed to any other form of diversity (e.g., measuring diversity of a fixed set or between two policies). Yet, these are all connected in the sense that one form of diversity can be written in another form. For instance, the impact (with respect to a population $\Pi$) of policy $\pi_a$ versus $\pi_b$ can be written as Div($\Pi \cup \pi_a$) versus Div($\Pi \cup \pi_b$). In PSRO, how to evaluate the **impact** of introducing a new policy (say $\pi_i$) plays an **important** role for the final performance (finding a full game NE). There are two aspects in our argument: - There should be an anchor (either a single policy or a population of policy), when we talk about the impact of $\pi_i$. We think for PSRO the anchor is better to be a population than any fixed single policy, simply because PSRO is a population-based algorithm (the final output solution is produced by the entire population, i.e., the meta NE). Nonetheless, it is worth mentioning that the impact of $\pi_i$ with respect to a population $\Pi^{t}_i$ is often approximated by a min policy $\pi^{min}_i$ 'contained' in $\Pi^{t}_i$: distance($\pi_i$, $\pi^{min}_i$). - When we talk about the similarity between two policies, there should be a metric. First, we need a representation of policy. For the purpose of approximating a NE, we define what is *a fine policy representation* (Line 205), which should have the property of linearity and bijection. For instance, the pay-off vector in the gamescape is **not** *a fine policy representation*, while the sequence form (Eq 11) and the occupancy measure are. Afterwards, we need a similarity function on the representation. Previous work on occupancy measure uses the f-divergence, while we use the Bregman divergence. Have you thought about how other diversity metrics might be recast to focus on "the impact of including another policy". - The occupancy measure mismatch between $\pi_i$ and $\Pi^t_i$ might be recasted for our purpose. However, as far as we know in the current literature, f-divergence on occupancy measure is roughly approximated using predictions of neural networks. In contrast, we have a closed form for our Bregman divergence on the sequence form (Eq 13). how your methods might be recast in other perspectives? - We believe our definition of *fine policy representation* might be helpful for other purposes as well, because it provides a guideline on how to represent a policy properly. - We believe our Bregman divergence on the sequence form (Eq 13) may be useful for clustering policies, which can be helpful to downstream tasks, such as opponent modeling. **A6**: It makes absolute sense that "taking a loss in the short term for larger gains in the long term". The problem with diversity on gamescape is enlarging the gamescape in either short term or long term does not **necessarily** lead to a better approximation to a NE, . We understand that the current writing in L164-166 makes you think we are implying that taking a loss in the short term is a strictly bad thing. Thanks for pointing this out, and how about we change L164-166 to the following: "In other words, enlarging the gamescape in either short term or long term does not necessarily lead to a better approximation to a full game NE." **A10**: We acquired the AlphaStar888 data from https://github.com/oslumbers/diverse-psro. We have sent an email to ask one of the authors of diverse PSRO for the exact source (presumably from DeepMind) of data. Many thanks for your suggestion, and we will cite the DeepMind source of data in the revision once we get it. **A11**: Thanks for your suggestion. We would change the text "BD&RD-PSRO is hard to tune" to "the code for BD&RD-PSRO on complex games, where f-divergence on occupancy measure is approximated using prediction errors of neural networks, is not available". BTW, we did compare to BD&RD-PSRO on simpler games in Figure 1&2 in the paper. Reimplementation of the full version of BD&RD-PSRO is definitely one of our future work.

Reviewer Mw942023-08-14

Thank you for answering my questions and entertaining my request for speculation. I think these changes will help future readers of the paper. I am happy to move my score up to a 7.

Reviewer YW2F2023-08-13

Thank you for your response! I think A2 and A3 have addressed my questions with respect to the computational complexity. However, I am still not convinced by the argument in A1. In specific I don't think step 4 in A1 is convincing enough for me. I guess my question is still, since the gradient is taken on $\pi_i$ and that $\pi_i^{min}$ is a function on $\pi_i$, how come the gradient term can ignore the gradient on $\pi_i^{min}$? Also the response says ''$\pi_i^{min}$ indeed depends on $\pi_i$" yet also mentioned in step 4 that the term is "not a function of $\pi_i$". I'm quite confused by the seemingly conflicting arguments. It would be nice if the authors could write down some more rigorous arguments to show that the proof is correct.

Authorsrebuttal2023-08-14

The fourth point in A1

Thanks very much for your reply. The last term in Eq 21 ($\mathbb{E}[\nabla \text{KL}(\pi_i(s_i) \| \pi_{i}^{min}(s_i))]$), which is a gradient with respect to the parameters of $\pi_i$ of a KL between $\pi_i$ and $\pi_i^{min}$. As we said in **A3**, we sampled multiple policies in the policy hull of $\Pi_i^t$ and used the sampled min policy (say $\hat{\pi}_i^{min}$) to approximate the true $\pi_i^{min}$: - We first obtain the fixed policy $\hat{\pi}_i^{min}$. - Then, the last term in Eq 21 becomes $\mathbb{E}[\nabla \text{KL}(\pi_i(s_i) \| \hat{\pi}_{i}^{min}(s_i))]$, where $\hat{\pi}_i^{min}$ is a **constant**. More precisely, if we undstand it correctly, the reviewer's question is whether the above two steps make sense in approximating $ \nabla \min_{\pi_i^k} \mathbb{E} [ \text{KL}(\pi_i(s_i) \| \pi_{i}^{k}(s_i))]$. The answer is yes: - In Equation 15 in the paper, we have used a property that $\nabla_{\theta} \mathbb{E}[\text{KL}(\pi_i(s_i; \theta) \| \pi_i^{min}(s_i))] \in \partial_{\theta} \min_{\pi_i^k} \mathbb{E}[\text{KL}(\pi_i(s_i; \theta) \| \pi_{i}^{k}(s_i))] $, where $\pi_i^{min}(s_i) = \arg\min_{\pi_i^k} \mathbb{E}[\text{KL}(\pi_i(s_i; \theta) \| \pi_{i}^{k}(s_i))] $. - To show the correctness of the equation above, we can prove a proposition as follows: > *Proposition*: For any locally Lipschitz continue function $f(x,y)$, assume $\min_y f(x, y)$ is well-defined, then $ \partial_x f(x,y) |\_{y \in \arg\min f(x, y)} \in \partial_x \min_{y} f(x, y)$ (1), where $\partial f$ is the generalized gradient [1], which is the convex hull of the limits of the form $\lim \nabla f (x + h_i) $ where $h_i \rightarrow 0$. - The proposition is an immediate result according to [1]. > *proof*: According to Theorem 2.1 (property (4)) in [1], (1) is immediate, as $\partial_x \max_{y} g(x, y)$ (or $\partial_x \min_{y} f(x, y)$) is the convex hull of $\\{\partial_x g(x,y) |{y \in \arg\max g(x, y)} \\}$. - Note that the generalized gradient $\partial_x f(x, y)$ is the set of subgradients when $f(x, y)$ is convex on $x$ [1] (Proposition 1.2). - This theory has been used widely in existing literature. For example: 1. GAN (Proposition 2) [2], which shows that the subgradient of the generator G can be obtained by computing the gradient of G at the optimal discriminator D. 2. ED (Theorem 1) [3], which proves that the gradient of $\pi_{\theta_i}$'s value against a best response, $\beta = b_{-i}(\pi_{\theta_i})$, is a generalized gradient of $\pi_{\theta_i}$'s worst-case value function, i.e., $\nabla_{\theta_i} v_{i}(\pi_{\theta_i}, \beta)|\_{\beta = b_{-i}(\pi_{\theta_i})} \in \partial_{\theta_i} \min_{\pi_{-i}} v_{i}(\pi_{\theta_i}, \pi_{-i})$ Hope the above clarifies your concern, and we will make it clearer in the revision. We would be happy to discuss if there is still anything unclear. [1] Clarke, Frank H. "Generalized gradients and applications." Transactions of the American Mathematical Society 205 (1975): 247-262. [2] Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems 27 (2014). [3] Lockhart, Edward, et al. "Computing approximate equilibria in sequential adversarial games by exploitability descent." arXiv preprint arXiv:1903.05614 (2019).

Reviewer YW2F2023-08-14

I sincerely thank the authors for the detailed explanation! I think now I am convinced that the proofs as far as I read make sense to me. It would be nice if the authors could add the elaboration (especially add the proposition from Theorem 2.1 in [1]), which might help readers like me to better understand how the derivation is done. I'm changing my evaluation score to 5 based on the conversation.

Authorsrebuttal2023-08-15

It is our pleasure, and many thanks for your insightful review. We will refine the paper according to your suggestions.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC