Block-Coordinate Methods and Restarting for Solving Extensive-Form Games

Coordinate descent methods are popular in machine learning and optimization for their simple sparse updates and excellent practical performance. In the context of large-scale sequential game solving, these same properties would be attractive, but until now no such methods were known, because the strategy spaces do not satisfy the typical separable block structure exploited by such methods. We present the first cyclic coordinate-descent-like method for the polytope of sequence-form strategies, which form the strategy spaces for the players in an extensive-form game (EFG). Our method exploits the recursive structure of the proximal update induced by what are known as dilated regularizers, in order to allow for a pseudo block-wise update. We show that our method enjoys a $O(1/T)$ convergence rate to a two-player zero-sum Nash equilibrium, while avoiding the worst-case polynomial scaling with the number of blocks common to cyclic methods. We empirically show that our algorithm usually performs better than other state-of-the-art first-order methods (i.e., mirror prox), and occasionally can even beat CFR$^+$, a state-of-the-art algorithm for numerical equilibrium computation in zero-sum EFGs. We then introduce a restarting heuristic for EFG solving. We show empirically that restarting can lead to speedups, sometimes huge, both for our cyclic method, as well as for existing methods such as mirror prox and predictive CFR$^+$.

Paper

References (49)

Scroll for more · 37 remaining

Similar papers

Peer review

Reviewer XCKi5/10 · confidence 1/52023-06-27

Summary

This work proposes a cyclic coordinate descent method to solve the two-player zero-sum extended form game (EFG) and derives the convergence.

Strengths

To me, solving problems with non-separable constraints by coordinate-descent-type methods is novel and interesting. Therefore, I believe that this is indeed a contribution.

Weaknesses

I'm not familiar with this area so please forgave me if I made some mistakes. 1. Possibly expensive computational cost per iteration. Following the literature survey in the manuscript, I read reference [1] which studies coordinate descent for solving optimization problems with non-separable non-smooth objective functions. I found that in both the algorithm in [1] and the algorithm in this paper, although only partial gradient is needed at each update, the proximal step with respect to the whole nonsmooth function is needed. More specifically, in Lines 8 and 11, the argmin step might be difficult and computationally expensive since the argmin step is over the whole X and Y and involve all the blocks. 2. This concern follows the first one but focuses on the experiments: the comparison with non-coordinate algorithms may be unfair. The current comparison is in terms of the number of full gradient computations. However, gradient computation is not the only computation cost, Lines 8 and 11 also cause computation costs. I would suggest a comparison in terms of wall clock time.

Questions

no

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

I do not see any potential negative societal impact.

Reviewer rJqj6/10 · confidence 3/52023-07-05

Summary

This paper combines the local prox update technique with the extrapolated cyclic algorithm and proposes the ECyclicPDA algorithm. While the local prox update technique is well understood, the authors reinterpret it as a coordinate method (CM). Then, the method is combined with a new extrapolated method. Theoretical analysis shows that the proposed algorithm can converge at a rate of O(1/T).

Strengths

1. This paper presents a new understanding of the local update rules of OMD with dilated DGF. The combination of local OMD updates and extrapolated updates is original and interesting. 2. The theoretical results seem sound. 3. The paper is well-written.

Weaknesses

1. It is hard to understand why we want to reinterpret the well-known local update rules of OMD with dilated DGF to CM. As we know, people already compute the strategy in a bottom-up fashion in previous work [1, 2, 3]. So, I think the main contribution is the combination of the local update method and the extrapolated cyclic method. 2. However, the ECyclicPDA algorithm seems less efficient than the traditional MP algorithm. For every infoset, an extra traversal of the subgame rooted at the infoset is needed to compute the “extrapolated” vector. This could be infeasible for large-scale games. 3. The experimental results show that the proposed ECyclicPDA performs better than MP. The results are not surprising when considering that ECyclicPDA can be very time-consuming. 4. The algorithm does not compare with related optimistic OMD [4] algorithms for EFGs. [1] Farina, Gabriele, Christian Kroer, and Tuomas Sandholm. "Optimistic regret minimization for extensive-form games via dilated distance-generating functions." Advances in neural information processing systems 32 (2019). [2] Farina, Gabriele, Christian Kroer, and Tuomas Sandholm. "Better regularization for sequential decision spaces: Fast convergence rates for Nash, correlated, and team equilibria." arXiv preprint arXiv:2105.12954 (2021). [3] Liu, Weiming, et al. "Equivalence analysis between counterfactual regret minimization and online mirror descent." International Conference on Machine Learning. PMLR, 2022. [4] Lee, Chung-Wei, Christian Kroer, and Haipeng Luo. "Last-iterate convergence in extensive-form games." Advances in Neural Information Processing Systems 34 (2021): 14293-14305.

Questions

none

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have stated the limitations properly. No potential negative societal impact.

Reviewer Y4SP7/10 · confidence 3/52023-07-09

Summary

This paper introduces ECyclicPDA, a new first order method for solving extensive form games. The main idea is to implement something in the spirit of coordinate descent where improving directions can be found by considering a part of the current iterate in isolation. Empirical results show it generally outperforms other first-order methods and approaches the performance of CFR+. Additionally, a heuristic to restart the process is introduced which improves the performance of both ECyclicPDA and CFR+

Strengths

ECyclicPDA is a nice contribution that takes the growing collection of FOMs for EFGs in a new direction. The analysis seems to involve non-trivial technical innovations. The empirical results are convining and show continued progress toward cloasing the gap with CFR-based approaches. The restarting heuristic is also very interesting, and the ability to get performance improvements with CFR+ is particularly nice (and intuitive in hindsight).

Weaknesses

As far as I can tell, the details of how restarting is implemented are never clearly explained. The clearest explanation I can find is in the introduction on lines 88-91, but even taking that as the full specification, I don’t see where the tuning of the parameter about when to restart is specified. It would also be nice to have a bit of explanation / intuition for how restarting is benefiting various algorithms. For CFR+ I can see how resetting the regret sums and the averaging process could be beneficial. For FOMs I’m less clear why it is useful. Is it purely from resetting the averaging process?

Questions

Please comment on the questions about resetting.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Adequate

Reviewer udTT6/10 · confidence 3/52023-07-09

Summary

The draft considers solving the extended-form game (EFG) with extrapolated block-coordinate descent methods, proving that it achieves O(1/T) convergence. The authors further show that with a restarting strategy, the proposed algorithm may be comparable sometimes to state-of-the-art algorithms like CFR+. In my understanding, the proposed algorithm is essentially a generalization of the CODER algorithm proposed by Song [37] in a way from coordinate descent (CD) to blockwise CD and specializes the domain to be linear functional and treeplex. I.e., the algorithm is essentially the same (line-by-line corresponded) as CODER but generalizes the scalar coordinate to blocks of "separable" variables. For the specialization, the authors focus on the context of EFG with bilinear score function and 2-player treeplex (linear probability simplex), where the prox is taken on L1-norm instead of the L2-norm by CODER. Implementation-wise, the proposed algorithm uses the recursive structure of the treeplex for efficient linear time computation.

Strengths

The major contribution of the draft is that it provides an instance to efficiently implement the CODER algorithm on EFG with blockwise L1 proximal and shows that with restarting, such implementation may be comparable with the state-of-the-art algorithm like CFR+. It also analyzes the scenario and provides an O(1/T) convergence rate without dependence on the number of variables. To summarize, the draft shows that a simple CODER extrapolation works well when taking L1 specialization on EFG.

Weaknesses

However, I would like to claim that the original CODER method may already cover the blockwise updates. I.e., if you look into the proof of CODER, it generally does not use the scalar property of the proximal and treats variables as blocks. They are using the $d$-dimension notation for simplicity and already claiming they are doing block CD in their abstract. In the EFG case, the d=2, so naturally (if the proximal works), the convergence complexity doesn't depend on the number of variables since blocks=2 is a constant. So the draft's theoretical result (independence on the number of variables) is not surprising. And the proposed method does not relax the "separability assumption" since the 2-player treeplex is separable as two blocks. The major difference/contribution here is that it performs the analysis on the L1 proximal and does the analysis. Further, although it's good to see that the simple method works with restarting, the restarting part doesn't have theoretical support. I.e., without restarting, the theoretically supported method is not comparable to the state-of-the-art. And with restarting, the "full" method is comparable but not really much better than CFR+. Thus these factors put the paper in a borderline condition. However, the analysis seems solid, and I would give a borderline acceptance.

Questions

1. (Figure 2) Only doing the step-size optimization for your method and not the baseline is tricky for the experiment. For fairness, it should be either all constant step-size or all step-size tuned for the baseline. Otherwise, the reader cannot be sure whether the proposed method is better than MP. 2. (Figure 4) The vertical drop in the curves looks like either a numerical issue or a wrong optimal value. 3. (Algorithm 1) Adding parenthesis and denoting what you maintain in the computation may be good for clarity.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Not applicable since the paper does not have a limitation section.

Reviewer u8q25/10 · confidence 1/52023-07-13

Summary

The paper introduces a novel method for solving Extensive-Form Games based on a block-coordinate approach. The authors motivate and explain their idea and experimentally evaluate its performance in terms of primal-dual gap in four different games. The method features a favorable theoretical convergence rate but the empirical results are often comparable or worse than existing methods in terms of the duality gap.

Strengths

- The paper is well written and free of grammar and stylistic issues. - The introduction to the problem is very approachable even to domain non-experts. - Experimental validation of the block construction strategy choice. - Favorable converge properties.

Weaknesses

- It seems that in empirical studies the method is consistently worse than PCFR+ (Fig 2 and 4). I have a difficulty to find a complete justification for this in the paper. - Only a theoretical convergence rate and not an actual computation cost measured in units of time is reported. - The plots lack error bars (unless no variance is possible to obtain). - The contributions are explained indirectly. They could be listed in a more compact and more explicit form (e.g., a numbered list) to allow for critical assessment.

Questions

- How does the run-time of the tested methods compare in practice? *** Rebuttal Acknowledgment *** I have read the author's rebuttal. The answer covers my questions well and the explanation the authors provides logical justification of the potential shortcomings. Therefore, I am of the opinion that the paper is sound and I increased my rating accordingly. I do not go higher due to my very limited familiarity with the subfield.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

- The discussion of limitation is not very thorough and it is mostly limited to a mention of a lack of understanding of the game type effect on the performance.

Reviewer Lrca7/10 · confidence 1/52023-07-24

Summary

This paper develops a cyclic block-coordinate-descent-like method for two-player zero-sum extensive-form games (EFG). Such methods for EFG are difficult due to non-separable nature of block structure of the problem. The decision problem for a player in a EFG can be formulated using Treeplex, for which regularizing functions can be constructed through the framework of dilated regularizers. These dilated regularizing functions allow recursive prox computations. This paper utilizes this frame work to develop an extrapolated cyclic algorithm to perform pseudo-block updates. They demonstrate O(1/T) convergence rate for two-player zero-sum Nash equilibrium using this method, and provide a specific algorithmic implementation which shows that runtime of the proposed method is independent of the number of blocks. Experimental evaluation on EFG benchmark games is performed using three different dilated regularizers -dilated entropy, dilated global entropy, and dilated $\ell_2$ with different block construction strategies. Experiments demonstrate the benefit of using blocks over non-block based approach in some games, and no significant diffference between different block construction methods in others. The results show improved performance over the state of the art first order method (Mirror-Prox). Further, they introduce a restarting heuristic which speeds up the proposed method as well as the baselines.

Strengths

The paper is generally well written, describing necessary background to help readers understand the paper. The proposed algorithm seems novel, though I must admit I am not familiar with the literature. The experimental evaluation seems convincing, the proposed method outperforms SOTA FOM

Weaknesses

line 105-``As discussed before RCMs are not applicable to our setting''. It is not discussed anywhere in the paper why randomized coordinate methods are not applicable. The paper could include a discussion in the related work about [i] which develops primal-dual coordinate methods for solving bilinear saddle-point problems. [i] Carmon et al. Coordinate Methods for Matrix Games

Questions

-

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss open question of why restarting work with regularizers other than $\ell_2$. The authors adequately addressed the limitations.

Reviewer c8Uw5/10 · confidence 3/52023-07-25

Summary

The proposed Extrapolated Cyclic Primal-Dual Algorithm (ECyclicPDA) is a solution technique for large-scale extensive-form games (EFG) that resembles first-order coordinate descent. To enable pseudo block-wise updates, it takes use of the recursive nature of the proximal update caused by dilated regularizers. A restarting heuristic for EFG solution is also presented by the authors, and it has the potential to significantly speed up both their cyclic technique as well as other current methods like mirror prox and (predictive) CFR+.

Strengths

In summary, the ECyclicPDA algorithm introduces a novel and effective solution technique for tackling large-scale sequential games, demonstrating superior performance compared to current state-of-the-art techniques in some scenarios. The paper examines prior research on coordinate descent methods and first-order techniques for solving EFGs, highlighting the originality and benefits of their proposed ECyclicPDA approach.

Weaknesses

The execution times and computational demands of the ECyclicPDA method in comparison to other cutting-edge approaches are not thoroughly analyzed in the paper. It is essential to evaluate such a comparison in order to ascertain the proposed algorithm's applicability and practical effectiveness. Although it is mentioned that the runtime of ECyclicPDA is independent of the number of blocks, a comprehensive evaluation is still lacking. Additionally, the paper overlooks competitive alternatives like DCFR, despite making references to these methods.

Questions

1) The authors claim, "For the first time, a first-order method has surpassed CFR+ in performance on non-trivial EFGs." However, it seems that in the paper titled "Equivalence Analysis between Counterfactual Regret Minimization and Online Mirror Descent" by Liu et al., the authors also demonstrate instances where first-order OMD methods can outperform CFR+. 2) How strong is the assumption that the strongly convex functions are nice, see page 5, paragraph "Dilated Regularizers"? 3) In the appendix, the last three figures share the same caption. Do these plots refer to linear, quadratic, and uniform averaging?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

During the experiments, ECyclicPDA demonstrated superior performance compared to CFR+ solely in the Battleship game. This stands as the only instance where the results were competitive with PCFR+, which still remains the overall best-performing strategy. However, it is not entirely clear from the overall text why ECyclicPDA outperformed CFR+ specifically in the Battleship game.

Reviewer Y4SP2023-08-11

Thank you for the response

Reviewer u8q22023-08-11

Thank you for the response.

Reviewer Lrca2023-08-15

Thank you for your response. I read all the reviews and the rebuttal provided by the authors. I stick to my original rating of "Accept".

Reviewer udTT2023-08-16

> First, CODER is not directly applicable CODER is not directly applicable to every leaf node, but it is applicable with $d=2$ as I mentioned, that is, separating the variables into only 2 large blocks (two players). The proof in CODER does apply to variable blocks. And I understand your contribution on the L1 norm proof is different form CODER's L2 proof. Just highlighting the relationship of the L2 to L1 generalization, which also appears in other proximal papers. After reading the authors' rebuttal, I decided to keep my rating.

Authorsrebuttal2023-08-17

We would like to thank you for continuing to engage in a discussion about our paper. Yes, CODER is applicable only in the trivial cases as we acknowledged in our response (see "except for the trivial primal-dual separation mentioned in the review" in the response above), but not in the general block coordinate case that we focus on in our paper. This is explained in detail in the above response and in the paper. In particular, it is inapplicable for three of the four block construction strategies (children, postorder, and infosets) that we discuss in Section 4. Note that what we are doing is more general than treating “the leaf nodes” as blocks. The fourth block construction strategy we consider consists of using a single block for each player and coincides with the trivial decomposition you mention, and thus is already covered by our experiments. Note that this latter trivial block construction strategy corresponds to the well-known scheme of *alternation* in a two-player zero-sum game. Alternation is already known to work e.g., in the context of self-play via regret minimization, as well as in e.g., the Chambolle & Pock primal-dual algorithm. Alternation is quite special, in that it leverages the primal-dual structure of a zero-sum game, and we do not think it is meaningfully a form of “block decomposition” in the spirit of what our paper is accomplishing. Note also that existing work does not describe this as a form of block decomposition.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC