AlphaDou: High-Performance End-to-End Doudizhu AI Integrating Bidding

1. To what extent does considering bidding affect the complexity of the game of DouDizhu? Bidding provides additional information that influences the model's decision-making. The bidding process prompts the model to further assess the card types held by the Landlord or teammates. This includes, but is not limited to: (a) If the Landlord wins with a low bid, there is a lower probability of bombs in their hand, indicating weaker card strength; conversely, a higher bid suggests stronger cards. (b) After becoming the Landlord, if someone else also bid, the model should be more cautious of the risk of bombs. (c) As a Peasant, if a teammate placed a bid, the model should consider greater coordination with them. Bidding also affects the card distribution differently than in games without bidding. Players with stronger hands are more likely to become the Landlord, and games with a bidding stage tend to have a higher Landlord win rate, whereas Peasants win more often in games without bidding. This also impacts the model’s decision-making. In certain scenarios, AlphaDou may adopt an aggressive playstyle assuming the Peasant’s hand is weaker, similar to professional human players. However, models like Douzero or Douzero Resnet, being more concerned about strong Peasant hands, may miss opportunities as a result. 2. You should directly competing AlphaDou with RARSMSDou since RARSMSDou is the strongest publicly available Doudizhu model. Different models employ distinct strategies. A higher win rate against the same benchmark by one model does not necessarily imply that it can defeat another model. Before AlphaDou, the strongest AI was RARSMSDou; however, since it is not open-sourced, direct comparison with it is challenging. We manually implemented an AI similar to what was described in the paper, but we were unable to reproduce its results. We plan to add results of comparisons with open-source AIs, such as perfectDou or AP-MCTS (Zhang, Yunsheng, et al. "Combining Tree Search and Action Prediction for State-of-the-Art Performance in DouDiZhu." IJCAI. 2021). 3. In line 152, "i = 4" is an error, and it does not match the illustration in Figure 1(a). I will correct this mistake in the upcoming version of the submission. 4. In line 389, it seems that the reported results in the column for the 1st position are worse than those in the column for the 2nd position. Is this correct? Yes, this is correct, as this table should be viewed vertically. It uses a controlled variable method, comparing the replacement of one model or two models against the control. Changing the bidding strategy does not necessarily suppress the opponents' win rates. In fact, the addition of the bid model increases the win rates of the remaining two opponents, as the bid model is more willing to take on the role of the Landlord (reflected in a higher Landlord Percentage, or LP) compared to the original strategy. A higher LP tends to lower the Landlord’s win rate, as the Landlord’s loss results in a win for both Peasants. However, through vertical comparison, it’s evident that the bid model consistently achieves a higher score than it did before the strategy change. 5. The input of the model includes the number of bombs played in the game, and I think this point involves the use of expert knowledge. An algorithm is end-to-end if only the direct observation is used as the input with post-processing. I don’t believe that the "number of bombs played in the game" involves expert knowledge, but it does help the model understand the game. The same applies to "remaining cards" and "cards played by each position." For the former, it requires subtracting one’s own cards and the cards others have played from the full deck, while for the latter, it involves accumulating the cards played by each position. I also directly provided these results to the model.

• Regarding the First Question： Estimating the win rate is not a binary classification problem, so using BCE loss isn’t ideal for reinforcement learning tasks. Most models, like AlphaGo, DouZero use MSE, while supervised tasks like Stockfish use BCE. Since you’ve raised concerns about MSE, I’ll explain briefly why it’s better suited for these tasks.In binary classification, when a model outputs a value like 0.6 or 0.99, it reflects the model’s confidence. However, in reinforcement learning, the win rate is a probability, not a confidence level. If the win rate is 0.6, it means there's a 40% chance of losing, and the model should output 0.6, not 1. Similarly, if the win rate is 0.99, the model should output 0.99, not 1. Why BCE Loss is not ideal for reinforcement learning: On the other hand, MSE works better for win rate estimation, as it gives smaller gradients in highly probable outcomes (like when a win or loss is nearly certain) and allows the model to adjust slightly for rare events (like comebacks).Using 1 and -1 as rewards provides better symmetry, which is natural for games with different roles (like in Go, where the output magnitude is similar for both black and white players). (a) BCE loss works well in classification tasks because it gives large gradients when there’s a misclassification (like when the label is 0 but the model outputs 1). (b) But in reinforcement learning, if the model's output is close to the label (e.g., 0.99 for a 99% win rate), the model shouldn't overreact or try to correct it too forcefully, especially in rare events (like a comeback). • Regarding the Second Question: For the method of "selecting moves based on win rate after expectation pruning" and the "bombs check," we conducted an ablation study, which is included in the appendix B. This is mentioned in line 339 of the main text and has been proven to be effective. From a theoretical perspective, we combine win rate and expected value, clearly choosing win rate as the basis for decision-making. In zero-sum card games, previous research has rarely (in fact, I haven't seen any) used win rate as a standard for selecting moves. However, the model's estimation of win rate is often more accurate; the labels for win rate are 1 and -1, while labels for scores can be 3, -3, 6, -6, and so on. This means the uncertainty in estimating win rate is smaller than that of estimating scores. To give a simple example: a move with a win rate of 0.99 corresponds to an expected value of 2.98 points, while a move with a win rate of 0.98 corresponds to an expected value of 2.99 points. In a zero-sum game, choosing a move with an expected value of 2.98 instead of 2.99 results in a loss of 0.01 points. Considering the uncertainty, it is clear that the move with a win rate of 0.99 should be chosen because the actual loss is 0.01 + δ (uncertainty). This is the core idea behind the improvement and a key reason for AlphaDou's success. We require the model to use win rate-based decision-making in the absence of "Boom Check" passing, accounting for uncertainty. Incorporating this idea into the data generation process during training helps the model generate higher-quality data for gradient descent. • Explanation of the Significance of Experimental Results The experimental results show that AlphaDou's performance improves significantly; DouZero Resnet achieves a winning percentage of 0.57 in 2021 and is ranked number one in the Botzone for three consecutive years, while RARSMSDou improves this to 0.582 and AlphaDou to 0.596. In the bid test set, the winning percentage reaches 0.620, about 0.4 higher than RARSMSDou's. PerfectDou’s win rate against DouZero is 0.54. • Explanation of the Ablation Study Question: In the Douzero ResNet project, it’s been shown that a convolutional network outperforms Douzero’s LSTM network. Our architecture is slightly different, but it’s based on Douzero ResNet, so the difference is not significant. Providing an exact analysis of the model architecture's impact would be difficult and costly. Regarding the Bid phase, introducing it definitely improves performance by providing additional information. Our experiments show that while the model performs better with bidding than with random bidding, it still achieves state-of-the-art results. Random bidding does cause some performance degradation due to bias between the test and training sets. However, whether training without the bidding phase would improve performance on a non-bidding test set remains unclear. That said, this isn’t the focus of our research. The main goal is to develop a state-of-the-art model that incorporates the bidding phase, as it aligns with the actual rules of DouDiZhu. • We agree with your understanding of "solved" and will modify. We will also modify the formula section.

• Are there any architectural innovations in deep learning? There are innovations in the theoretical perspective. By combining win rate and expected value, we are explicitly choosing win rate as the basis for decision making. In zero-sum card games, very little previous research (in fact, I haven't seen any) has used win rate as a criterion for choosing moves. However, models tend to be more accurate in estimating win rates; win rates are labeled 1 and -1, while scores are labeled 3, -3, 6, -6, and so on, meaning that win rate estimates have less uncertainty than score estimates. In a zero-sum game, choosing a move with an expected value of 2.98 instead of 2.99 means a loss of 0.01. Given the uncertainty, it is clear that the move with a higher winning percentage should be chosen, since the loss is actually equal to 0.01 + δ (uncertainty). This is the main idea behind the improvement and the main reason for the success of AlphaDou. We ask the model to take uncertainty into account even when there is no “factor that doubles the reward” and use decisions based purely on winning percentage. Incorporating this idea into the data generation process for training allows the model to generate higher quality data for gradient descent. This research provides a new approach for future similar games (or extended tasks such as stock prediction), suggesting a focus on win rate estimation. • Can it be compared to the Transformer base apporoach[1]. The paper mentioned in the question uses a Transformer to fit the public dataset of Stockfish 16. First, it is important to clarify that Stockfish trains a 4-layer fully connected neural network using its public training set with supervised learning. Stockfish incorporates the idea of reinforcement learning, but at its core, it is supervised learning. The 4-layer fully connected network can perform quick computations on a CPU (by adjusting all neural network parameters to integers) to support the use of the MinMax algorithm. Any network architecture more advanced than a fully connected network could fit this public dataset better. However, these advanced networks, such as residual convolutional networks or Transformer networks, cannot support quick search on a CPU, and the resulting AI strength would not outperform Stockfish. LeelaChessZero (with Transformer architecture) using Monte Carlo Tree Search also failed to beat Stockfish's 4-layer fully connected network. In the field of reinforcement learning, Transformer network architectures are used by LeelaChessZero (https://github.com/LeelaChessZero/lc0), Kanachan (https://github.com/Cryolite/kanachan), and others. There are also AIs using convolutional neural network architectures, such as Mortal (https://github.com/Equim-chan/Mortal) and KataGo (https://github.com/lightvector/KataGo). Unlike supervised learning tasks, in reinforcement learning game research, there is currently no strong conclusion suggesting that convolutional neural networks are inferior to Transformers. The reason this study does not use Transformers is that their training cost is too high. For the same number of parameters, the training speed of a Transformer is only one-third that of a convolutional network. After training 100 million frames of actions, the win rate of Transformers is only 40% compared to convolutional networks. This could be due to the slow-start phenomenon of the Transformer network structure, and it does not rule out that the final result of the Transformer structure might eventually be worse than that of the convolutional network. However, this directly led to the decision in the paper to abandon the more expensive Transformer architecture, which performs better in supervised learning tasks, and instead choose the cheaper convolutional network architecture. [1] Amortized Planning with Large-Scale Transformers: A Case Study on Chess, NeurIPS2024.