Maximum Entropy Heterogeneous-Agent Reinforcement Learning

Response to Reviewer 9JkX

Thank you sincerely for appreciating our work. Your acknowledgment and encouragement have motivated us to work even harder to achieve a thorough refinement of our current work. Q1: The IGO assumption is not explained. > A1: Thank you for pointing it out. We have included the definition of IGO and the explanation of its limitations in Appendix A.2. Q2: HASAC is proved to converge to QRE, but not the optimal NE. Is it possible to have guarantees to reach the optimal NE? > A2: We agree that in the cooperative setting, there exists an optimal joint policy (i.e., Pareto optimality) to which our proposed methods cannot guarantee convergence. It is because although our setting is cooperative, the agents' policies are decentralized. Consequently, finding a Pareto-optimal policy may prove infeasible, as it could involve joint exploration and joint optimization—an impractical task for agents unaware of other agents' policies. Therefore, our method cannot guarantee convergece to Pareto-optimal policies, and obtaining such policies requires directly learning the joint policy.

Response to Reviewer iakN

Thank you sincerely for appreciating our work. As for the weaknesses you mentioned, please allow us to address them in the most respectful manner. Q1: The effect of the sequential updates is unclear. Could the authors provide results of MASAC on the benchmarks? > A1: Thank you for the valuable suggestion. The advantages of sequential updates compared to simultaneous updates in MaxEnt MARL are similar to those mentioned in standard MARL in [1]. MASAC cannot provide a rigorous guarantee of monotonic improvement during training. To illustrate this, let's consider two agents $(1, 2)$ that take actions $a^1, a^2 \in \\{-2, -1, 1, 2\\}$. They receive the reward $r(a^1,a^2) = a^1 \cdot a^2$. Every agent $i$ takes its action $a^i$ from its policy $\pi^i$. We initialize $\pi^1 = (0, 1, 0, 0)$ and $\pi^2 = (0, 0, 1, 0)$. Under this setting, the initial reward is $r(-1, 1) = -1 \cdot 1 = -1$. The maximizing action for agent $1$ is $a_\text{max}^1 = 2$, and for agent $2$ it is $a_{\max}^{2}=-2$. Consequently, MASAC will simultaneously update $\pi^1$ and $\pi^2$, where $\pi^1$ will assign a highest probability to action 2, and $\pi^2$ will assign a highest probability to action $-2$ (the specific probabilities are influenced by $\alpha$), resulting in a worse outcome. In contrast, sequential updates could avoid such situation, ensuring monotonic improvement. > Following your suggestion, we have conducted an additional ablation study to investigate the effect of sequential updates in the updated submission version. We compare performance of the original HASAC and HASAC without sequential updates (or MASAC) on Bi-DexHands tasks. As shown in Appendix I, experimental results show that HASAC consistently achieves better performance than MASAC. Q2: Could the authors implement an adaptive entropy temperature tuning method for better exploration? > A2: We respectfully disagree with the statement that our automatically tuning temperature method is not effective. The automatically adjusting temperature mechanism we implement for HASAC draws on the method proposed in [2], as stated in Appendix H.1. It is similar to the Equation 13 in ADER [3]. However, regarding the method mentioned in ADER for learning individual target entropies, since we only have a centralized Q function, unlike ADER, which has both a centralized value function and local value functions, we cannot directly apply their approach. Nevertheless, we believe that their idea is indeed enlightening and aids in better temperature adjustment. We will explore how to adjust temperature more effectively in future work. Besides this, our method is already fairly effective in practice. As shown in Appendix H.3, HASAC performs well with automatically adjusting $\alpha$ in most tasks; only a few tasks require additional manual tuning of $\alpha$. As shown in the Appendix C2 of ADER, ADER needs to set different values of $k_{\text{ratio}}, \xi, \alpha^i_\text{init}$ for almost every task on SMAC. In contrast, HASAC achieves good performance across various tasks with auto-tuned $\alpha$, except for setting $\alpha$ to 0.002 on the 'corridor' task. > [1] Kuba et al., Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning. [2] Haarnoja et al., Soft actor-critic algorithms and applications. [3] Kim et al., An Adaptive Entropy-Regularization Framework for Multi-Agent Reinforcement Learning.

Response to Reviewer ZAWK

Thank you so much for your time and effort in reading our code and paper. We really appreciate your deep engagement that will surely turn our paper into a better shape. Q1: The novelty and contribution are limited. > A1: We respectfully disagree with the statement that the novelty and contribution of our work are limited. Firstly, our novelties are three-fold: We first use PGM to derive MaxEnt objective of MARL and provide a representation of QRE in the context of MaxEnt MARL. Then we develop the theoretically-justified HASAC algorithm to maximize this objective. Finally, we introduced a versatile template, MEHAML, for rigorous algorithmic design. > Secondly, our application of SAC algorithm to MARL is non-trivial. We not only prove rigorous theoretical properties but also demonstrate strong experimental results. > Lastly, ****Reviewer 9JkX**** also agrees with us on the novelty and theoretical soundness, and believes that our work is not a simple combination of MaxEnt RL and MARL. Q2: Could the authors provide more details about the sensitivity of HASAC to $\alpha$? > A2: Thank you for raising the question. Actually, HASAC is indeed sensitive to $\alpha$, as shown in Section 5.2. Since $\alpha$ affects the stochasticity of each policy, it leads to the convergence to different QREs. Figure 5 and 4(c) intuitively illustrate the impact of different $\alpha$ on the final QREs and performance. Therefore, setting different $\alpha$ for various tasks is necessary to enable policies to discover and converge to higher reward equilibria. To mitigate such sensitivity to $\alpha$, we draw on the method of auto-tuned $\alpha$ mentioned in [1] and have implemented it for HASAC. As shown in our Appendix H, auto-tuned $\alpha$ does achieve this goal. In most tasks, HASAC performs well with automatically adjusting $\alpha$; only a few tasks require additional manual tuning of $\alpha$. This significantly alleviates sensitivity of HASAC to $\alpha$. Q3: How about the performance of HASAC in scenarios where sample efficiency is important? Can the authors provide experimental results in such settings? > A3: We regret that we may not currently have the capability to test our algorithm on real robots. However, Bi-DexHands [2] and MAMuJoCo [3] are both simulated environments for robotic control where sample efficiency is crucial. As shown in our experimental results, on-policy algorithms exhibit low sample efficiency in these environments, especially in Bi-DexHands, where on-policy algorithms fail to learn any meaningful joint policy within 1e7 steps. In contrast, HASAC shows high sample efficiency in these tasks, demonstrating sample-efficient even compared to other off-policy algorithms. Therefore, we believe that our current experiments could showcase the sampling efficiency of HASAC. Q4: The validity of the experimental results is questionable. The authors only present a selection of results in the main paper. > A4: The reason why we only present a selection of results in the main paper is the 9-page limit and the abundance of experimental results. Therefore, we have to selectively include a portion of our extensive experiments in the main paper. Our selection is primarily based on the difficulty of tasks, choosing one or two difficult tasks from each benchmark that other baselines cannot solve effectively, while our algorithm still perform well. This allows for a better demonstration of the advantages of our algorithm. Despite this, we recommend you referring to Appendix H for a comprehensive view of all experimental results. In 31 out of the 35 tasks across all benchmarks, our method achieve the best performance. Furthermore, our method demonstrate better stability, smaller fluctuation and variance in most tasks. Q5: Could the authors provide more experimental results from a larger number of seeds? > A5: Thank you for the precious suggestion. We initially ran each algorithm with at least four seeds. Due to the large number of experiments and algorithms, and the fact that each method's performance difference across the four random seeds were not substantial, we did not run additional random seeds. > Following your suggestion, we rerun each algorithm with eight random seeds on three Bi-DexHands tasks, namely ShadowHandCatchAbreast, ShadowHandTwoCatchUnderarm, and ShadowHandLiftUnderarm. We have updated the experimental results in the updated submission version. From the results of these three tasks, the difference between the results from eight seeds and four seeds is minimal, indicating that our original experiments are sufficiently comprehensive to statistically demonstrate the advantages of our algorithm. [1] Haarnoja et al., Soft actor-critic algorithms and applications. [2] Chen et al., Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning. [3] Peng et al., FACMAC: Factored Multi-Agent Centralised Policy Gradients.

Response to Reviewer YGsv

Thank you very much for your careful review. We deeply appreciate your acknowledgment and the helpful comments you have given. Please allow us to address them as follows. Q1: IGO assumption should have been defined and explained more clearly. > A1: Thanks for pointing this out. Due to the 9-page limit, we have included the definition of IGO and the explanation of its limitations in Appendix A.2. Q2: Lack of elaboration of the automatically-adjusted $\alpha$ schedule. > A2: Thank you for the helpful suggestion. We implement the automatically-adjusted $\alpha$ for HASAC following the method proposed in [2]. And we have included the optimization objective for $\alpha$ in Appendix H.1. Q3&Q4: The design of the drift functional and neighborhood operators is key to the MEHAML based methods. Could the authors provide some discussion of the design of these terms? What is the connection between MEHAML and PPO methods? > A3&A4: Thank you for raising the question. We would like to answer it as follows. First, HASAC can be seen as the most natural instance of MEHAML template with a drift functional of $0$ and neighborhood of $\Pi^{i_m}$. Nevertheless, HASAC has demonstrated effective performance across a majority of tasks. > Furthermore, when designing more sophisticated drift functionals and neighborhoods, the most crucial factor is undoubtedly to design operators that satisfy the definition. As discussed in [3], drift functional can be designed as KL-divergence, squared L2 distance, etc., while neighborhood can be designed as KL-ball, like in TRPO. What's more, drift functional can be obtained by meta-learning, as shown in [4]. The objective of our theory is primarily theoretical, aiming to provide theoretical guarantees for all these potential methods in MaxEnt MARL. The specifics of which drift functional and neighborhood lead to better performance in a certain task, as well as how to design practical algorithms with different operators, remain for future research. > In conclusion, the broad scope of drift functionals and neighborhoods encompasses abstractions for TRPO, PPO, and many other methods. Hence, the MEHAML template offers theoretical guarantees for applying policy updates with additional constraints, including PPO and TRPO, to MaxEnt MARL. Q5: Can the authors explain what QRE means in practice compared to NE? In what situations is QRE more favorable than NE? > A5: We apologize if the explanation of QRE is not clear enough. In fact, QRE is a generalization of NE. In NE, each player is perfectly rational, deterministically selecting the strategy with highest payoff. In QRE, however, each player is bounded rational, resulting in their strategies being probabilistic distributions related to payoffs. Put simply, QRE represents the equilibria of stochastic behavior, while NE represents the equilibria of deterministic behavior. > In our case, due to the additional entropy regularization term in the MaxEnt objective, each agent's policy at equilibrium becomes stochastic (Boltzmann distribution) rather than a deterministic one, as demonstrated by Theorem 1. Therefore, it corresponds to QRE. As $\alpha \rightarrow 0$, the MaxEnt objective is reduced to the standard objective, the Boltzmann distribution is reduced to a one-hot distribution, and QRE is reduced to NE. Hence, it is not accurate to claim any inherent advantages of QRE over NE; the observed advantage in experimental results arises from the effectiveness of stochastic policies, which inherently facilitate more efficient exploration, leading to the discovery and convergence of equilibria with higher rewards. Q6: What is the $\omega$-limit set? > A6: The $\omega$-limit set is a concept used in the study of dynamic systems, which describes the set of states that a system tends to reach as time progresses indefinitely. In our case, the sequence of joint policies would reach fixed points of soft policy iteration, and these fixed points are QRE policies. Therefore, we refer to these fixed points as the $\omega$-limit set of joint policies, and their $\omega$-limits set consists of QRE. Q7: Some minors. > Thank you for the valuable suggestion. We have rewritten '2 hundred' as '200' and added a table of acronyms in Appendix A.1. [1] Wang et al., More centralized training, still decentralized execution: Multi-agent conditional policy factorization. [2] Haarnoja et al., Soft actor-critic algorithms and applications. [3] Kuba et al., Mirror learning: A unifying framework of policy optimisation. [4] Kuba et al., Discovered Policy Optimisation.