Thanks for the reviewer's time and effort to read our response. We are eager to discuss with the reviewer regarding our work. Here, we describe our work further to give the reviewer a better overview of the value factorization methods. Our work is a value factorization approach. Value factorization (or value decomposition) methods learn a factored state-action value function (utility function) for each agent. Each agent acts greedily according to the state-acton function.
For value factorization methods, the **individual-global-max (IGM)** principle is an important principle. It requires that the optimal joint action equal to the collection of individual optimal actions. Many approaches have been proposed. For example, VDN [1] models the joint state-action value function as the sum of each individual state-action value function. QMIX [2] models the joint state-action value function as a monotonic increasing function with respect to each individual state-action value function. QTran [3] can factorize any factorized state-action value function and does not suffer from representation limitations. However, it is sample inefficient. QPLEX [4] and WQMIX [5] use different methods to learn state-action value functions that do not suffer from representation limitations.
DMIX [6] extends the individual-global-max (IGM) principle to the **distributional individual-global-max (DIGM) principle**, which is used for distributional MARL. DIGM principle requires that for distributional state-action value function (stochastic value function), the optimal joint action is equal to the collection of individual optimal actions. However, it suffers from representation limitations too. ResZ [7] uses a residual function-based approach, it satisfies the DIGM principle without representation limitations.
In this work, we believe that for risk-sensitive MARL scenarios, to learn factored risk-sensitive state-action value functions for each agent, a new IGM principle should be developed. The **risk-sensitive individual-global-max (RIGM)** principle requires that for risk-sensitive MARL, the optimal risk-sensitive joint action must equal the collection of individual risk-sensitive optimal actions.
In this work, we use distributional RL algorithms and use risk-metric (e.g., CVaR) to calculate the risk from the return distribution of a state-action pair. It is non-trivial to design an algorithm that satisfies the RIGM principle. As we show in Theorem 1, methods that satisfy the IGM principle **do not guarantee adhere to the RIGM principle**, and in Theorem 2, methods that satisfy the DIGM principle **do not guarantee adhere to the RIGM principle** for some risk-metrics. Further, we have shown that the risk-sensitive MARL algorithm RMIX [8] satisfies the RIGM principle only for a special case, and the risk-sensitive MARL algorithm DRIMA [9] does not satisfy the RIGM principle for the CVaR metric.
We have developed RiskQ, which satisfies the RIGM principle for the VaR and distorted risk measures. In the paper and the response PDF, we have shown that RiskQ performs significantly better than the other algorithms in risk-sensitive MARL scenarios. Through ablation studies, we show that it is not satisfying the RIGM principle will lead to a performance drop. And RiskQ performs better than its representation-limitation-free variants.
**Reference**
[1] Sunehag et al. Value-decomposition networks for cooperative multi-agent learning based on
team reward. AAMAS 2018.
[2] Rashid et al. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. ICML 2018.
[3] Son et al. QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. ICML 2019.
[4] Wang et al. QPlex: Duplex dueling multi-agent q-learning. ICLR, 2021.
[5] Rashid et al. Weighted QMIX: expanding monotonic value function factorisation for deep multi-agent reinforcement learning. NeurIPS, 2020.
[6] Sun et al. DFAC framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. ICML, 2021.
[7] Shen et al. ResQ: A residual q function-based approach for multi-agent reinforcement learning value factorization. NeurIPS 2022.
[8] Qiu et al. RMIX: learning risk-sensitive policies for cooperative reinforcement learning agents. NeurIPS, 2021.
[9] Son et al. Disentangling sources of risk for distributional multi-agent reinforcement learning. In ICML, 2022.