Aligning Individual and Collective Objectives in Multi-Agent Cooperation

Summary

The paper lies in the intersection of multi-agent and game theory. In particular, dealing with "mixed-motive cooperative games". This is, games in which the maximization of individual rewards hampers the maximization of collective rewards. This is, when agents seek to maximize their individual reward/utility, and disregard the collective utility they are worst off as a collective. These type of problems are usually called "social dilemmas" in game theory. The objective is to achieve the 'social optimum', this is, the maximum collective reward a group of agents can obtain, beyond the individual reward. The methodology proposed in the paper is to align the individual rewards towards the collective maximization rewards via "gradient shaping". This is, move the gradients of the individual reward maximization policy towards the collective reward. In order to shift the individual rewards, the algorithm includes an "alignment" parameter that aligns two optimization problems (individual maximization vs collective optimization) that otherwise, will pull against each other. Since the collective optimization point is an equilibrium point, the algorithm shows convergence.

Strengths

1.Simplicity: The algorithm is simple and intuitive. It can be easily implemented in the current major multi-agent algorithms. 2. Testing in well selected benchmark environments: The environments selected to test the algorithm are very standard in the field. Results show superiority against the selected benchmarks. 3. Ablation studies. The paper presents ablation studies for the relevant parameters. In particular, the "lambda" term controlling the alignment. 4. Presentation: the narrative is well written and makes the reading very straightforward. The paper provides intuition as well as rigor on the methodology proposed.

Weaknesses

See below.

Questions

I don't think this is fair or true. The mentioned papers do provide substantial theoretical analysis. This is not the way to place your paper. "On the other hand, several studies focus on auto-matically modifying rewards by learning additional weights to adjust the original objectives [Gemp et al., 2020, Kwon et al., 2023]. However, these studies often suffer from a lack of interpretability and insufficient theoretical analysis regarding the alignment of individual and collective objectives."

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors have addressed the limitations of the paper on the Conclusions section.

Summary

The paper proposes Altruistic Gradient Adjustment (AgA) which adjusts the gradients of individual and collective losses to align individual and collective objectives. Besides, the authors prove that AgA effectively attracts gradients to stable fixed points of the collective objective while less sacrificing individual interests. Experiments and example illustrations demonstrate the effectiveness of AgA.

Strengths

1. The paper is well-written and well-organized. 2. This topic is important with less attention in the MARL community. 3. The experiments are performed on several environments and some example illustrations are given to help understand the proposed technique. 4. The most important contribution of this paper is Corollary 4.3. This algorithm tries to seek stable fixed points for the collective objective while previous works aim to seek stable fixed points for individual objectives. This will possibly benefit the collective welfare as well as the individual interest at the same time, which would be useful as demonstrated in the experiments.

Weaknesses

1. AgA introduces an additional adjustment term, which needs to be tuned case by case. 2. AgA introduces additional computation complexity to compute Hessian-vector products. 3. It is not clear whether AgA also has the ability to always keep all individual objectives best. For example, in Figure 1(c), Simul-Co could achieve a better reward for Player 2 than AgA while Simul-Co also achieves a higher collective reward than AgA. 4. There is only one map of SMAC to be tested.

Questions

1. Could the authors compare the running time of AgA with other baselines? 2. Could the authors provide more discussion about AgA and Simul-Co? For example, if Simul-Co could achieve a higher collective reward than AgA like in Figure 1, one can redistribute the collective reward for each agent so that the redistributed individual rewards for each agent can be better than AgA. 3. In Algorithm 1, are the parameters w=[w_1,…,w_n] configured manually or determined by the task itself?

Rating

5

Confidence

2

Soundness

2

Presentation

3

Contribution

3

Limitations

NA.

Summary

This paper introduces a novel optimization method called AGA that employs gradient adjustments to progressively align individual and collective objectives. They prove that this method attracts gradients to stable fixed points of the collective objective while considering individual interests. Their method is empirically validated on sequential social dilemmas games, Cleanup and Harvest, and a high dimensional environment, StarCraft II.

Strengths

- Preliminaries are clear - Proofs are given graphs to explain the intuition - Good comparison to baselines - Varied experimental environments (grid game + modified SMAC) - Strong experimental results

Weaknesses

- No clear weaknesses

Questions

- Figure 1. It’s not clear how to interpret the reward contours graphs. Can you clarify what the x-axis and y-axis? How are the contours formed? They appear to be integers corresponding to the Player’s actions but where is the action space defined? - What is the significance of AGA moving along the summit? - In Proposition 4.1, what is the gradient without the subscript referring to? - L292, how is the collective loss equation determined?

Rating

8

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes.

Summary

The paper investigates the topic of cooperation in a mixed-motive multi-agent setting. They first propose the formulation of a mixed motive game as a differentiable game. Leveraging the structure of the latter, they propose a gradient-based optimization algorithm, AgA. The paper both discusses the theoretical guarantees of AgA, and its effectiveness in an empirical setting. Finally, the authors introduce a modification of the MMM2 map in the StarCraft II game, to further test the empirical performances of AgA.

Strengths

- The paper is theoretically sound. The AgA algorithm is simple, but at the same time well justified. - The experiment are broad and well designed. The Selfish-MMM2 environment addresses some of the limitations of the other ones, and I think is a principled way of showing the performance of the algorithm in larger and more complex environments

Weaknesses

- There are some typos in the paper, and I feel the presentation can be improved. For example, in the Related Work section in page 2, there are some sentences which are incomplete or non-sensical. I am specifically referring to ' While PED-DQN enables agents 79 to incrementally adjust their reward functions for enhanced collaborative action through inter-agent 80 evaluative signal exchanges [Hostallero et al., 2020], Gifting directly reward other agents as part of 81 action space [Lupu and Precup, 2020].' - I think the author should discuss more in depth the complexity of this method. How does it scale with respect to the other algorithms proposed? There is no experiment that highlights this in the paper - There is no discussion of how AgA satisfied individual incentives

Questions

- How much slower is AgA compared to the other algorithms? Do you have any empirical experiments which can shows this in practice? - Do you have any results which show the individual metrics achieved by the agents when using AgA vs other algorithms? It could be valuable to see some plot which shows both the cooperative and individual rewards, and how these vary depending on the value the hyperparameter $\lambda$ is set to - Did you run the experiments for Figure 3 for more seeds? Do the results still hold?

Rating

5

Confidence

2

Soundness

3

Presentation

1

Contribution

2

Limitations

No limitations

Thank you to the authors for their response.

Thank you for replying to the rebuttal and including my suggestion into your work. My rating stands.good luck with your paper!

Main concerns addressed

Dear Authors, Thanks for the answer and for the additional experiments provided. I suggest the authors to both include the discussion and results provided here on the individual incentive and the computational complexity of AgA in the final version of the manuscript. Since the main concerns I raised were addressed, I'll increase my score accordingly.