Rebuttal during the discussion
We appreciate the reviewer’s feedback!
**1. The non-stationarity and the scalability issue.**
We provide the state and joint actions as the input to train the world model (avoid the non-stationarity) and utilize the Transformer model (GPT2-like dynamics and Bert-like reward models) to provide enough representational capacity, where their attention mechanism performs feature selection and aggregation (for the scalability).
To further elucidate our approach, we list the following paradigms:
*(1) Decentralized training with decentralized execution*: While this paradigm excels in scalability during execution, it often leads to non-stationarity as agents treat others as part of the environment.
*(2) Centralized training with decentralized execution*: It maintains execution scalability while introducing a centralized mechanism (e.g., a mixing network to aggregate the individual Q-values) to mitigate the non-stationarity problem.
*(3) Communication-enabled training and execution*: Each agent learns and executes its policy based on their observations and (weighted) messages from others (e.g., actions, observations). This paradigm creates a trade-off between interaction complexity and representational capacity, a great compromise for non-stationarity and scalability issues.
*(4) Centralized training with centralized execution*: While this paradigm eliminates the non-stationarity issue, it suffers from limited scalability during execution.
Our world model serves as the environment, which leverages the Transformer model’s attention mechanism to capture interdependencies among agents and timesteps, similar to communication-enabled paradigms (e.g., MAT [1]). The concurrent work [2] also proposes a Perceiver Transformer as a communication module, which utilizes the querying and self-attention mechanisms to perform agent-wise aggregation, transforming the joint representation sequence (joint observations and actions) into lower-dimensional features. The difference is that our world model integrates these processes into the large transformer model.
**2&3. MAMBA and MARIE, and the low data regime.**
We acknowledge that MAMBA [3] and MARIE [2] are indeed Dyna-style algorithms that learn policy in the imagination, and their induced policies are decentralized. We would like to clarify our stance:
* MAMBA and MARIE assume that the reward function is presented in the replay buffer. They train the reward predictor using log-likelihood and cross-entropy loss, respectively. **Conversely**, we suppose the reward is unknown in the offline dataset. We construct the reward function using a bidirectional Transformer, which maximizes the likelihood of trajectories in hindsight relabeled expert demonstrations.
* MAMBA and MARIE are online model-based RL methods. They explore the environment using a policy, store the collected data in a replay buffer, update the world model with samples from this buffer, and use rollouts from the world model to refine the policy. They can benefit from interacting with the real environment, allowing them to update and rectify their world models continuously - e.g., if agents imagine trajectories in out-of-distribution regions that lead to suboptimal updates, they can correct this during exploration by gathering more data in these regions. Consequently, they conducted experiments in low-data regimes like Atari-100K to show their data efficiency.
**In contrast**, our dynamics model is trained in an offline manner via a static, offline dataset. Unlike online methods, no exploration is available for the model to rectify itself, leading to the distributional shift issue - where the deployment scenario may differ significantly from the historical dataset regarding state and action space. To mitigate this, we employ a behavior regularization term to encourage exploitation, specifically by discouraging agents from executing out-of-distribution behaviors during inference.
**4. The agent death masks.**
Thank you for your insightful suggestion. We would like to use a binary gate function for each agent to replace the handcrafted death masks and learn it via binary cross-entropy loss in a supervised manner. We acknowledge the potential for further generalization.
However, we respectfully disagree with the notion that the current approach oversimplifies the learning process. As shown in Fig.4, the learned world model captures the dynamics and learns a meaningful reward function even in this complex scenario. This demonstrates that the model correctly understands the transitions within the game, validating its ability to grasp the underlying mechanics of SMAC.
[1] Wen, Muning, et al. Multi-agent reinforcement learning is a sequence modeling problem.
[2] Zhana, Yang, et al. Decentralized Transformers with centralized Aggregation are Sample-Efficient Multi-Agent World Models.
[3] Micheli. Vincent, Eloi Alonso, and Francois Fleuret, Transformers are Sample-Efficient World Models.