Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning

- **Limitation in highly dynamic environments with rapidly changing behaviors** We appreciate the reviewer's insightful comment regarding the challenges of modeling and inference with policy changes over time. As the EFT agent should continuously adapt to evolving strategies and behaviors, the complexity of modeling and inferring these changes increases significantly. This issue is further compounded as the number of agents grows, potentially exacerbating the intractability of the problem. The dynamic nature of policy introduces additional layers of complexity, making it increasingly difficult to predict and manage the interactions among agents effectively. This is an ultimate goal for the research community, and we consider it as future work. Although we did not address rapidly changing behavior in this study, our work demonstrates promising results, such as successful interactions with changes in surrounding characters across different episodes. In accordance with NeurIPS policy, we would like to clarify that we have already discussed this limitation in our manuscript. --- Once again, we deeply appreciate the insightful comments and suggestions. We hope our clarification and additional empirical studies could address the concerns raised by the reviewer. Should there be any leftover questions, please let us know and we will make every effort to address them during the subsequent discussion period.

- **Experimental results as the number of EFT agents increases (W1, L1)** Although we have discussed this potential weakness in our limitation section, we additionally explored how our EFTM behaves as the number of agents increases. The additional empirical results are as follows. Here's the table with the bold formatting removed: |Ratio of EFT agent|Baseline (single EFT)|10%|20%|30%|40%|50%|60%| |-|-|-|-|-|-|-|-| |Performance|2899 $\pm$ 217|2910 $\pm$ 193|2818 $\pm$ 316|2376 $\pm$ 991|2041 $\pm$ 752|1650 $\pm$ 548|1728 $\pm$ 683| Empirical result indicates that EFTM performance remains robust when the interaction between EFT agents is infrequent, such as around $20\%$, and that performance gradually declines thereafter. As per our expectations, potential instability happens when a larger proportion of agents in the system are equipped with EFT simultaneously. This is similar to the ongoing debates in the theory of mind (ToM) research, where the complexity and depth of understanding others' mental states—from zero- to higher-order ToM—are crucial points of discussion. Determining the optimal level of complexity for specific scenarios is an interesting direction for future research and could offer valuable insights into EFTM. --- - **Standard deviation for main results and additional experiments on SMAC (W3, W7, Q2, L3)** We apologize for the inconvenience. We wanted to report it in the main body, but due to the page limit, we included it in the appendix of the original manuscript. Our appendix includes Tables with the standard deviation as follows. |Character diversity|n=1|n=2|n=3|n=4|n=5| |-|-|-|-|-|-| |Proposed|**2899** $\pm$ 217|**3047** $\pm$ 162|**2976** $\pm$ 196|**2948** $\pm$ 91|**3051** $\pm$ 109| |FCE-EFT|**2899** $\pm$ 217|2784 $\pm$ 161|2646 $\pm$ 196|2566 $\pm$ 103|2629 $\pm$ 125| |MADDPG|2763 $\pm$ 126|**3006** $\pm$ 103|2800 $\pm$ 106|**2933** $\pm$ 98|2856 $\pm$ 121| |MAPPO|2753 $\pm$ 206|2862 $\pm$ 201|2597 $\pm$ 144|2529 $\pm$ 131|2763 $\pm$ 190| |QMIX|2199 $\pm$ 56|2310 $\pm$ 39|2288 $\pm$ 118|2118 $\pm$ 82|1861 $\pm$ 132| |Dreamer|**2911** $\pm$ 312|2813 $\pm$ 283|2733 $\pm$ 351|2631 $\pm$ 521|2701 $\pm$ 433| |MBPO|2089 $\pm$ 804|1964 $\pm$ 753|1523 $\pm$ 948|1893 $\pm$ 792|1633 $\pm$ 821| |Algorithm|MAPPO|MADDPG|QMIX|Proposed| |-|-|-|-|-| |Spread|-149.29 $\pm$ 0.94|-157.10 $\pm$ 2.30|-154.70 $\pm$ 4.90|**-149.12** $\pm$ 1.38| |Adversary|9.61 $\pm$ 0.07|7.80 $\pm$ 1.43|8.11 $\pm$ 0.37|**10.01** $\pm$ 0.33| |Tag|13.78 $\pm$ 4.40|6.65 $\pm$ 3.90|**15.00** $\pm$ 2.73|14.57 $\pm$ 2.95| These tables show that **the standard deviation of EFTM is similar to that of other methods**. The model-based solution has the highest variance due to the uncertainty of other agents. Overall, EFTM achieves the best performance with a mid-level variance compared to all other baselines. Additionally, we have run additional experiments on SMAC [1], which is widely used for evaluating the MARL algorithm, to address the reviewer's concern. We report the performance ($4$ seeds) with MARL baselines, as follows. |SMAC Task|EFTM|MAPPO|MADDPG|QMIX| |-|-|-|-|-| | 2s3z|98.8 $\pm$ 2.3| **100** $\pm$ 1.5|90.3 $\pm$ 5.3|95.3 $\pm$ 2.5| | 3s5z vs 3s6z |**84.3** $\pm$ 9.1|63.3 $\pm$ 19.2|18.9 $\pm$ 4.8|82.8 $\pm$ 5.3| | 6h vs 8z|**93.8** $\pm$ 6.7|85.9 $\pm$ 30.9|68.0 $\pm$ 34.7|9.4 $\pm$ 2.0| | Total|**276.9**|249.2|177.2|187.5| This result also demonstrates that EFTM still has surpassing or matching performance with previous solutions. It means that EFTM is capable of generalizing to solve widely-used MARL tasks, achieving the best total scores. Notably, we set a simple setup for the SMAC and MPE environments that is, we follow a vanilla setup with a single character diversity $n=1$. **While fully leveraging the advantages of EFTM in these environments can be challenging, EFTM is nonetheless capable of delivering competitive performance in such settings.** --- Once again, we deeply appreciate the insightful comments and suggestions. We hope our clarification and additional empirical studies could address the concerns raised by the reviewer. Should there be any leftover questions, please let us know and we will make every effort to address them during the subsequent discussion period.

Thank you for your active response! As discussed earlier, we set character diversity as n=1 on SMAC and MPE. It means that the EFT agent does not need to infer the character because they have the same; in addition, the EFT agent only predicts teammates’ future actions, not including opponents. Additional experiments aim to study whether EFTM-based action selection works in other environments. Next, we promise to follow the reviewer’s suggestion about performance highlighting style.

- **Standard deviation for main results and additional experiments on SMAC** We apologize for the inconvenience. We wanted to report it in the main body, but due to the page limit, we included it in the appendix of the original manuscript. Our appendix includes Tables with the standard deviation as follows. |Character diversity|n=1|n=2|n=3|n=4|n=5| |-|-|-|-|-|-| |Proposed|**2899** $\pm$ 217|**3047** $\pm$ 162|**2976** $\pm$ 196|**2948** $\pm$ 91|**3051** $\pm$ 109| |FCE-EFT|**2899** $\pm$ 217|2784 $\pm$ 161|2646 $\pm$ 196|2566 $\pm$ 103|2629 $\pm$ 125| |MADDPG|2763 $\pm$ 126|**3006** $\pm$ 103|2800 $\pm$ 106|**2933** $\pm$ 98|2856 $\pm$ 121| |MAPPO|2753 $\pm$ 206|2862 $\pm$ 201|2597 $\pm$ 144|2529 $\pm$ 131|2763 $\pm$ 190| |QMIX|2199 $\pm$ 56|2310 $\pm$ 39|2288 $\pm$ 118|2118 $\pm$ 82|1861 $\pm$ 132| |Dreamer|**2911** $\pm$ 312|2813 $\pm$ 283|2733 $\pm$ 351|2631 $\pm$ 521|2701 $\pm$ 433| |MBPO|2089 $\pm$ 804|1964 $\pm$ 753|1523 $\pm$ 948|1893 $\pm$ 792|1633 $\pm$ 821| |Algorithm|MAPPO|MADDPG|QMIX|Proposed| |-|-|-|-|-| |Spread|-149.29 $\pm$ 0.94|-157.10 $\pm$ 2.30|-154.70 $\pm$ 4.90|**-149.12** $\pm$ 1.38| |Adversary|9.61 $\pm$ 0.07|7.80 $\pm$ 1.43|8.11 $\pm$ 0.37|**10.01** $\pm$ 0.33| |Tag|13.78 $\pm$ 4.40|6.65 $\pm$ 3.90|**15.00** $\pm$ 2.73|14.57 $\pm$ 2.95| These tables show that **the standard deviation of EFTM is similar to that of other methods**. The model-based solution has the highest variance due to the uncertainty of other agents. Overall, EFTM achieves the best performance with a mid-level variance compared to all other baselines. Additionally, we have run additional experiments on SMAC [1], which is widely used for evaluating the MARL algorithm, to address the reviewer's concern. We report the performance ($4$ seeds) with MARL baselines, as follows. |SMAC Task|EFTM|MAPPO|MADDPG|QMIX| |-|-|-|-|-| | 2s3z|98.8 $\pm$ 2.3| **100** $\pm$ 1.5|90.3 $\pm$ 5.3|95.3 $\pm$ 2.5| | 3s5z vs 3s6z |**84.3** $\pm$ 9.1|63.3 $\pm$ 19.2|18.9 $\pm$ 4.8|82.8 $\pm$ 5.3| | 6h vs 8z|**93.8** $\pm$ 6.7|85.9 $\pm$ 30.9|68.0 $\pm$ 34.7|9.4 $\pm$ 2.0| | Total|**276.9**|249.2|177.2|187.5| This result also demonstrates that EFTM still has surpassing or matching performance with previous solutions. It means that EFTM is capable of generalizing to solve widely-used MARL tasks, achieving the best total scores. Notably, we set a simple setup for the SMAC and MPE environments that is, we follow a vanilla setup with a single character diversity $n=1$. **While fully leveraging the advantages of EFTM in these environments can be challenging, EFTM is nonetheless capable of delivering competitive performance in such settings.** --- Once again, we deeply appreciate the insightful comments and suggestions. We hope our clarification and additional empirical studies could address the concerns raised by the reviewer. Should there be any leftover questions, please let us know and we will make every effort to address them during the subsequent discussion period.

- **Difference between role and character** We appreciate the reviewer for bringing out insightful discussion. A 'role' in a multi-agent system represents a responsibility or function for achieving the objective of a cooperative team [6]. Roles can be interpreted as subtasks for each agent. A 'character' refers to the specific behavioral strategies an agent employs to perform its assigned role. To illustrate, suppose a cooperative multi-agent task, including two different roles necessary to achieve the team’s goal, and two agents, being assigned a specific role. The agent aims to solve its subtask which can be solved using different strategies. The character endows a behavioral preference to the agent. These two concepts are considered and debated significantly in the MARL domain. While 'role' has been the focus of several prior works, 'character' concept remains relatively overlooked. **We sincerely emphasize that it is essential to consider a task with multiple agents with diverse characteristics in the MARL community.** We believe that our work can serve the beginning, and the broader impact on the community will be meaningful. Taking this into account, we will include extensive related works about 'role' and 'character' in the appendix of the final version. --- - **Standard deviation for main results** We apologize for the inconvenience. We wanted to report it in the main body, but due to the page limit, we included it in the appendix of the original manuscript. Our appendix includes Tables with the standard deviation as follows. |Character diversity|n=1|n=2|n=3|n=4|n=5| |-|-|-|-|-|-| |Proposed|**2899** $\pm$ 217|**3047** $\pm$ 162|**2976** $\pm$ 196|**2948** $\pm$ 91|**3051** $\pm$ 109| |FCE-EFT|**2899** $\pm$ 217|2784 $\pm$ 161|2646 $\pm$ 196|2566 $\pm$ 103|2629 $\pm$ 125| |MADDPG|2763 $\pm$ 126|**3006** $\pm$ 103|2800 $\pm$ 106|**2933** $\pm$ 98|2856 $\pm$ 121| |MAPPO|2753 $\pm$ 206|2862 $\pm$ 201|2597 $\pm$ 144|2529 $\pm$ 131|2763 $\pm$ 190| |QMIX|2199 $\pm$ 56|2310 $\pm$ 39|2288 $\pm$ 118|2118 $\pm$ 82|1861 $\pm$ 132| |Dreamer|**2911** $\pm$ 312|2813 $\pm$ 283|2733 $\pm$ 351|2631 $\pm$ 521|2701 $\pm$ 433| |MBPO|2089 $\pm$ 804|1964 $\pm$ 753|1523 $\pm$ 948|1893 $\pm$ 792|1633 $\pm$ 821| |Algorithm|MAPPO|MADDPG|QMIX|Proposed| |-|-|-|-|-| |Spread|-149.29 $\pm$ 0.94|-157.10 $\pm$ 2.30|-154.70 $\pm$ 4.90|**-149.12** $\pm$ 1.38| |Adversary|9.61 $\pm$ 0.07|7.80 $\pm$ 1.43|8.11 $\pm$ 0.37|**10.01** $\pm$ 0.33| |Tag|13.78 $\pm$ 4.40|6.65 $\pm$ 3.90|**15.00** $\pm$ 2.73|14.57 $\pm$ 2.95| These tables show that **the standard deviation of EFTM is similar to that of other methods.** The model-based solution has the highest variance due to the uncertainty of other agents. Overall, EFTM achieves the best performance with a mid-level variance compared to all other baselines. --- Once again, we deeply appreciate the insightful comments and suggestions. We hope our clarification and additional empirical studies could address the concerns raised by the reviewer. Should there be any leftover questions, please let us know and we will make every effort to address them during the subsequent discussion period.

Thank you for your active response! To ease any remaining concerns, we leave our opinions on additional questions below. --- **Experimental details** Thank you for this comment. Our implementation follows the official Git repositories from ToM2C [4] and opponent modeling [5] (In accordance with NeurIPS 2024 policy, we cannot upload hyperlinks in OpenReview). Given that we consider the POMDP setup, it is important to set how many other agents that an agent has access to. For ToM2C, we consider full access in accordance with the paper: they reported that full access has better performance than partial access. On the other hand, for opponent modeling, we consider six surrounding vehicles, not entire agents. That is because the reference paper aims to model the other agents in local information. Finally, training and validation setups are the same as other baselines. --- **Role and Character** Thank you for this constructive comment regarding the future direction of our community. A promising approach for combining the concepts of role and character would be to use a hierarchical structure. Each agent within a cooperative team first defines its role or subtask. The agent could then decide on the most effective strategy to achieve its subgoal, taking into account the characters and behaviors of other agents. We genuinely believe that this approach could be valuable in various studies, e.g., multi-agent planning tasks, as it emphasizes setting broad objectives first and then making detailed decisions based on interactions within the multi-agent system. --- **3D Environments** We apologize for our mis-clarification regarding the language models in VirtualHome. The authors of VirtualHome [3] reported they consider video with text, so we have a misunderstanding about the need for a language model. Thank you for your correction, and it may not strictly require a language model. We believe our concept could still be relevant to the testbeds you suggested. Since these environments are based on images or video, they would require more advanced forward prediction and representation networks to manage the complexities of 3D data. Specifically, VirtualHome operates in a 2D or 3D observation space, requiring at least 64 x 64 x 3 features as input. In contrast, SMAC and MPE tasks use a 1D observation space with about 100-200 and 10-20 features, respectively. By implementing an appropriate module for handling 3D data, our model could be extended to function in these more demanding 3D environments. We deeply acknowledge the value and importance of the reviewer’s request, so we would like to explore additional results in various domains. Regrettably, our group has limited GPU resources, unlike tech companies, making it challenging to get results for more computationally intensive tasks. At the same time, while applying our work to 3D environments is relevant, we believe that it is not the most critical aspect of our work. Our main focus is to develop a social decision-making process in a heterogeneous society where multiple characteristics coexist. We claim that the value of our method has been fully demonstrated in testbeds such as autonomous driving tasks, MPE, and SMAC. Sorry again that we could not include VirtualHome results, and we would greatly appreciate your understanding of our computational resource limitation. If you have any other questions or comments that could raise your score, we would be happy to continue the discussion, given the time! [3] P. Xavier et al., Virtualhome: Simulating household activities via programs. CVPR 2018. [4] Y. Wang et al., ToM2C: Target-oriented multi-agent communication and cooperation with theory of mind. ICLR 2022. [5] P. Georgios et al., Agent modelling under partial observability for deep reinforcement learning. NeurIPS 2021.