Taking on feedback and responding to comments
Thank you for taking the time to read our rebuttal and for sharing your further concerns. We strongly believe that these are addressable and endeavour to do so within this comment.
> It should be made more clear in the text that the figures relate to the evaluation and not the training phase.
We take this onboard and adapt our captions accordingly.
> How come even in in-weights training, the plot for the single-lifetime training is essentially a fixed line?
Thank you for pointing this out! To ensure that our single-lifetime baseline is fair, we use the same population size (i.e., number of seeds) as for accumulation and report the best performing agent's (i.e., argmax over seeds) maximum achieved return over the duration of its training as the dashed single-lifetime baseline. We now explicitly state this in the manuscript.
> We currently have no view on the learning curves of RL^2.
We can provide these in the Appendix. We would like to have been able to include them in the rebuttal if they are of importance to the reviewer, but they were not initially requested and we cannot upload new figures or amend the pdf during the discussion period.
> There are potentially a lot multi-agent RL techniques to compare to [...] techniques for POSGs, implicit and/or explicit communication in multi-agent RL...
We would like to humbly remind the reviewer that we use the setting of a POSG to introduce the formalism for our algorithms before explaining how the setting reduces to a POMDP because the policies of previous generations are fixed. This means that we are training successive single-agent RL policies with PPO and using the methods we introduce to achieve performance improvements via cultural accumulation. For in-weights accumulation, the baseline of partial network resets [1] therefore makes sense, which we show *additionally* improves accumulation itself in Figure 5 (right).
Approaches to implicit communication in MARL [2, 3] assume that agents are incentivised to *learn to communicate* due to cooperative rewards. We make no such assumptions, as each agent in our study is simply maximising its own independent reward. We believe that combining our work with implicit or explicit learned communication in cooperative settings would be an exciting direction for future work.
> ...or indeed cultural transmission across generations...
Could the reviewer please clarify what they mean by a baseline here? We are precisely exploring cultural transmission across generations, where previous work has explored a single step (i.e., one generation) of cultural transmission [4, 5].
> It is not clear what are the comparative scores of the techniques compared to the state-of-the-art.
We would like to emphasise that prior work on social learning and cultural transmission in RL [4, 5] has focused on learning from an expert for a single generation of transmission, both in a setting similar to our in-weights setting [4] and where human experts provide demonstrations at test time [5]. In both cases, the reported baselines are ablations of their method, as the purpose was to demonstrate that social learning can be achieved and provide performance benefits in RL. In our work, we extend these ideas to span multiple generations, whilst making no additional assumptions and removing the assumption of having an expert at test time from [5] by creating this generational bootstrap.
The goal is therefore not to achieve state-of-the-art performance on a set of benchmarks, but to demonstrate that cultural accumulation can be modelled in RL and that it can reap performance benefits relative to running the same algorithm for one continuous lifetime.
In an effort to further address this concern, we provide the final performance of successive generations of policy distillation (i.e., generational training) [6] in comparison to the final performance at each generation of in-weights accumulation.
| Task | Generational Training | In-Weights Accumulation |
|-------------------------|-----------------------------|------------------------------|
| Memory Sequence | 0.6, 2.7, 6.6, 7.5, 8.5 | 0.6, 4.2, 7.9, 8.3, 11.4 |
| Goal Sequence | 0.7, 1.8, 2.9, 3.1 | 0.7, 3.7, 4.2, 4.3 |
> ...do not seem to mention which noise level is used for the results in the evaluation plots.
We take this onboard and now include them in our stated hyperparameters.
[1] The primacy bias in deep reinforcement learning, Nikishin et. al, 2022
[2] Learning to communicate implicitly by actions, Tian et. al, 2020
[3] Foraging via multi-agent RL with implicit communication, Shaw et. al, 2021
[4] Emergent social learning via multi-agent reinforcement learning, Ndousse et. al, 2021
[5] Learning few-shot imitation as cultural transmission, Bhoopchand et. al, 2023
[6] Open-ended learning leads to generally capable agents, Stooke et. al, 2021