Adversarial Environment Design via Regret-Guided Diffusion Models

Summary

This work focuses on Unsupervised Environment Design (UED), a problem setting whereby a teacher designs environments for a student, learning to solve the task. This area of research has been in focus recently due to its ability to train more general agents in an open-ended setting. The authors look to build on recent work using generative models to generate environments, and propose a diffusion based approach ADD. The method seems sound and leads to gains in two of the canonical UED environments. I recommend acceptance as it incrementally improves UED literature.

Strengths

The main strengths are listed below. Since I recommend acceptance, the focus of the review is on weaknesses to improve the paper as much as possible for the camera ready version. * The method makes sense intuitively and seems to work as expected, especially with the curriculum results. * The paper is well written and clearly builds on previous works (e.g. in the related work discussion). * Experiments are well presented and show clear performance gains.

Weaknesses

* The x axis for the plots is showing total steps and not student gradient updates. This puts replay methods at a huge disadvantage since they take ~2x the number of environment steps, but the policy isnt training so arguably it should not count when comparing the effectiveness of the curriculum. I think this should be changed to match the literature. * The baselines are fairly weak, aside from PLR and ACCEL (which have been weakened by the previous point). Can the authors compare against CLUTR and show why ADD produces a better curriculum? To me that is the obvious baseline. Further, note the improvements to PAIRED proposes in [1] which makes it a much stronger baseline. I would say the original PAIRED is more of a concept than a baseline at this point due to known deficiencies. * There is a clear limitation of your method that you require a pre-training phase with access to the environment, and do not count the number of steps as part of training because it is "unsupervised". This is fine, but then makes it wrong to show training steps for the replay methods. Please either 1) include the steps from this in the x axis for ADD (which will drastically weaken performance) or 2) switch the x axis to student gradient updates. Otherwise it is an apples to oranges comparison in your favour. * I would love to see examples of the generated environments in the main body, for me it is more useful than the theory part, but maybe that is subjective :) * More of a general comment, I find it strange that UED papers don't cite [2]. It is the largest scale demonstration of the value of UED and motivates research on these methods. It explicitly shows PLR is effective for curricula over a massive task space with a 500M parameter transformer based policy. This clearly shows that UED research can have impact on large scale AGI focused projects, so I would have expected the UED community to all be very excited about this. [1] Mediratta et al. "Stabilizing Unsupervised Environment Design with a Learned Adversary". CoLLAs 2023 [2] Bauer et al. "Human Timescale Adaptation in an Open-Ended Task Space". ICML 2023

Questions

* The meta data plots show ACCEL starting with the same number of stumps etc as the other baselines. However in the original implementation/paper it should be starting with very simple terrains. Was this the case in your work?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation of requiring additional data to pre train the diffusion model is not mentioned, I think this is actually the largest weakness by far. It could be interesting to see if pre trained foundation models work well here too.

Summary

This work applies regret-guided diffusion models to the UED setting in order to generate adversarial environments that preserve diversity.

Strengths

* The contributions are well motivated and appear to be novel. * The writing is generally clear and concise. * The paper is contextualized well within prior literature.

Weaknesses

* Minor typos/grammatical issues: * Line 253: “challenging”—This word alone does not hold any descriptive power. Because the nature of the tasks are described next in impartial terms, this word seems redundant/rhetorical. * The limitations do not mention the assumed structure of the environments; I am not convinced that diffusion can be applied to all environment parameterizations. * Why is shortest path length used as a complexity metric? It seems to describe the tail of the distribution; Why not average path length? Also, number of blocks does not seem to be a well-motivated complexity metric either. * While ADD “successfully generates adversarial environments while preserving diversity”, the behavior improvements seem to be minimal based on the training curves / zero shot performance. * A more general point: it is hard to tell what exactly the authors are trying to show from the experiments; the conclusion nicely sums up the analysis, but it would have been nice to have been told at the top of “experiments” what each experiment is trying to test. * Given the prevalence of UED domains in the literature, a third evaluation domain would really strengthen this work. Otherwise, I am concerned about the applicability of this method to other environments, especially since the authors have not addressed the implicit assumptions about workable environment parameterizations for their diffusion approach.

Questions

* What are the assumptions on the structure of the environment parameterization? Can this be applied to all simulated environments? Or only a subset with certain properties? * Why were RGB channels used to describe the maze environment instead of e.g. a binary (continuized for the diffusion process); it seems odd to parameterize the environment based on the observation space, and then to extract these parameters themselves based on the observation values. Are there certain parameterizations for the diffusion process that work better/worse? What properties does this representation space need to have? * Are the t-SNE embeddings generated using all points, and then each individual plot displays just the relevant method’s points?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

* The limitations are outlined accurately by the authors in Section 6. However, I would like the authors to address the first question above, which I believe is an additional limitation that would be worth discussing.

Summary

This paper proposes an approach for gradient directed, regret-based UED based on guiding a pre-trained diffusion model.

Strengths

This paper addresses a major shortcoming of prior UED approaches. In the past gradient-based UED approaches have been out-performed by sample-based or evolutionary approaches for searching for environments. While there was a general intuition that gradient based approaches would ultimately be more scalable, it was difficult to realise this intuition due to the fact that environment-design is a high-dimensional optimisation problem for which RL algorithms have struggled. Using insights from generative modelling is a natural approach to bridging this gap, and this paper provides a few tricks which seem to have been missing from prior attempts. Specifically, pre-training the diffusion model on random levels, and using learned guidance seems to be a particularly powerful combination.

Weaknesses

My main concern is that It seems like the baseline results in the bipedal walker domain don't replicate the results from prior work, suggesting some sort of bug/lack of tuning in the implementation? It appears that if I compare the numbers of ADD to the results in the ACCEL paper it gets lower or equivalent performance to ACCEL itself? My main reason for not currently raising my score is questions about the accuracy of this evaluation. There are some tricks that are rediscovered from prior work which should be attributed. Specifically, entropy regularisation for neural generators is studied in [1], and training a sort of critic to evaluate levels is studied in [2]. That being said, ADD puts a unique spin on both of these. [1] Mediratta, Ishita, et al. "Stabilizing unsupervised environment design with a learned adversary." _Conference on Lifelong Learning Agents_. PMLR, 2023. [2] Bhatt, Varun, et al. "Deep surrogate assisted generation of environments." _Advances in Neural Information Processing Systems_ 35 (2022): 37762-37777. For the citation of environment design, it is correct to cite the UED formalism was from Dennis et al. but for the more general concept of designing environments it would be best to also cite contemporaries POET[3] and GPN [4]. [3] Wang, Rui, et al. "Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions." _arXiv preprint arXiv:1901.01753_ (2019). [4] Bontrager, Philip, and Julian Togelius. "Learning to generate levels from nothing." _2021 IEEE Conference on Games (CoG)_. IEEE, 2021. It would also be best to be careful not to equate minimax regret based UED approaches with all UED approaches as is done on line 83, as there are many non minimax regret approaches to UED such as POET, SAMPLR, CLUTR, and DRED. Pre-training on random levels seems like it leaves only a limited amount that the diffusion model could learn. It would be interesting to periodically fine-tune the diffusion model on newly generated levels to increase the power of the generator over time. This would generally be much more convincing as it could scale much further off of the distribution of random levels. #### Clarity It would be helpful to include the per-transfer environment bar-plots as is traditional with UED papers, it seems like the same information is included in Table 5 and 6 but they are much harder to read in that format, and it is difficult to tell where error bars overlap. It would also be helpful to include the bootstrapped CI's plot often in UED papers as recommended by [5] Figure 4 and Figure 9 would be amazing to include in the main body, and go a long way towards explaining and demonstrating the method. [5] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." _Advances in neural information processing systems_ 34 (2021): 29304-29320.| |APA|| How do you arrive at the ADD acronym? It is a bit difficult for me to remember it and what it stands for.

Questions

Have you tried replacing equation 12 with a PAIRED-style loss maximising the expectation between a protagonist and antagonist?

Rating

8

Confidence

4

Soundness

2

Presentation

4

Contribution

4

Limitations

See weaknesses.

Summary

This paper proposes a diffusion model with differentiable regret estimate for unsupervised environment design. The authors write a diffusion process to model environment parameters where the process is described in terms of a scoring function and derivative of the regret. The scoring function is pre-trained on a set of random environments. The diffusion process is further fine-tuned with the regret to generate environments for curriculum learning of an agent. A critic, which is trained with cross-entropy loss using binned environment returns, is used to approximate the regret. By using the derivative of the critic w.r.t. environment parameters, diffusion process is fine-tuned with entropy augmented regret. Experimental results on continuous and discrete control domains show that the model is competitive with previous best models. Using regret guidance significantly improves the performance.

Strengths

The paper introduces a diffusion process which is well suited for modeling continuous parameter. It implements a differentiable regret approximation that better assigns credit for the diffusion process. The formulation through the optimal environment distribution with a trainable scoring function is also interesting.

Weaknesses

There are a few things that need more clarification and ablations. 1. While the paper shows that the diffusion process with a differentiable regret estimate gives good results, it is not clear which component is the most critical. Is the diffusion process critical for the success? Can you train PAIRED with a differentiable regret? 2. Similar to above, is the entropy term critical? Can you train the diffusion process without it and get comparable results? 3. Can you explain the reason why you trained the critic model with binned returns rather than using actual returns in a regression objective? 4. How critical is the pre-training for the scoring function? How does the performance change with less or more number of environment samples?

Questions

Please see above for specific questions.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper addresses the limitations.

No score change

I am already a high score for this so the rebuttal was unlikely to see an increase. Please include a discussion of Weakness1 in the main body. I don't agree with the comment "[4] eventually utilizes replay in the bipedal walker" because there are still stronger baselines than the ones in your paper. These should be in the main body, why wouldn't they be?

Response to Rebuttal by Authors

>ACCEL was implemented in two ways, ACCEL+ and ACCEL++ (we replace dagger of the original paper with "+" for readability in the review format). ACCEL++ This makes sense! I think ACCEL++ is the conical version as one of the core insights of that paper is that there is often a small number of conical "empty" levels which are a better initialisation for evolution. But it does make sense to compare against ACCEL+ given that the inductive bias is orthogonal to your approach and we want to isolate the effect of the different level optimiser. You should include them both and flag this nuance, but I agree this is a valid experimental methodology and a fair comparison. >We will include an explanation of this difference in the paper to ensure that readers are aware of these differences. Additionally, we will add the performance of our method trained for only half of environmental steps used in the original experiment to accommodate the perspective that using the number of policy updates as a metric is more reasonable. It would also be useful to compare at the number of steps as the original ACCEL, since the longer horizon will give a better sense of the long term performance. Including this both in terms of environment steps and policy steps are interesting, and would be useful for the community to have a sense of the nuances in current SOTA. Give that I'm now convinced the empirical evaluation is correct, and since this paper presents a novel and promising attack against one of the biggest shortcomings of prior UED approaches, I expect it will have a large impact on the field. I'm raising my score to reflect this.