Improving Environment Novelty Quantification for Effective Unsupervised Environment Design

Summary

After rebuttal I have upped my score to 7. I think this paper is good, as it makes a small, easy to implement and simple to understand change to existing UED methods, and it delivers improved empirical results. ----- This paper uses a GMM-based method to quantity novelty of levels in UED. It uses the state-action distribution (unordered, i.e., not trajectory distribution) induced by the agent on a level and fits a GMM to the data from previously sampled levels. A new level's novelty can be determined by computing the likelihood of its induced state-action distribution under the fitted GMM.

Strengths

- The idea of using state-action distributions to represent a level is not new, but it makes sense. - Using a fitted model and computing the likelihood using this also makes a lot of sense, compared to computing a pairwise novelty score between two levels. - Having the ability to have different numbers of GMM kernels is an interesting idea, and seems to provide a good way of allowing complexity to be dynamically altered. - Results demonstrate that the new method performs better than PLR/ACCEL, and the change in code is small.

Weaknesses

- While the focus on unordered transition tuples is mentioned as a benefit, I would think that there are certain environments where the temporal nature of trajectories are important. Could you comment on this please? - Minor - Line 88, for minimax one also often cites [1] - The $\tau$'s beneath the max/expectation in equation (1) don't have the A and P superscripts. [1] Pinto, Lerrel, et al. "Robust adversarial reinforcement learning." _International Conference on Machine Learning_. PMLR, 2017.

Questions

- Algorithm 1, please indicate what the blue text means (I assume changes to ACCEL but being clear on this would be helpful.) - Please provide aggregate results averaged over all of the minigrid eval. levels. It is hard at a glance to see how the algorithms compare. - Do the GENIE generated levels actually look more diverse than those generated by e.g. ACCEL? Please show a sampling of levels. - In table 1, PLR-GENIE has the most state-action coverage, but performs much worse than the ACCEL-based methods. This seems to contradict the claim that the increased diversity is causing better performance. Could you explain this please? - There may be some confusion between your GENIE and [1] - For car racing you do use images, is the image simply flattened? Could this have problems with e.g. not being translationally invariant? [1] Bruce, Jake, et al. "Genie: Generative interactive environments." _Forty-first International Conference on Machine Learning_. 2024.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

- I think putting limitations in the main paper would be better. Figure 5 feels not super necessary and can be moved to the appendix if space is an issue. - Some other limitations I can think of, please comment/explain - Choosing the range of $K$ (number of GMM) kernels can be challenging? - Do I understand correctly that you just concatenate the observation and actions to form $x$, which is then used to fit the GMM? How can this scale to much larger observations?

Summary

This paper focus on the unsupervised environment design (UED) problem, whereby a student trains in an adaptive curriculum of environments proposed by a teacher. The authors propose GENIE, a method for assessing novelty of environments, which essentially means the teacher prioritizes environments with high exploration potential/info gain for the student. Experiments are conducted in three different relevant domains and the results are clear and show a reasonable gain. I thus favor acceptance. One thing to note is the general idea of choosing environments based on novelty (rather than regret) is itself a new idea, thus, the authors do not need to focus as much on the use of GMMs, which may be limiting.

Strengths

* The method makes sense as it is essentially an intrinsic reward for the policy, i.e. selecting environments where the agent will have higher information gain. * The empirical results are strong, with the method improving both PLR and ACCEL in three different domains. * The use of the evaluation protocol from Agarwal et al is always refreshing. * Ablations are sensible and easy to follow.

Weaknesses

* There is a strong assumption that all transitions are independent, which is not always true. What if there is an environment where an agent needs to conduct some initial behavior (e.g. finding a key) and then using it to act later (e.g. opening a door)? It just feels like the method is designed based on the toy environments we use in RL research, and is not actually scalable for larger more complex domains in the future. * Hate to be that person, but the name GENIE is taken by multiple works already and recently by Bruce et al (2024) in a highly relevant paper. It would be recommended to find a more relevant acronym or simpler name. * The arrows in figure 2 are too small.

Questions

What other approaches could be used to assess novelty at larger scale? Can you show any examples of levels that have high novelty but low regret, and turn out to be useful training levels for the agent?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Discussed in the Appendix.

Summary

This paper proposes using novelty quantification in an unsupervised environment design for training a more generalizable policy. Built on an intuition that environments with unfamiliar states are novel environments, their proposed algorithm uses Gaussian mixture models to allow an RL agent to explore novel environments. The authors of this paper compare their proposed method in various benchmarks against multiple baselines to show empirical improvement in performance.

Strengths

This is a well-written paper with a structure that is easy to follow. The key concepts are simple and easy to understand.

Weaknesses

- The idea of using unfamiliar states to quantify uncertainty seems similar in concept to the curiosity-driven approaches in RL. However, this paper does not address the relevant literature. It would be much better if the authors could explain how their idea fits in with the findings and theory in curiosity-based RL approaches and what the paper contributes to this avenue of thinking. - When it comes to UED in curriculum learning to train a more generalizable policy, there is Genetic Curriculum (Song et al, 2022) which also uses UED in curriculum learning to train a generalizable policy. Since the paper was also evaluated on the BipedalWalker and BipedalWalkerHardcore environments, it would be better to compare and contrast how GENIE with Genetic Curriculum. - GENIE uses a fixed window FIFO buffer, but would this be able to reach an equilibrium? For example, if the agent explores level set A at the expanse of forgetting level set B, and goes back to exploring level set B at the expanse of forgetting C, and so on, would agents trained by GENIE converge to a steady-state behavior? - Finally, there are some minor typos and editorial mistakes. For example, line 63, PLR, and ACCEL are mentioned without explaining what those are.

Questions

The questions I hope to be addressed by the authors are the ones listed above. 1) Explanations on how this paper fits in with curiosity-based approaches and the GENIE's contributions, 2) explanations and comparisons with Genetic Curriculum, and 3) Will the agents trained by GENIE reach a steady-state equilibrium?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors has addressed the limitations of this paper.

Summary

This paper proposes adding a domain-general metric for promoting novelty to state of the art UED methods in order to help the environment generator better explore and cover the space of environments. The novelty bonus is based on a surprise of a (state, action) pair model which is a learned Gaussian mixture model.

Strengths

The method is clearly useful and easy to implement, fixing a limitation of existing UED approaches which are known to miss modes in the space of levels. Empirically the method seems to perform strongly, matching or exceeding existing methods. This is a valuable direction to pursue as it is important for the UED community to have a sense for how novelty effects performance, and the approach addresses a well-known flaw in existing UED approaches.

Weaknesses

The method would be more convincing if the generated levels for each method where visualised and compared directly. Ideally one could demonstrate that there are a motif of level being generated which were not being generated before. For instance, displaying the distribution over stump hight late in training of ACCEL and ACCEL-GENIE to show that ACCEL-GENIE has a more even coverage of the space, and sampling a few levels randomly from each so that this can be visually confirmed. Similarly, the paper would be much improved if the mechanism behind the results was dug into in more depth. If the theory is that the diversity term results in kinds of high-regret levels being presented which were left out of the buffer previously, it would be ideal to demonstrate that this happens directly. On line 258 results are over-claimed, since ACCEL-GENIE is within the margin for error of ACCEL in all, or nearly all, of the environments. This claim should be corrected. I don't think this is essential for acceptance of the paper, as the minigrid benchmark appears pretty close to being saturated, and it serves largely as an MNIST for the field. The empirical results in the other two environments stand on their own. #### Clarity: It would be good to have error bars in Table 1 The plots in Figure 7 should be fixed so the colors of the same method match There is a stray parenthesis on line 303 Equation 1 shows the PAIRED objective "optimized" for deterministic domains, the PAIRED objective as a direct comparison between the two expected returns is canonical. I would also suggest reconsidering the name, as another prominent environment-generating model named GENIE has been released recently. I expect that, in talking about this work, people will want to talk about how the two could be combined, which could get quite confusing. In the abstract I think it is not clear that novelty, on it's own, is critical for agent's generalisation ability. Two situations which are different but not in a way that is relevant for the task and are processed by the network in the same way are effectively the same, and would not effect generalisation ability. It also appears that in a few places the "underspecified" in UPOMDP is taken to mean that there is a one-to-many mapping between parameters and environments. In general this is not the case, as the minigrid environment has a one-to-one mapping between parameters and environments, the "underspecified" simply means that these parameters are not given by the designer. It is fair to want UED algorithms to work even when there is such a one-to-many mapping, but it is not required as part of the problem formulation.

Questions

What sort of qualitative difference do you see between ACCEL-GENIE and ACCEL levels?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See weaknesses.

Discussion

Thank you for your detailed response! A few more follow ups - For car racing I meant specifically how the states are input into the GMM and not the policy. Do you use a CNN as a preprocessing step before the states are passed to the GMM/do you use the agent's CNN, or do you just flatten the obs and pass it to the GMM? - Could you please expand a bit more on how you can incorporate a temporal relationship between states? Naively the thing to do would be to effectively frame stack to get an augmented state space. Are there other ways? - Relatedly, the policy's representation does not have to be the same as GENIE's, right? So you could framestack for one but not the other? - Then regarding the increased state coverage. If I understand correctly, you are saying the order & diversity matters, and not diversity alone? In that case I think rephrasing the discussion in l297-l300 would be beneficial, as I at least did not get that impression from reading it.

No change to score

Thank you for the rebuttal. I don't see much scope to increase my score, I think the paper has sufficient merit to be accepted. If it is accepted please include a discussion on scaling the approach in the future work/conclusion. This could even be by combining it with an environment generator like Bruce et al's Genie :)

Response to Rebuttal by Authors

Thank you for your response. The added visualisations increase my confidence in the method, so I will be increasing my score accordingly. > As such, it is impossible to directly measure whether prioritising or not prioritising a level via GENIE would lead to discovering a higher regret level down the road because the divergence in realities would result in different policies with non-commensurable subjective regrets. One test that could show this effect would be if the policies from both realities found the levels from the GENIE reality to have higher regret. I agree if this was not the case it would not be negative evidence. Another good measure would be to fix a policy (and freeze it so it does not train) and run both to see which finds higher regret levels.