Strengths
The paper introduces a, to the best knowledge of this reviewer, novel concept of over-generalization to the space of all possible parameters. The proposed method SSED is not terribly novel, but makes sense and shows the effect nicely. The open sourced maze environment is another good contribution of the paper.
Weaknesses
**TLDR:** the paper is interesting, but in a bad state. On the one hand, it is very confusingly written, overpromises and does not do all it claims. The first contribution is also dubious: the derivation is incomplete (many terms just appear without definition), seem to rely too much on intuition, and the results do not show the effect. On the other hand, the second contribution (SSED) is simple but interesting, and the results demonstrate that it works. Nonetheless, the reviewer cannot recommend to accept the paper in its current form.
1. The paper overpromises many things, for example from the abstract:
- "As a first step, we measure the mutual information (MI) between the agent’s internal representation and the set of training levels, which we find to be well-correlated to instance overfitting": there is no empirical measurement of MI or its correlation with overfitting in the paper. All evidence is circumstantial and the interpretation of Figure 2, in this reviewer's opinion, wrong.
- "adaptive sampling strategies prioritising levels based on their value loss are more effective at maintaining lower MI", "We then turn our attention to unsupervised environment design (UED) methods, which [..] minimise MI more effectively than methods sampling from a fixed set.", "SSED generates levels using a variational autoencoder, effectively reducing MI": again, MI is never measured and the claim that the evaluated methods minimize MI is pure conjecture.
2. The paper is confusingly written and introduces many terms without defining them. For example, $\text{MI}(i,\pi)$ is never defined in the main paper. $S^V_i$ is defined with $\hat V_t$ and $V_t$, which have never been defined. On Page 5 the authors state $V_i^\pi(s) = V^\pi(s) + v_i^\pi(s)$, but $V^\pi(s)$ has never been defined (only for a general POMDP, the CMDP only knows context-specific values called $V_{i_x}^\pi$). Sometimes the used PPO baseline uses non-recurrent intermediate representations (p.5), another time the same implementation uses an LSTM-based architecture (which is recurrent, p.7). The reviewer has the impression that many of these details could be understand upon reading the cited papers, but as is, the paper does a poor job explaining them.
3. The connection between generalization, MI and scoring the value loss ($S^V$) is indirect, and not rigorous enough. The connection between MI and generalization is not very clear: in Figure 1 both following the black and the green cells leads to the goal in (a) and (b), but only one of the two generalizes to (c). Both have thus the same MI, but different generalization. While the statement "An agent learning level-specific policies implies high MI between its internal representation and the level identities" (p.1-2) makes generally sense, the conclusion that agents with high MI "will not transfer zero-shot to new levels" is less clear. Moreover, it is not apparent why the negation *agents with low MI generalize well*, which seem to be the basis for this paper, should be true. So the only clear connection seems to be the bound in (eq.2). The reviewer also missed a connection why "Sampling levels associated with highly negative classifier cross-entropies therefore results in less mutual information" (p.5). Finally, the idea that $V_i^\pi(s) = V^\pi(s) + v_i^\pi(s)$ separates into two errors and the second is somehow connected to MI (without any clarification how exactly) is flawed, as the errors of $V^\pi$ and $v_i^\pi$ could also cancel each other out, yielding low error, but high MI.
4. The connection of MI to generalization and to the value-error scoring is not supported by the presented data in Figure 2. Here the first two rows are not significantly different, which would lead to the conclusion that using $S_2=S^{MI}$ does *not* change the algorithm's behavior much. How does this justify the statement "$S = S^V$ therefore appears to strike a good balance between sample efficiency and mutual information minimisation", if MI is not necessary for performance. The third row shows $S^{MI}$ having a smaller generalization gap, but the performance is also much smaller, and these two metrics are highly correlated! The results can therefore also be interpreted as "using MI reduces the performance, and *as a consequence* has a smaller generalization gap". For the same reason the second plot to the right is useless without knowing the algorithms' performances.
5. The entire Section 4 should be either removed or replaced by an experiment that actually links generalization (not just the generalization gap) to MI, demonstrates or proves that sampling levels with high MI *reduces* the overall MI, and shows a correlation between MI and the value-error score.
Questions
- The approximation in Lemma 4.1 only works if the levels are constantly sampled with $p(i)$, but your method aims to change this distribution. How does this work when your scoring selects the levels that make up the batch $B$?
- Your goal is to produce policies that have low MI on the training set, but approximated MI is not used to change the policy, only the sampling of levels. Is there a reason why you do not simply reward actions with lower MI?