Summary
This paper explores the joint training of deep ensembles, wherein the ensemble error is directly optimized during training. The authors find that joint training leads to poor performance, which they posit it due to phenomenon they call "learner collusion", where base learners artificially inflate their diversity, at the expense of test performance. The authors provide a new decomposition of a broad class of loss functions in terms of the average error rate, the ensemble error rate, and a generalized notion of diversity. The authors then perform an experimental evaluation into the prevalence of learner collusion during joint training, using an augmented objective function to probe the effect of explicitly encouraging diversity during training.
Strengths
Overall, the paper is very well-written and easy to follow, and in my view the topic is very relevant to the community.
To me, the most compelling experimental evidence for the learner collusion phenomenon are those presented under "learner codependence"/in Figure 4. From these results it is very clear that something distinct is happening in the joint training regime, and that the individual models are exploitating the joint loss. The definition of the augmented objective $\mathcal{L}^\beta$ seems like a very useful tool for probing these and other related phenomena.
Weaknesses
Unless I am misunderstanding, the conclusions drawn from Figure 2 do not seem to strongly support the claim that joint training significantly harms ensemble performance. Indeed, for the Facebook task, it seems the method actually helps, while for others it does not. This is in contrast to the CIFAR results in Table 1/Figure 6, where the degredation in performance is significant. Could the authors provide some clarification as the intended conclusion from these experiments?
There are a number of other papers in the literature that study some form of the gap (in the authors' notation) $\bar{\text{ERR}} - \text{ERR}$, and express and/or bound it in terms of various forms of "diversity", depending on the given loss function. For example Ortega et al., 2022, Abe et al., 2022 and Masegosa et al., 2020 give explicit expressions for the average - ensemble error gap in terms of diversity metrics that are easily interpretable. It's unclear to me that the form in Theorem 4.5 adds much to this literature, and indeed the stated expression for diversity doesn't seem to be used in the remainder of the paper.
Overall, while I think the paper is well-written and the topic is relevant, I think there is 1) insufficient use/novelty in the theoretical results and 2) somewhat incomplete conclusions to be drawn from the empirical results to recommend accepting at the the current stage. In particular, while it certainly appears like some type of "learner collusion" is happening, it seems very important to understand why this only appears at $\beta=1$, and exactly what impact this could have on the ensemble error.
As a minor point, there seems to be an issue with the references -- they don't seem to contain any titles of the referred papers.
**References:**
Taiga Abe, E Kelly Buchanan, Geoff Pleiss, and John Patrick Cunningham. The best deep
ensembles sacrifice predictive diversity, 2022.
Luis A. Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural
network ensembles, 2022.
Andres Masegosa, Stephan Lorenzen, Christian Igel, and Yevgeny Seldin. Second order PAC-
Bayesian bounds for the weighted majority vote, 2020.
Questions
I find the distinct change in behavior of the various metrics at $\beta=1$ very interesting; it seems the behavior is relatively benign for $\beta < 1$, and yet there appears to be a distinct transition at $\beta = 1$. Do the authors have any hypotheses as to why this occurs?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.