Summary
The paper proposes to apply deterministic simulated annealing to multiple choice learning (MCL) as a means to mitigate some of the drawbacks associated with the winner-takes-all (WTA) scheme used to train MCLs, such as sensitivity to initialization and hypothesis collapse. They demonstrate that the proposed annealed CML (aCML) method works well in practice, on par with previous approaches, and allows for an interesting and sound theoretical analysis of the training trajectory.
Strengths
- The proposed method, aMCL, is very well motivated both theoretically and based on existing open research questions in multiple choice learning. Moreover, the mathematical development of the paper is very clear and seems sound.
- The paper is very well written and easy to follow.
Weaknesses
- The main weakness of the paper is that the experimental results, with the exception of the quite enlightening toy experiments, are somewhat underwhelming and aCML does not seem to outperform the baselines in most cases. I think the paper could benefit from a more in-depth discussion as to why that is the case.
- Admittedly, I am not an expert on the CML literature and maybe guilty of hindsight bias, but the idea of applying simulated annealing to CML seems quite intuitive and, one might argue, somewhat incremental. For instance, simply annealing $\epsilon$ in $\epsilon$-CML (Rupprecht et al., 2017) seems quite natural, and perhaps a good baseline for the authors to compare against. That does not take away from the solid mathematical motivation and theoretical results in the paper though.
### Minor issues
- Line 201: “in no longer” should probably be “is no longer”.
- The work of Rupprecht et al. is mentioned under different names (Relaxed WTA, $\epsilon$-WTA and $\epsilon$-CML) which can be a bit confusing.
- It is not entirely clear to me what the authors want to show in Figure 2.
- The robustness result in Figure 8 is quite interesting and could be expanded upon (perhaps with more random seeds) or at least mentioned in the main paper.
### References
Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D Hager. Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In Proceedings of the IEEE international conference on computer vision, pages 3591–3600, 2017.
Questions
1. Do the authors compare against the score-based method of Letzelter et al.? In line 232, it is suggested that this was one of the baselines, but it was not mentioned again in the paper, unless I missed it.
2. On a similar note, could the authors elaborate on the relation between aMCL and rMCL (Letzelter et al., 2023)? It seems to be both have similar objectives, but the assignments in aCML are a function of the temperature of the system and the loss function, while in rMCL the assignments are learnable. If that is correct, would be fair to say that, given enough data and sufficient learnable parameters, we can always expect rMCL to outperform aCML?
3. It is somewhat surprising that aCML does not outperform $\epsilon$-CML in many cases. Do the authors have any intuition as to why that could be? Is it because the temperature schedule is hard to tune or aCML would require a larger number of epochs to fully converge? Or maybe the datasets are not particularly sensitive to bias of $\epsilon$-CML?
4. Have the authors considered stochastic simulated annealing as well?
### References
Letzelter, Victor, et al. "Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis." Advances in neural information processing systems 36 (2023).
Limitations
The limitations section is very well written and covers all possible shortcomings of the model and analysis proposed in the paper that I could think of.