Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis

Summary

The paper tackles the problem of multiple-choice learning (MCL) in the regression setting with specific focus on the overconfidence problem and hypothesis collapse problem found in previous approaches for MCL where predictions from the heads corresponding to rare events are overestimated. The proposed rMCL model frames the problem as a multimodal conditional distribution estimation problem. The authors propose a new loss function that adds a new hypothesis scoring loss to the existing multi-target winner takes all (WTA) loss. The proposed algorithm is compared against other baseline models on both a toy dataset, as well as a sound source localization (SSL) task. The experiments on the toy datasets visually show the effectiveness of the rMCL approach in tackling the overconfidence problem as compared to a method that does not use the additional loss term. The results on the SSL dataset show that the rMCL algorithm is practically applicable to problems with multimodal outputs with consistently better performance in the case of multiple sound sources as compared to the other algorithms.

Strengths

- The presented approach builds on top of existing literature and solves the overconfidence problem using an additional term to the existing loss function. - The authors visually demonstrate the effectiveness of the approach using a simple toy dataset, and follow up with application to a real-world regression task (SSL).

Weaknesses

- The experiment on SSL is not very clear to me and might benefit from some additional details. Specifically, I am not clear on the interpretation of the EMD and oracle error. My understanding is that for each input sound snippet, the model predicts the angles of each source from each of the output heads. Does the oracle only count the best prediction out of all the source angles predicted, while the EMD accounts for the overall error?

Questions

See above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The authors have addressed limitations with the presented work.

Summary

The authors propose Resilient Multiple Choice Learning (rMCL), a modification of the Multiple Choice Learning (MCL) approach, for conditional distribution estimation in regression contexts where each input can have multiple target samples. While MCL is a straightforward strategy for multimodal density estimation, it uses the Winner-Takes-All (WTA) loss for various hypotheses. In regression situations, the prevailing MCL versions focus on combining the hypotheses, which may compromise the diversity of the predictions. In contrast, rMCL employs a new learned scoring system, which is supported by a mathematical framework based on Voronoi tessellations of the output space. This approach allows for a probabilistic interpretation of the results. The authors tested rMCL using synthetic data and found it to be effective. They also applied it to the problem of sound source localization, demonstrating its practical utility and the relevance of its interpretation.

Strengths

Novel Approach: The proposed Resilient Multiple Choice Learning (rMCL) provides a fresh perspective to tackle the Multiple Choice Learning (MCL) overconfidence problem, especially in regression settings. This offers a new method for researchers and engineers to approach this issue. Learned Scoring: rMCL is based on a learned scoring scheme that handles multi-target settings. This flexibility could allow rMCL to perform well across a variety of tasks and datasets. Probabilistic Interpretation: The authors provide a probabilistic interpretation of the model, which could help in understanding the model's behavior, tuning its performance, or extending it to new applications. Evaluation and Application: The paper demonstrates the practical utility of rMCL, especially in the context of sound source localization. It illustrates the resilience of the model in both synthetic and real-world datasets, showing that the method can handle real-world complexity. Advantages Over Previous Methods: rMCL, applied to the Sound Source Localization (SSL) problem, seems to alleviate the issues related to imbalanced spatial positions and the source permutation problem. It does not require prior knowledge of the number of sources, which is a significant advantage in practical applications.

Weaknesses

The method heavily relies on the dependence on high-quality data: Like many machine learning models, the performance of rMCL might be significantly impacted by the quality and representativeness of the training data. Further evaluation might be required based on whether the model is affected in the presence of noise in the data and the quality of the labels. The performance of the rMCL approach may not be suitable when one is approaching to scale the data wherein the size and complexity of the dataset would affect the nature of the machine learning task at hand especially when one considers the hypothesis space which is large. It's not clear how rMCL would respond to different types of noise in the data or how robust it is to outliers.

Questions

Based on the weakness it would be helpful for authors to provide comments on how the rMCL approach might work in case of noise in the data and whether there is any comparative analysis of WinnerTakesAll type approaches which take into consideration of the label data noise. Despite the probabilistic interpretation offered by the rMCL approach, it might still be challenging to understand and explain the model's decision-making process, which could be a limitation in certain applications where interpretability is crucial. How does authors see the probabilistic interpretation aiding/modified to overcome the interpretability of the model learnt? It might be the case where the probabilistic interpretation may yield incorrect result with overfitting of the model.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors do state the limitations of the work clearly. They review the performance of the model and conclude that it appears that the performance of Resilient Multiple Choice Learning (rMCL) is influenced by the number of hypotheses being considered. Particularly, attempting to predict an overly large number of hypotheses without prior knowledge can introduce errors into score predictions. In addition, using rMCL to address the overconfidence issue inherent in Winner-Takes-All (WTA) variants can slightly degrade the performance of the best hypothesis, though the reason for this is not currently known. Lastly, while ε-WTA seems to behave as expected with rMCL, the same is not observed for top-n-WTA, necessitating further study.

Summary

This paper proposes a technique for resilient Multiple Choice Learning (rMCL), which extends the vanilla Muliple Choice Learning (MCL) paradigm to conditional distributions for regression where multiple targets maybe sampled for each training input. It is known that MCL uses multiple scoring heads to score multiple hypothesis for a given input and suffers from the twin challenges of hypothesis collapse (where only a small subset of the possible prediction heads are trained well, as a result of the Winner Take All strategy) and overconfident predictions (where rare classes are overly represented). This work focuses principally on addressing the latter issue. A key feature of the method is the use of a learned scoring scheme based on Voronoi tessellations which lends itself to a probabilistic interpretation. Results are reported on the sound source localization task and on synthetic data and compared against some standard alternatives.

Strengths

The following are the key strengths of the proposed approach: 1. This is possibly the first work that extends MCL to a regression setting. 2. The proposed solution attempts to overcome the overconfidence problem of a standard MCL system by casting the MCL as a conditional distribution estimation technique while allowing for a probabilistic interpretation of the same. 3. Experiments show that the method, which is capable of assigning prediction probabilities to low density regions, is also interpretable. 4. The code for the proposed method has been made available.

Weaknesses

The following are some of the principal weaknesses of the proposed approach: 1. The proposed method does not deal with the hypothesis collapse problem of MCL methods. 2. The contrast with prior approaches is not well brought out in the Related Works section. 3. No performance is reported on more recent sound source localization datasets such as LOCATA or the DCASE 2019 dataset 4. Moreover, the proposed approach falls way short of competing methods on the Oracle metrics, on both the reported datasets. The authors present too much emphasis on just the multimodal EMD results without justifying why the drop in Oracle distance metric should be ignored.

Questions

Following are some of the questions/comments for the authors: 1. How does the current work compare against prior works in Multilabel Learning? [1, 2] 2. How do the methods perform on the Euclidean distance measure, compared with the Oracle Error? 3. This reviewer fails to see the value in ensembling the proposed approach with WTA (Table 3). Why not combine with IE or PIT then? 4. Please rephrase Line 111: "More precisely...." References: [1] Zhu, X., Li, J., Ren, J., Wang, J. and Wang, G., 2023. Dynamic ensemble learning for multi-label classification. Information Sciences, 623, pp.94-111. [2] Kim, Y., Kim, J.M., Akata, Z. and Lee, J., 2022. Large loss matters in weakly supervised multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14156-14165).

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes the limitations of the method have been well addressed.

Thanks for addressing my concerns

In light of the additional experimental results presented and the response to my other questions, I am raising my score.