Disambiguated Attention Embedding for Multi-Instance Partial-Label Learning

Summary

This paper proposes a new algorithm for a weakly supervised learning paradigm called multi-instance partial label learning, where each sample is represented by a multi-instance bag and its label is provided as a set of candidate labels. This is a particularly challenging learning paradigm since learners do not have access to which label is the ground truth label from the candidate label set, and do not know which instance corresponds to the ground truth label. The authors proposed a deep learning algorithm to solve the problem integrating the attention mechanism with a disambiguation strategy. The attention mechanism map the multi-instance representation into a single-vector and the disambiguation strategy identifies the true label from the candidate label set. The synergy of these two components is validated with a wide range of benchmark datasets. Furthermore, this manuscript also proposed a new crowd-labelled medical dataset that can be used as future benchmarks for this learning task.

Strengths

* The problem setting that motivates this study is novel and potentially impactful. The multi-instance partial label learning paradigm simultaneously addresses the challenges two of the most widely studied weakly supervised learning setting, multi-instance learning and partial label learning. * The paper introduces a new real-world dataset for histopathology image classification from weak supervision in the form of multi-instance bags supervised with candidate label sets. This indicates that the proposed learning paradigm and algorithm can potentially alleviate the labelling cost of medical image dataset. * The proposed disambiguation attention mechanism applies to both binary and multi-class classification scenarios, whereas existing attention mechanisms designed for multi-instance bags can only handle binary classification. * The proposed algorithm demonstrates good performance improvement compared to a wide range of existing arts.

Weaknesses

* The synergy of disambiguation attention mechanism and momentum-based disambiguation strategy can be further explored/evaluated. See questions for more on this.

Questions

* DEMIPL performs classification by aggregates the instances into a single-vector bag representation. Would it be better if instance representation and bag representation are used simultaneously for classification and disambiguation? * Since the momentum-based disambiguation strategy degenerates to progressive disambiguation strategy proposed in [39], how would the performance be if momentum-based disambiguation is used directly in the algorithm proposed in [39]? This may provide empirical evidence on the synergy of momentum-based disambiguation and disambiguation attention. * Is there feasibility to learn which instance corresponds to the true label in the candidate label set, i.e., instance-level classification from MIPL datasets? * Is there any plan for releasing the CRC-MIPL dataset?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

This paper does not explicitly discuss its limitations.

Summary

The paper proposes the first deep learning-based method for multi-instance partial-label learning (MIPL) problem. The proposed method employs a disambiguation attention mechanism to approximate the contribution of each instance to the bag-level feature and then aggregates a multi-instance bag into a single vector representation, with an attention loss introduced to enhance the discriminativeness between positive and negative instances. The method then uses the extracted feature to identify the ground-truth label from the candidate label set using a momentum-based disambiguation strategy and simultaneously learns the classifier. Furthermore, the paper introduces a real-world MIPL dataset for colorectal cancer classification. Empirical results confirm the effectiveness of the proposed method and the contribution of the key designs included.

Strengths

1. The paper is reasonably clear and comprehensive. 2. The paper’s claims are supported by experimental and ablative results.

Weaknesses

1. A notation in Section 2.1 at line 91 is incorrect. Since none of the instances in the bag belong to any of the false positive labels as the author claims in line 42, the first \mathcal{Y} at line 91 should be replaced to \boldsymbol{S}_{i}. 2. I don’t think the weights calculated by equation (2) is “attention” scores, since the computation process only involves a single instance without interaction between different instances, and there does not exist query, key and value, or they are all the same instance. These scores are only weights that approximate the contribution of different instances to the bag. 3. I have some doubts about the proposed method. See the question part below.

Questions

1. I have questions about why does the method work. The momentum-based disambiguation strategy simply minimizes the cross entropy loss between the predicted label and all candidate labels, with importance weights added which are obtained by momentum updates from probability prediction. I wonder why this method ensure that the positive label in the candidate set to be correctly selected (disambiguated) ? If this does not give a correct direction, the representation of the instance bag will also be misled, and how can the model reach satisfactory state when converge as the author claims in line 158 under this situation? 2. The method proposes to minimize the “entropy” of the attention weights calculated to make the attention scores between positive and false positive labels discriminative enough. However, if multiple instances in the instance bag share this positive label, I worry that simply minimizing the entropy would result in a single instance with high attention score against all other low-score instances, neglecting the other positive instances, so the consistency between attention scores and instance-level labels is broken.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

The article proposes a novel algorithm called DEMIPL for addressing the challenges in multi-instance partial-label learning (MIPL) tasks. The algorithm uses a disambiguation attention mechanism to embed multi-instance bags into a single vector representation and a momentum-based disambiguation strategy to identify the ground-truth label from candidate label sets. Experimental results on benchmark and real-world datasets demonstrate the superiority of DEMIPL over existing MIPL approaches.

Strengths

1. The paper is easy to read and the core idea is clearly presented. 2. DEMIPL proposes a new algorithm that addresses the limitations of existing MIPL approaches and introduces a disambiguation attention mechanism and momentum based disambiguation strategy. 3. The article presents experimental results on benchmark and real-world datasets, demonstrating the superiority of DEMIPL over other approaches.

Weaknesses

1. Recently, using distribution of labels to generate partial labels have been considered as more difficult data scenario, which increases the difficulty for the model to identify similar samples. The authors are encoraged to incorporate such type of data for experiments. 2. The reason why Eq. 7 uses model prediction confidence and weights in previous training rounds needs to be further clearly clarified.

Questions

1. In addition to the MNIST-MIPL and FMNIST-MIPL data sets, what is the input of other data sets to attention, are the features alreadly extracted? 2. Comparing the attention-based network with the comparison method, is there any unfairness?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above questions.

Summary

The paper introduces a novel algorithm named Disambiguated attention Embedding for Multi-Instance Partial-Label learning (DEMIPL) to address the problem of multi-instance partial-label learning (MIPL). The proposed algorithm embeds multi-instance bags into a single vector representation using a disambiguation attention mechanism and a momentum-based disambiguation strategy. The paper also presents a real-world MIPL dataset for colorectal cancer classification. Experimental results demonstrate the superiority of DEMIPL over existing MIPL and partial-label learning approaches.

Strengths

1. DEMIPL introduces a unique scheme of embedding multi-instance bags into a single vector representation, addressing the limitations of existing MIPL approaches. 2. Disambiguation Attention: The use of a disambiguation attention mechanism allows for better aggregation of bag-level information, considering global context, and reducing sensitivity to negative instance predictions. 3. Momentum-Based Disambiguation: DEMIPL employs a momentum-based strategy to identify the ground-truth label, improving the accuracy of label prediction. 4. The paper introduces a real-world MIPL dataset for colorectal cancer classification, enabling evaluation in a practical setting. Experimental results demonstrate that DEMIPL outperforms other established MIPL and partial-label learning approaches, showcasing its effectiveness in solving the MIPL problem.

Weaknesses

1. When discussing the experimental results, provide more information about the benchmark datasets used for evaluation. Explain the characteristics and sizes of the datasets, as well as any preprocessing steps applied. Additionally, describe the performance metrics used to assess the classification accuracy and provide a clear comparison with existing MIPL and partial-label learning approaches. 2. While the proposed DEMIPL algorithm demonstrates superior performance compared to existing approaches, it is important to acknowledge any limitations of the proposed method. Discuss potential scenarios or datasets where DEMIPL may not perform optimally and propose directions for future research to address these limitations. 3. Proofread the paper for grammatical errors and improve the overall clarity of the writing. 4. Consider providing more details about the real-world MIPL dataset for colorectal cancer classification. Describe the acquisition process of the dataset, the number of images per class, and any challenges or biases associated with the dataset. 5. Consider providing a high-level visual representation or diagram of the DEMIPL algorithm to aid in understanding the proposed approach. 6. Ensure consistency in notation and symbols throughout the paper.

Questions

See the above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See the above.

Summary

In this paper the authors study multi-instance partial label learning, where a bag of instance is associated with several candidate labels of which it is assumed one is correct. The propose a deep learning approach for such a problem. The key contribution is a "disambiguation attention mechanism" where they find the relevance of each instance to each class. Each bag is then represented by a weighted average of instance features with the attention being the weights. They modify the loss function to improve consistency between attention scores and instance labels. Further they also propose a momentum based loss for the labels that gives higher weight to higher class probabilities. The provide experimental results on several synthetic datasets and one "real-world" dataset.

Strengths

+ the MIPL setting could have some applications in the real world. + Using attention to construct bag features seems like an interesting idea. + a variety of experimental results have been reported against several baselines. + additional experiments study the effect of the contributions in isolation.

Weaknesses

- The assumption of a single correct label seems very strong and unrealistic in practice. - The histopathology task seems really contrived. It is hard to imagine non-experts labeling such images, and if they did, how could it be assumed any labels were correct? - Without a compelling application, it is not clear what is the significance of this work.

Questions

How was label quality controlled in the histo dataset to ensure there was one correct label?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

not adequately discussed

I am satisfied with your response.

Thank you for your response. I would change the rating to accept.