Multi-Instance Partial-Label Learning with Margin Adjustment

Summary

The paper proposes an approach to tackle the multi-instance partial-label learning (MIPL) problem. In MIPL, each training sample is represented as a bag of instances associated with a set of candidate labels. The existing MIPL algorithms may gives the high prediction probability on non-candidate label, which can lead to suboptimal performance. They propose MIPLMA algorithm addresses this issue by introducing a margin-aware attention mechanism and a margin-compliant loss function. The algorithm dynamically adjusts the margins for attention scores, while the loss function ensures that the margins between predicted probabilities on candidate and non-candidate label sets are properly constrained. The paper presents experimental results on benchmark and real-world datasets that demonstrate the superior performance of MIPLMA compared to existing MIPL algorithms, as well as other multi-instance algorithms and partial-label learning algorithms.

Strengths

1. The paper is well-organized and easy to comprehend. 2. The experiments are generally comprehensive. 3. Theoretical analysis contributes to establishing the effectiveness of the proposed model.

Weaknesses

1. The margin-aware attention is performed by setting a changing temperature parameter which seems a trival idea. And the details of how the gap of attention scores help the model to decrease the prediction probability on non-candidate labels are also not given. 2. The form of margin-compliant loss seems not easy to be optimized in neural network framework, the paper should give more details on the proposed loss function.

Questions

1. The initial weights are uniformly distributed. After the training in first epoch, the weights change. Subsequent epochs can only make the margin of the distribution larger, so the results are greatly affected by the first epoch. If there are errors in the training of the first few epochs, Will this error accumulate?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

In this paper, the authors observe the presence of ‘margin-violations’ in the MIPL problem, which manifest in two aspects: between positive and negative instances, and between candidate and non-candidate labels. Therefore, the paper proposes modifications to the attention mechanism and enhancement of the loss function to rectify and widen the margin.

Strengths

1. This paper aims to address a challenging scenario, specifically the MIPL problem, and identifies an intriguing phenomenon known as ‘margin violations’. 2. This work proposes MIPLMA to solve this issue, which combines several effective but not complicated tricks. 3. The authors conduct extensive experiments and compare with a number of methods to valid the effectiveness of the proposal.

Weaknesses

1. In Figure 1, the authors compare the performances of MIPLMA and DEMIPL, however, according to the results in Table 2 and Table 3, ELIMIPL surpasses DEMIPL on FMNIST-MIPL, and both ELIMIPL and MIPLGP perform better than DEMIPL on CRC-MIPL-Row dataset, so the comparison with these two methods would be more beneficial. 2. When discussing the linear-decay strategy for temperature in Equation 3, the sudden introduction of the constant coefficient 0.95 requires some clarification for better understanding. 3. In Sec.3.1.1, the authors claim that they are the first to employ ResNet on CRC-MIPL dataset, so it is suggested that the authors should list the detailed baseline performances on C-R34-16 and C-R34-25 datatset bag generators, just as in Table.3. 4. The authors assert that “features learned by ResNet-34 outperform those obtained via image bag generators in terms of classification performance.” However, it remains unclear whether this still holds on other benchmarks such as FMNIST-MIPL. To strengthen this claim, it would be beneficial to add further experiments or discussions on this.

Questions

See weaknesses

Rating

7

Confidence

5

Soundness

3

Presentation

4

Contribution

4

Limitations

NA

Summary

This paper deals with an emerging learning framework, i.e., multi-instance partial-label learning framework, which can be regarded as an extension and combination of multi-instance learning and partial-label learning. Overall, such dual inexact supervision makes the MIPLL difficult to resolve. This paper introduces a margin-aware attention mechanism and margin-compliant loss to deal with such issue.

Strengths

1. The motivation is clear and straightforward; 2. The paper is well-written and easy-to-follow. 3. Empirical results show satisfactory performance on several benchmarks as compared with existing methods.

Weaknesses

1. I am confused with the definition of MIPLL as the authors claimed in the paper: "positive instances refer to the instances that belong to the true label in the setting of MIPL"，I agree with the definition of positive instances. However, for the negative instances, " while negative instances represent the remaining instances in the bag that are not associated with any label in the label space". Does the author indicates that the negative instances are not associated with the whole label space or just the candidate label set for the given bag? I think the authors should be careful with this definition. I don't think the whole label space is the correct description w.r.t the negative instances. 2. The so called margin-aware attention mechanism and the margin-compliant loss are not novel, which in my opinion have already been utilized in other well-defined issues. So the novelty cannot reach the bar of NeruIPS. 3. In addition, although the multi-instance partial label learning framework is relatively novel and interesting, there has already existed at least two pioneer works, which again decreases the novelty of this work. 4. As compared with existing methods, the proposed algorithm cannot achieve the dominant performance on all the datasets under all the settings.

Questions

N/A

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

See the weakness.

Summary

In this paper, the learning scenario of multi-instance partial learning (MIPL) is studied. The paper proposes a new approach for MIPL, whose key technical idea is margin regularization. The paper points out that an important issue for previous MIPL approaches is the ignorance of margin information in both the outputs of attention modules and final predictors, leading to incorrect attention weights and prediction confidence. To address this issue, the paper proposes a novel margin loss for MIPL, whose effectiveness is verified under various MIPL benchmark datasets.

Strengths

1. In my view, the idea of margin regularization for MIPL is quite interesting, in special the regularization for the margins of attention weights. Even though attention weights are usually considered as a reflection of feature correlations, the accuracy of this correlation is difficult to quantify and receives less study. I think the margin measure is a good point for this quantification, and this idea can be indeed useful in the MIPL problem. 2. The proposed method revives the technique of margin distribution optimization. I like this idea due to its elegance in quantifying margin information. This also leads to the future possibility of theoretical studies for MIPL since the margin distribution framework is well theoretical grounded. 3. The experimental results are fruitful. The performance gain over baseline methods is significant.

Weaknesses

1. The discussion on the scheme of modifying the strength of margin regularization can be augmented.

Questions

I suggest including more discussions on setting the parameter \lambda. In my view, this parameter could significantly affect the learning performance. In my understanding, this parameter can be kept small at the beginning and be gradually increased. The reason is discussed in the paper: at the beginning, the model is less accurate, making the margin less informative, and the thing turns different as learning process proceeds. I find that currently, \lambda is kept as a constant. So I would like to ask for more discussions on the interpretations for this choice.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The paper proposes MIPLMA, a new Multi-Instance Partial Label algorithm that focuses on margin adjustments in both instance and label space. While computing the bag level representations, it introduces a temperature parameter in the margin-aware attention mechanism to widen the gap between attention scores for positive and negative instances. It also employs a margin loss to maximise the margin between the highest predicted probability on the candidate label set and on the non-candidate label set.

Strengths

- Introduces and adapts margin-based approaches from weakly supervised learning paradigms for multi-Instance PLL - The paper is well-written, and I was able to understand it clearly

Weaknesses

- In many cases, either the number of classes is low or the number of false positives is low, which raises concerns regarding the generalization of the algorithm - Paper has not been compared with some of the recent PLL algorithms like PICO [A], etc. A discussion on why such comparisons were not done would be helpful. Maybe adaptation was difficult, or they focused on a single modality? - Contributions are limited, it is a direct adaptation of margin-based losses to the problem - The applications of this setup need to be more convincing to me. They do give one application on a Cancer detection dataset, however, I believe it could be achieved through an architecture capable of processing higher-resolution images. [A] Wang et al. PiCO: Contrastive Label Disambiguation for Partial Label Learning. ICLR 2022

Questions

- The creation of the synthetic datasets is unclear. While Table 1 presents details regarding the number of instances in a bag, the distribution proportion of true positives in a bag is not mentioned. This could have implications for the performance of the algorithm. - In section 3.1.2 (line 200), It is mentioned that “PLL algorithms can be equipped with either the linear model or multi-layer perceptrons (MLP) as backbone networks”. However, DNNs like ResNet can be used to learn feature representations. Ideally, the comparison should be made using the same feature extractors, the ones mentioned in lines 208 - 213. - I would willing to reconsider my ratings post the author rebuttal

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Looking Forward to Your Feedback

Dear Reviewer BX8M, Thank you for your valuable feedback, which has significantly improved our paper. We have carefully addressed your concerns regarding the definition of negative instances and the novelty of our work. We have reviewed and discussed two related MIPL studies [A, B] with different negative instance setups, which will be discussed in the revised manuscript. Should any issues remain, we welcome further discussion. Best regards, The Authors [A] Wang et al. On learning latent models with multi-instance weak supervision. NeurIPS 2023. [B] Wang et al. On characterizing and mitigating imbalances in multi-instance partial label learning. arXiv:2407.

Looking Forward to Your Feedback

Dear Reviewer b79F, Thank you for your constructive feedback, which has significantly enhanced our manuscript. We have addressed your concerns by conducting experiments on the temperature parameter decay rate, providing results for the C-R34-16 and C-R34-25 datasets, and offering further explanations on the feature extractor. If there are any remaining issues, we are open to further discussion. Best regards, The Authors

Looking Forward to Your Feedback

Dear Reviewer Unv6, Thank you for your valuable feedback, which has greatly improved our paper. We have carefully addressed your concerns by providing visualizations of attention scores and bag-level features, detailing the optimization of the margin-compliant loss, and demonstrating how our dynamic disambiguation loss mitigates early errors. Should any issues remain, we are happy to discuss them further. Best regards, The Authors