ALIM: Adjusting Label Importance Mechanism for Noisy Partial Label Learning

Summary

This paper addresses the noise partial label learning problem, i.e., the ground-true may also be in the non-candidate set. The proposed method adjusts the importance of the labels by considering the output of the candidate label set model in an integrated manner. The proposed method in this paper can be viewed as a post-processing means for prediction models, and the proposed method achieves by superior performance compared to existing methods on multiple benchmark datasets.

Strengths

1. The noise PLL issue is important and the authors have a good summary of the unresolved issues in that paper from existing work. 2. Rich experimental results.

Weaknesses

1. This paper mainly uses the candidate label set and non-candidate label set information to post-process the prediction results, and the innovation is weak. 2. The framework of the method has a stronger assumption that P(x) can also achieve better results in noise partial label learning problem, otherwise it is difficult to inform the subsequent processing. From Figure 1, we can see that for noise partial label learning, P(x) has been able to calculate the confidence of each label well, and the confidence of the true label is higher compared to ALIM. 3. While the way of adjusting label importance proposed in the paper avoids errors when detecting noise directly, the method used in this paper will also assign a certain confidence level to all non-candidate labels, which will also introduce noise. 4. One of the key points of the method proposed in this paper is the setting of the hyper-parameter lambda. An adaptive method is proposed in this paper, but the adaptive parameter is not analyzed in the experimental results to know at what value the setting is reasonable. 5. For Mixup Training, the paper does not explain why it can counteract the effects of noise in PLL.

Questions

1. In line 91, the authors describe that w(x) should keep very small data, so how can the true labels be extracted from the set of non-candidate labels? 2. How is the conclusion derived from the different values of K in the paper, which correspond to different normalizations? 3. What is the principle of Mixup Training in noise partial label learning against noise?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

N/A

Summary

This paper focuses on a relaxed version of partial label learning, called noisy PLL. Different from PLL, noisy PLL allows the ground-truth label may not be in the candidate set. To deal with this problem, this paper proposes a framework called ALIM, which is a plug-in strategy with theoretical guarantees. Experimental results on CIFAR-10, CIFAR-100, and fine-grained datasets prove the effectiveness of ALIM on noisy PLL. This paper is well-written and easy to follow.

Strengths

1. This paper presents a plug-and-play framework for the challenging noisy PLL task. 2. This paper conducts theoretical analysis from objective functions and EM algorithms. 3. ALIM shows competitive performance compared with the currently advanced strategies.

Weaknesses

$Scale(\cdot)$ normalization function introduces a scaling factor $K$. But in the following experiments, this paper does not specify the selection process of this factor. Please explain its impact on the performance and how it was chosen.

Questions

Can the authors explain the impact of the scaling factor $K$ on the performance and how it was chosen?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the weakness and question.

Summary

The paper considers noisy partial label learning when the provided label is not in the candidate label set. The paper proposes "Adjusting Label Importance Mechanism" to trade off the initial candidate set and model outputs. Some theoretical analysis is done and good empirical results are shown against selected baselines.

Strengths

1. The considered problem (i.e. the provided label is not in the candidate label set) might be a relatively new problem and is less considered in existing work. 2. Some theoretical analysis is done. 3. A few tricks are proposed to make the method more effective. 4. Good experiment results are shown against baselines.

Weaknesses

1. Writing can be improved. Currently, it seems the manuscript is prepared in a hurry. For example, there are errors when referring to appendices: “Appendix ??”. Overall, I had a hard time following. 2. The core contribution is not clear. The motivation in Section 2.2 is hard to follow. From Section 2.2, it seems the paper is trying to consider the case when the ground-truth label is not in the provided candidate set of labels. It should be made clear why this is an important problem and how the proposed method addresses the problem. Also, there are concerns on the core technique. see Q1 and Q4. 3. The theoretical analysis is not clear see Q2.

Questions

1. When the label is not in the candidate set, it’s also likely that it's not in 1-S(x). Actually, not being in the label space might even be the main reason that the label is not in the candidate set. How to handle this case? It seems there is an assumption that the label is for sure in the given label space, and it might not be in the provided candidate which is a subset of the label space. This should be made clear. 2. In Equation 5, why does M correspond to the lambda in equation 2? In section 2.4.2, it is also unclear what is the connection to EM. 3. Abstract mentions theoretical guarantees, what are the guarantees? It is not mentioned in the paper anywhere. 4. From the theoretical analysis, it seems there is an equivalence between the proposed method to an (existing?) objective function. Then, why would one need the proposed method?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Limitations are not discussed. Authors should discuss when the proposed method doesn't work.

Summary

This paper focused on the problem of noisy partial label learning (noisy PLL), which means the candidate label set in the partial label learning does not always contain the ground-truth label. Compared with the existing studies which attempted to detect noisy samples yet lead to the accumulation of detection errors, this work proposed to reduce the impact of the detection errors by training off the initial candidate set and model outputs. The proposed method, named "Adjusting Label Importance Mechanism (ALIM)", can adaptively adjust the weighting coefficient between the initial candidate set and the model predictions (i.e., pseudo-labels output by the model). The theoretical analysis showed that the proposed method can be well interpreted from two perspectives of heuristic design and classic Expectation-maximization (EM) algorithm. Combined with Mixup training strategies, the proposed approach can be further plugged into the existing noisy PPL baselines and bring consistent improvements. Extensive experiments demonstrated the effectiveness of the proposed framework.

Strengths

- The motivation is clear. The analysis of the issues of the existing methods motivated the proposed method in a straightforward manner. - The proposed objective can be interpreted under the EM framework, which provided a convincing mathematical explanation. - The proposed method, ALIM, is also compatible with other related approaches. It can work as a plug-in in the existing noisy PLL algorithms, which enhanced the flexibility of ALIM. - The experimental part conducted empirical comparisons/ablation studies with respect to different factors, e.g., the different noise levels and ambiguity levels, and the different choices of the key components within the proposed framework.

Weaknesses

- Some technical details were not clear to the reviewers. See the next part for more details.

Questions

- In Line 140, what is the definition of $\mathcal{L}_{pll}$? It suddenly appeared here without any pre-defined formation. Which kind of PLL loss did it indicate, the objective in Eq.(5)? - I recommend the authors reformulate the optimization objective in Eq. (5) as a $\min$ form, which will be better compatible with the form of the loss function. - Compared to the results reported on the CIFAR10 dataset, the reviewer found that ALIM-Scale consistently performed better than ALIM-Onehot on other datasets like CIFAR100 and CUB200. Is it a necessity that scaling performs better than one shot on datasets with larger class numbers, or a coincidence? Could the author provide more analyses and explanations about this point? - Some potential typos: - Line 95, Appendix ?? - Line 107, Appendix ?? - Line 126, Appendix ??

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No

Response to Author Rebuttal

Thanks for the effort from the authors in answering my questions. My concerns have been addressed and I accordingly increase my rating to 7.