Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels

Summary

This paper proposes a Label-Retrieval-Augmented (LRA) diffusion model for learning from noisy labels. The model leverages the neighbor consistency principle and incorporates pre-trained models to improve performance. The paper introduces the label-retrieval-augmented component, an accelerated label diffusion process, and a new conditional mechanism for incorporating pre-trained models. The proposed model achieves good performance both on synthetic and real-world benchmark datasets and the experimental results show that it can boost accuracy by 10-20 absolute points in many cases.

Strengths

1.The idea of using the diffusion model to address the noisy label learning problem is interesting. 2.Extensive experimental results demonstrate the proposed method can achieve the state-of-the-art method compared with multiple peer methods.

Weaknesses

The experiment part needs to be improved: add state-of-the-art method, discuss the influence of the label noise type, and the influence of different pre-trained models.

Questions

1. The paper mentions that the proposed model is flexible and general, but it does not provide a detailed discussion of the types of label noise that the model can effectively handle. For example, feature-dependent label noise. Since the comparison method [15] is a baseline that focuses on feature-dependent label noise. 2. It seems the pre-trained model is important for the proposed method. It would be better to discuss the effect of different backbone networks as well as pre-trained model on model performance. 3. The description of the comparison algorithms is lacking in the paper. 4. In Table 1, the comparison methods are outdated. It would be better to add some state-of-art methods published in 2022 or 2023.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the weakness and limitations.

Summary

The paper focuses on learning from noisy labels using diffusion models. They reformulate the noisy label problem from a generative perspective. Specifically, the clean label is our target variable ($y_0$) and the noisy label is the diffused variable ($y_T$), which they call Label-Retrieval-Augmented (LRA) diffusion model. Actually, we don't know the clean label, they extract the pseudo clean labels using the neighborhood consistency.

Strengths

* The paper is well-written. * The proposed method is a well-extended approach that builds upon the utilization of diffusion models in classification tasks and applies it effectively to tackle the noisy label problem.

Weaknesses

* Motivation of LRA diffusion model * It is important to explain why the noisy label process should be formulated as a diffusion process and to discuss in detail the advantages it offers. * Since I think it is a novel approach, it is crucial to provide detailed explanations of the motivation behind it. * When using the neighbor consistency principle, it is important to address whether the same problems mentioned by the authors in lines 44-51 would arise. If not, it is necessary to explain how the proposed model overcomes these challenges. * Comparison with CARD * It appears that the original CARD paper introduced two pre-trained encoders, while the current paper seems to have used only one. This distinction needs to be clarified, and the roles of the two encoders should be explained separately in the description. * Equation 4 seems to be adapted from the formulation presented in CARD, and proper citation should be provided. * Experiments * The utilization of CLIP in the performance evaluation raises concerns because it involves the incorporation of additional data. It is important to conduct a fair comparison by addressing this aspect. * An explanation is needed to clarify why the diffusion model is not a computational bottleneck. It is hypothesized that this may be due to the smaller dimensions, but this should be verified. * Minor comments * The appearance of $\tilde{y}$ in line 136 without prior definition should be addressed. * "Require: Input:" in Algorithm 1 is typo. * The inconsistent use of $\mu_{\theta}$ and $\epsilon_{\theta}$ should be corrected. * The subscript in line 153 needs to be written correctly.

Questions

Please see Weaknesses part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

They discussed in Limitations section.

Summary

The paper proposes the application of denoising diffusion probabilistic models for modeling the true class probability distribution when training with noisy labels. In particular, the paper extends Classification and Regression Diffusion Models for this problem and uses pretrained self-supervised representation models for Label Retrieval Augmentation. At test time, they approximate the true label by starting with the reverse diffusion at the mean of the final gaussian distribution. The empirical results on CIFAR10, CIFAR100 and other real-world noisy datasets show improvements over considered baselines.

Strengths

The main strength of this work is that it demonstrates successful application of diffusion models for learning the probability distribution of labels in noisy label settings. The paper considers a variety of datasets and includes the relevant ablation studies and inference times.

Weaknesses

I believe the key weaknesses are: 1) The requirement of a pre-trained self-supervised image embedding network. 2) The diffusion model does not sample discrete labels and does not even model the categorical distribution. 3) I think the evaluation should also consider the following baseline: train a classifier network exactly as in Algorithm 1 except in step 5 where the network is trained with an MSE loss to predict one-hot vector obtained in step 4. More specifically, it would be good to check the performance when the classifier network can take in the image and the simclr/clip features and have a linear prediction head. This baseline would be more closer to the LRA-diffusion model than the other baselines considered in Table 2 in terms of the model input and output.

Questions

1. In line 179, you write that you simply set $f_q({\bf x}) = {\bf 0}$ whereas you write in line 151 that the DDIM generation begins with a non-zero centered gaussian. So, the generalized DDIM is essentially used for the CARD+LRA Diffusion experiments in Table 2? And, you have a zero-centered gaussian distribution for the LRA-diffusion that you actually implement? 2. Do you have any intuition about the performance of the baseline proposed in Weakness (3)? 3. Do the baselines considered in Tables (3), (4) and (5) have access to extra training data? 4. Will you release all the trained model checkpoints used for reporting the results in tables?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes, the limitations are adequately addressed.

Summary

The paper proposes using a diffusion model to perform classification on a noisy-label dataset. The authors utilize diffusion to learn how to conditionally generate labels given an image. By combining recent work on classification diffusion models with label retrieval augmentations they demonstrate state-of-the-art results on real-world noisy datasets.

Strengths

- The method provides a simple, yet seemingly effective way of performing classification on noisy data using generative models. The idea of using diffusion models for purpose this is novel and has the potential to significantly contribute to the "Learning With Label Noise" literature. - Some of the real-world dataset results showcased outperform the previous state-of-the-art by a significant margin.

Weaknesses

- The novelty of the paper is limited. The authors combine the existing classification diffusion model with the retrieval augmentation technique to capture the noisy label distribution. The methodologies for both are inherited almost unchanged from their respective papers and there is no significant extension to either. Extending DDIM to a non-zero mean latent distribution is a minor contribution. - In lines 127 and 166, the authors claim that initializing with the predicted label mean and running the DDIM sampler approximates the maximum likelihood estimation of the labels. This is not justified anywhere, and there is no evidence to support the claim. - The experiments on the WebVision and ILSVRC2012 datasets should be examined further. The method presented in this paper relies heavily on having a high-quality pre-trained encoder network that can provide meaningful representations during training. For these two experiments, the authors utilized CLIP, whereas the next best-performing approaches, C2D and NCR, were only trained using samples from the smaller datasets. It is possible that the majority of the performance increases shown in this paper can be attributed to the discriminative ability of the CLIP encoder, since additionally on the Clothing1M dataset, where CLIP is not expected to perform well, the authors did not manage to greatly outperform the previous state-of-the-art results.

Questions

- For a single sample, is the frequency of the neighbors seen in training the same regardless of their distances in the latent space? If so, how is a neighborhood defined?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The limitations are addressed. Broader societal impacts are not applicable.

Summary

This paper considers image classification with noisy labels. Rather than consider a single label for each image, it considers the label distribution within the set of neighboring examples (determined using a pre-trained feature extractor), modelling this distribution by a learned diffusion process that is conditioned on image features, similar to CARD. The diffusion process maps a random label in the neighboring set to a normal distribution whose mean is either zero or the prediction of a baseline classifier, such that the reverse process should sample labels from the neighboring set. Inference is performed using DDIM to find the maximum likelihood label. The method assumes a pre-trained feature extractor (SimCLR trained on training data or CLIP trained on external data) is available for the purpose of finding neighbors and conditioning the diffusion process. The method is compared to strong baselines on several synthetic and real-world datasets and often provides a large improvement. Ablative studies demonstrate the importance of each component.

Strengths

1. The ablative study (Table 2; CIFAR10/100 with synthetic noise) demonstrates that the combination of CLIP/SimCLR features and LRA-diffusion is key to the effectiveness of the method: linear probing, LRA (label retrieval) and CARD (diffusion-based classification) alone do not achieve such an improvement. 1. Idea is well motivated. 1. Good coverage of related work. 1. Well written. 1. Synthetic experiments use strong type of noise based on second-highest confidence. 1. Inference time included in empirical results.

Weaknesses

1. Real-world evaluation (WebVision, ILSVRC, Food-101N) does not include simple baselines using CLIP features (KNN classifier and linear probe). This is important because it lets us verify that the results do not simply reflect the strength of the CLIP features. 1. Real-world evaluation does not include SimCLR features. This is important because it shows the performance of the method without external data. In general, it would be good to highlight which methods use (which) external datasets and/or pre-trained weights in the SOTA comparison. 1. (Sections 3.3 and 3.4) The modification of the DDIM procedure to allow non-zero mean seems to be redundant, since all experiments then used a zero-mean distribution by setting $f_q = 0$ (line 179)? 1. Unclear what "Linear probing + LRA" is in Table 2. Does this entail training a linear model using the average of the neighboring labels as the softmax-cross-entropy target? 1. It does seem inelegant to use a normal distribution to represent probability vectors (as acknowledged in text). Did the authors try working in real-valued logits instead? Suggestions: 1. I'm not a big fan of naming the method "retrieval", since the term suggests to me that training data is retrieved from a larger, external source of training data (not the case). 1. Discussion of impact of $k$ (number of neighbors) is deferred to the supp mat and I didn't see any references to this in the main text. It would be good to at least reference it and ideally add a plot. 1. Could be good to include some discussion of singly-labelled vs multiply-labelled noisy labels, since the proposed method effectively constructs a multiply-labelled dataset from a singly-labelled dataset.

Questions

Please address weaknesses.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have identified limitations including: 1. dependence on pre-trained feature extractors 1. method is less effective beyond ~50% corrupted labels 1. normal distribution not ideal for working on probability simplex I do not see any ethical issues.

Response to Rebuttal.

Thank you for the extra experiments and answers to my questions.