Curriculum Fine-tuning of Vision Foundation Model for Medical Image Classification Under Label Noise

Summary

The authors propose Cufit, a curriculum fine-tuning method for improving the performance of medical image classification under the noisy labels setting. The method shows strong performance against other baselines on several medical datasets. The authors have also provided results on a non-medical dataset.

Strengths

**Strengths** 1. The proposed strategy Cufit outperforms other baselines included in the evaluation 2. The strategy is agnostic to models (CNNs and ViTs) and images (medical and natural) 3. Cufit does not require knowledge about certain hyperparameters which are required in other methods. 4. The authors have presented results on non-medical images as well.

Weaknesses

**Weaknesses** 1. The paper is very poorly written with spelling and grammatical errors throughout the text. Certain elements in the text are written in a convoluted way that confuses the reader. 2. The entire paper is about the proposed method “Cufit” but no section in the paper describes the algorithm in detail. Section 6.1 (“How does Cufit work?”) does not talk about the method but only about the results. 3. It is important to cite previous work on PEFT in medical image analysis such as [1] in the text. 4. The paper lacks an explanation of the baseline methods used in the evaluation. Methods like CoDis are briefly mentioned in the Related Work section but have not been defined anywhere else. People unfamiliar would not be able to understand Cufit and how it differs from previously proposed methods. 5. The experiments should include the CheXPert dataset. Apart from being frequently adopted for medical image analysis problems, it would also evaluate the proposed methodology for multi-label classification problems. Furthermore, CheXPert is supposed to contain noisy labels due to automatic labelling from free-text reports. Hence, it would adequately test the proposed method Cufit under a noisy label setting. 6. To make the experiments more extensive, more datasets and PEFT methods (see [1] for reference) should be included. LoRA is one of the most popular PEFT strategies used for transformer-based models (especially in the case of medical image classification [1]) and should be included in the experiments. 7. For natural image classification, the authors have adopted the CIFAR dataset. Firstly, in order to provide conclusive results, several natural imaging datasets should be included. Secondly, there are many datasets much more appropriate than CIFAR that should have been used instead. References 1. Dutt, Raman, et al. "Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity." *Medical Imaging with Deep Learning*, 2024, https://openreview.net/forum?id=LVRhXa0q5r.

Questions

**Questions** 1. After reading the paper several times, I am yet to understand how Cufit actually works. There is no section dedicated in the text that explains this. 2. Why are methods like LoRA that are very frequently used for doing PEFT in transformer-based models and datasets like CheXpert excluded from the analysis? Please address the **Weaknesses** section for more questions.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Please address the **Weaknesses** section.

Summary

In this paper, the authors propose a curriculum learning strategy for fine-tuning on noisy medical datasets. The key insight comes from that linear probing with limited training samples can be more robust to label noise. The performance is good compared to the former methods.

Strengths

Generally, I think this is a good paper. Noisy label is a severe and practical problem for medical scenarios due to diagnosis uncertainty and the intuition behind the proposed method is clearly stated. The performance looks good and the proposed method is clean and efficient.

Weaknesses

However, I still have to point out some details are not clearly demonstrated. Please check the questions for more details. I will change my score based on the authors' responses.

Questions

1. Figure 2 is confusing. The core content in the method may fall in the Curriculum order and the agreement criteria. But for now, it is not clearly reflected in the figure which makes section 4.2. I suggest the author reorganize the figure and detail the purple block. 2. I am not very clear about how the mode is trained. Commonly, when referring to curriculum learning, I guess the model will be trained with multiple steps but in equation 6 authors also modify the loss function which makes it also like a multi-task training pipeline. This confuses me a lot and the authors have to clarify it more clearly. 3. For the noise simulation. The authors have to state the details more, e.g., how are the correct labels changed just randomly or with some rules? and for multi-label or multi-class medical classification tasks, are there any differences? Similarly, this also confuses me in the method part. If a case with both pneumonia and bone fracture labels but the pneumonia part is wrong, will the whole case be dismissed for the latter training? 4. From lines 203 to 221, in introducing the dataset, it is important to specifically point out what datasets are multi-label tasks and what are multi-class, since in medical these two types all wide exist. 5. In section 6.2, it is also encouraging to explore more performance differences between general VFM and medical-wise VFM like PMC-CLIP and BiomedCLIP. While considering the rebuttal time limitation, this is just a bonus suggestion. 6. A minor suggestion, I suggest the authors clarify the definition of noise in their introduction part more formally as though for ML domain noise is clear, in medical domain, the noise types are complex which may be hard to understand for the audience in clinical background.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors adequately addressed the limitations.

Summary

This paper presents a curriculum fine-tuning paradigm called Cufit. This method is designed to fine-tune Vision Foundation Models (VFMs) for medical image classification tasks under the presence of noisy labels. The approach leverages the robustness of linear probing and the generalization capabilities of fine-tuning adapters to improve model performance.

Strengths

- Cufit is technically sound and is well-validated through extensive experimental results. - The paper is well-structured and effectively conveys the motivation, approach, and outcomes. - Demonstrated significant improvements in medical image classification performance under label noise. - Applicability to both medical and natural image classification enhances the relevance of the framework.

Weaknesses

- The training process seems to be complex and computationally intensive. - I have concerns about the scalability of the proposed method. It may not scale well for very large datasets or in resource-constrained environments.

Questions

- How does the computational cost of Cufit compare to other noise-robust training methods?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

None.

Summary

The paper presents Cufit, a curriculum fine-tuning paradigm for Vision Foundation Models (VFM) aimed at improving medical image classification under label noise. This method leverages the robust feature extraction capabilities of pre-trained VFMs and employs a linear probing strategy to mitigate the impact of noisy labels. The curriculum fine-tuning process then utilizes clean sample selection to enhance the classification performance. The experimental results demonstrate that Cufit outperforms existing methods on various medical image benchmarks, showing significant improvements in classification accuracy.

Strengths

1. The presentation is good.

Weaknesses

1. The paper includes experimental comparisons with methods like JoCor and CoDis, but the discussion about these methods' performance is insufficient. The authors should provide a more detailed analysis of why JoCor and CoDis do not perform as well as Cufit. Understanding the strengths and weaknesses of these methods in comparison to Cufit would offer valuable insights. 2. The paper should further discuss the impact of noisy labels on different types of biomedical images. For some image types, noise may be less detrimental, while for others, it could significantly affect diagnostic accuracy. A more detailed exploration of how noise impacts various biomedical image datasets would enhance the comprehensiveness of the study.

Questions

1. The paper includes experimental comparisons with methods like JoCor and CoDis, but the discussion about these methods' performance is insufficient. The authors should provide a more detailed analysis of why JoCor and CoDis do not perform as well as Cufit. Understanding the strengths and weaknesses of these methods in comparison to Cufit would offer valuable insights. 2. The paper should further discuss the impact of noisy labels on different types of biomedical images. For some image types, noise may be less detrimental, while for others, it could significantly affect diagnostic accuracy. A more detailed exploration of how noise impacts various biomedical image datasets would enhance the comprehensiveness of the study.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Thanks for the Rebuttal

Thanks for the Rebuttal. My questions have been answered to some extend and I have revised my rating.