FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification

Summary

This paper proposes a few-shot learning approach for WSI (Whole-Slide Image) classification. This approach, built upon Tip-Adapter, leverages a cache branch to memorize the knowledge from few-shot instances and then retrieve label information from the cached knowledge. In addition, a prior branch, which utilizes the knowledge from CLIP and GPT4-V, is built to boost the predictive performance. The experiments on two WSI datasets show the superiority of the proposed method over the original Tip-Adapter and Tip-Adapter-F.

Strengths

- Originality. This paper explores the few-shot setting in the context of WSI classification. It is under-studied in the field of computational pathology. - Significance. This work shows that the proposed approach could obtain a performance near that of fully supervised learning, only with a few labeled instances. Overall, it is an interesting work worthy of investigation in computational pathology, given that labeling WSIs at pixel level is extremely time-consuming and labor-expensive. - Quality. This work presents a good experimental design. Its experiments are conducted from different angles to verify the effectiveness of the proposed algorithms.

Weaknesses

In summary, my main concerns lie in (1) writing quality, (2) limited technical contribution, and (3) missing experimental comparisons with important vision-language-based models in computational pathology. Based on these critical weaknesses, it may be hard to recommend accepting this paper to NeurlPS. The details are given below: - This paper is overall rough and sub-par in writing, requiring substantial improvements in clarity. Some obvious flaws are as follows: i) slice-label in line 102, ii) a efficient annotation strategy in line 111, iii) undefined V in line 160, iv) some undefined notations in Section 3.2, and v) some citation errors such as [20]. The authors are encouraged to check these errors and improve their presentation for better academic communication. - The technical novelty is quite limited. Most key designs of the proposed approach have been proposed in Tip-Adapter (Zhang et al., ECCV 2022). Compared to Tip-Adapter, the proposed approach does not present valuable or substantial technical contributions, since its two crucial components, few-shot knowledge retrieval in the cache branch and the prior branch, seem borrowed from Tip-Adapter. - The authors claim that their work differs significantly from Tip-Adapter (line 142) because the key-value cache model built by Tip-Adapter only allows the key to be learnable; in contrast, their approach allows both the key and the value to be learnable. The authors are encouraged to rephrase this sentence, as i) also allowing the value to be learnable should not be called a significant modification from my humble understanding; ii) Tip-Adapter actually has proposed to use a learnable value in the key-value cache model but it leads to collapse during training. Moreover, given that the learnable value leads to collapsed training in Tip-Adapter, could the authors explain to readers why their methods can avoid collapsed training? - Some important vision-language-based models in computational pathology are not cited and compared to the proposed methods, such as **PLIP** (Huang et al., Nature Medicine, 2023) and **CONCH** (Lu et al., Nature Medicine, March 2024). These models show exciting zero-shot performance in WSI classification combined with MI-Zero. The proposed approach should at least show better performance than their zero-shot performance. It could be crucial for justifying the value and significance of this work. Minor issues: - It is suggested to rewrite Section 3.2 to make sure that all notations (those in texts and figures) and the implementation of components are clear and well-explained. - The original Tip-Adapter is proposed for traditional single-instance settings. Its implementation for few-shot WSI classification is not clear.

Questions

- Few-shot instances are randomly selected from the core set. Since only a few instances, e.g. 16, are selected from a very large instance pool and pathological patches (instances) often present heterogeneity, the final selected instance set could have a large variety and thus lead to unstable performances, calling into question the usability of the proposed method. This could also be observed from Fig 3 in the paper. So, the authors are encouraged to analyze and discuss the impact of randomly selected few-shot instances. - The original TCGA-RCC is not annotated at the pixel level. This work does a good job in terms of annotating the fine-grained region-of-interest of WSIs. However, if the annotation is not made public, this work would be difficult to follow and be very limited in research impact. The authors are encouraged to make their annotated dataset public. I would like to raise my score if the authors could resolve my concerns & questions above. ---------------------------------------------After Rebuttal------------------------------------------------- My main concerns have been addressed. I am happy to increase my score.

Rating

5

Confidence

5

Soundness

2

Presentation

1

Contribution

2

Limitations

There is no explicit limitation that should be included in the paper.

Summary

To address the challenges of expensive fine-grained annotation and data scarcity encountered in the clinical application of deep learning-based WSI classification methods, this paper proposes a novel and efficient dual-tier few-shot learning paradigm named FAST. Under this new paradigm, the authors introduce a dual-level annotation strategy that includes bag-level few-shot and instance-level few-shot, modeling the WSI classification problem as a new few-shot classification problem. Building on this, the authors further propose a classification framework composed of a learnable image cache branch and a CLIP prior knowledge branch, fully leveraging the value of the limited data and labels. Extensive experimental results show significant improvement over other few shot methods in both binary and multi-class classification tasks. Interestingly, the proposed method FAST achieves performance close to fully supervised methods with only 0.22% of the annotation cost. This showcases its efficiency and great potential for practical applications.

Strengths

1. The paper is well written and easy to read. The authors intuitively demonstrate their methods and contributions through numerous figures and tables. Extensive comparative and ablation experiments illustrate the efficiency and generality of the proposed method. To ensure fairness and prevent randomness, the authors conducted multiple random experiments in their study. The results show significantly better performance in both bag-level and instance-level classification compared to other methods. 2. The proposed dual-level WSI annotation strategy is a highly innovative and suitable method for WSI data annotation. It addresses the issues of single-level annotation and provides patch-level supervisory information at a cost close to slide-level annotation. Compared to fully supervised methods, the proposed method has significantly lower annotation costs, astonishingly reaching as low as one-thousandth or even one-ten-thousandth. 3. This paper is inspired by Tip-adapter and proposes a learnable cache branch where both labels and image features are learnable. The final classification framework further integrates a CLIP prior knowledge branch incorporating GPT-4V. Comparative experiments show that this method achieves performance close to fully supervised methods with only 0.2% of the data annotation cost, which is an exciting advancement. Ablation experiments also demonstrate the importance of the proposed components.

Weaknesses

1. The function ϕ(∙) in Figure 2 is not mentioned or explained in the paper, which may confuse readers. 2. The authors conducted extensive comparisons in terms of accuracy and annotation cost but lacked analysis of time and memory consumption. 3. In section 3.2, the authors mention obtaining the optimal fusion weight α through grid search but lack specific details in the paper.

Questions

Will the authors open source the relevant code and all model weights for this project? For other issues, please refer to the weaknesses.

Rating

8

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

In the conclusion section, the authors acknowledge the limitations of their proposed method. I agree that such limitations exist and look forward to future work.

Summary

In this article, the authors propose a novel few-shot learning paradigm for WSI classification. This paradigm is based on two branches: the first is a learnable cache model that utilizes both labeled and unlabeled instance data, and the second, the Prior Branch, leverages the prior knowledge of a pre-trained CLIP model. By combining these two branches, efficient few-shot learning is achieved, and extensive experiments have been conducted on the CAMELYON16 dataset and the TCGA-RENAL dataset.

Strengths

1. Few-shot learning is inherently important in the field of WSI classification. The authors have proposed a new few-shot learning paradigm tailored for WSI classification and have achieved notable results. 2. The experiments on few-shot learning are quite comprehensive, thoroughly comparing the effects of different numbers of instances and bags.

Weaknesses

1. The study lacks experiments with V-L models specific to the pathology field. Since CLIP is not originally based on pathology images, the authors should include comparisons using PLIP [1] and CONCH [2]. 2. The few-shot learning method proposed by the authors operates at both the instance-level and bag-level. Therefore, the comparative methods should include both instance-based methods and bag-level methods (multi-instance learning). However, the fully supervised methods chosen for comparison are only instance-based. The authors should supplement their comparisons with bag-level methods based on multi-instance learning, such as R2T [3]. 3. Comparing the third and fourth rows in Table 3 of the paper reveals that adding the Prior Branch on top of existing components brings almost no improvement. However, it requires first processing through GPT and then the Text-encoder, significantly increasing the cost without enhancing performance. [1] PLIP: A visual–language foundation model for pathology image analysis using medical Twitter. Nature Medicine 2023 [2] CONCH：A Vision-Language Foundation Model for Computational Pathology. Nature Medicine 2024 [3] Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology. CVPR 2024

Questions

What the difference between the proposed work and previous few-shot WSI classification methods like [4]? [4] The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. NeurIPS 2023

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors only tested up to 16 instances and bags, but there is still a significant performance improvement from 8 to 16. I am curious at what data ratio in few-shot learning the model will begin to overfit.

Summary

This paper investigates the issue of Whole Slide Images (WSI) classification, a study with practical value. It proposes a new working paradigm that is an improvement based on Tip-Adapter. Theoretically, this new paradigm can effectively address the problem and has strong scalability.

Strengths

This study has practical significance, and the proposed method demonstrates strong scalability. The paper is clearly written and easy to understand, with comprehensive experiments.

Weaknesses

1. This paper lacks some important related work. The proposed method is based on the Tip-Adapter. While, there are many improvements based on Tip-Adapter, such as [1-4]. I think the experiments should include comparisons with these related methods, or at the very least, mention and briefly analyze them. [1] Collaborative Consortium of Foundation Models for Open-World Few-Shot Learning. AAAI, 2024. [2] Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners. CVPR, 2023. [3] DeIL: Direct-and-Inverse CLIP for Open-World Few-Shot Learning. CVPR, 2024. [4] Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement. ICCV, 2023. 2. In Figure 1, the blue and red boxes should be explained. Additionally, if space permits, I suggest that in future work, the authors could add more detailed descriptions in the caption. By this way, readers can understand the general idea of the method just by looking at the figure and caption, without having to spend effort finding the corresponding description in the main text. 3. For instance-shot, only the results of 16-shot are shown, without the results of 1-shot, 2-shot, etc. 4. Line 160 seems to have a typo; it should be y_1L instead of y_1V.

Questions

see weakness

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

authors didn't present the limitation.

Thank you for your rebuttal, which clearly addressed my concerns.

Thank you for your rebuttal, which clearly addressed my concerns. I have read other reviewers' comments and the author's rebuttal, particularly the comparison with vision-language-based models in computational pathology. The experimental results and the authors’ responses clearly and effectively demonstrate the contribution of this paper. Overall, the authors' rebuttal resolves my concerns, and their answers to other reviewers' questions also seem reasonable to me. I think this paper is highly valuable for advancing computational pathology. Thus, I increase my rating to strong accept.

Response to the Authors

Thanks for the authors' responses. The author has resolved my questions and I agree to accept this paper.