The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification

Summary

The paper introduces a new approach called Few-Shot Weakly Supervised Learning for Pathology Whole Slide Image classification (FSWC), which aims to classify bags and instances within a WSI with only a limited number of labeled bags. The proposed solution utilizes a large language model, GPT-4, and prompt learning. The approach leverages CLIP to extract instance features for each patch and uses a prompt-guided pooling strategy to aggregate these instance features into a bag feature. The language prior knowledge is obtained using GPT-4 in a question-and-answer mode at both the instance and bag levels. The method is evaluated on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, and demonstrates notable performance in bag and instance classification.

Strengths

The idea is interesting, and is one of the early works trying to apply large language and cross-modal models to WSI (Whole Slide Imaging) image analysis problems. Due to the knowledge learned wit the prompt, requirement on large number of traing samples can thus be alleviated.

Weaknesses

While the idea is interesting, there are significant problems in presentation and languages of the paper. Lots of technical details are not clearly explained. The experiments are not intensive as well, more sota methods need to be compared. Pls see the section of questions for details. After reading other reviewer's comments and the rebuttal, I think the paper reauires significant works to improve it, and currently it's not ready to be included in NeurIPS

Questions

1.Unlike general few-shot scenarios，what does shot represent in medical images ? And how is the dataset divided ? 2.In one part of the instance-level prompt, various instance phenotypes are generated. For different datasets, whether the generated functional descriptions are the same or not, without professional knowledge, how to determine these functional descriptions ? 3.What is learnable in prompt learning ? 4.The description of methodology is not clear enough. For example, how does the guided pooling work ? 5.More comparative experiments based on MIL should be added. 6.The reviewer would suggest the author to proofread the manuscript and the quality of the writing and the presentation should be significantly improved.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

n/a

Summary

This work proposes TOP - a framework using Vision-Language models for few shot Multiple Instance Learning on pathology datasets. TOP uses features generated by passing natural language prompts through a LLM to guide the pooling process for the bag level aggregation in MIL. Results show improvements over other few shot MIL approaches in 3 pathology datasets.

Strengths

- Few shot learning for MIL in histopathology is important since the effective number of data points is greatly reduced when predicting at WSI-level instead of patch-level. This paper thus addresses this highly relevant problem by providing bag level predictions with only a few training bags. - The prompt-guided pooling idea, where similarity of natural language embeddings of text descriptions of different kinds of pathology to image embeddings is used to ground and calculate the instance-level weights, makes a lot of intuitive sense.

Weaknesses

- My main concern pertains to section 3 in supplementary material, where it is mentioned that "..we randomly trained the network five times with different labeled bags and reported the highest performance of each method..". This implies that the performance comparison is biased towards TOP. - There can be additional ablations around using pathology specific LLMs [1] instead of generic ones to check if this improves the large variance in performance for TOP across lower number of shots - The motivation behind adding the correlation loss between the instance prototypes can be clarified further. Also, details around the sensitivity of the model performance to the relative weight of this loss should be shared. [1] - Santos et al, PathologyBERT - Pre-trained Vs. A New Transformer Language Model for Pathology Domain

Questions

- For comparison on instance-level predictions, [1] uses the Additive MIL framework to get exact instance-level classwise predictions. Can this method be added to the comparison? - Any specific reason for restricting the learnable params to 10 tokens for instance and bag prompts?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

- Can the authors give a ballpark estimate of the cost associated with using LLMs for the description generation as a function of the number of prototypes?

Summary

The authors introduce a two-level prompt learning framework for label-efficient classification of WSIs using pretrained visual language encoders. The main novelty of the proposed method seems to be that at the instance-level, domain-knowledge + GPT-4 guided prompt prototype groups are used to guide pooling of instance embeddings into the slide-level embedding and while the use of learnable prefix tokens + pretrained text embeddings for few-shot classification follows closely of the established V+L few-shot literature. Experimentation was performed on several WSI datasets (CAMELYON16, TCGA, and an in-house cervical cancer dataset).

Strengths

- This work explores several interesting ideas regarding instance- and bag-level prompts. Previous and concurrent works (e.g. - MI-Zero, PLIP [1,2]) have only explored the application of "instance-level" prompts (text prompts correlated with pathology region-of-interests) for computational pathology. The application of using bag-level prompts in combination of with GPT-4 provides new mechanisms for training and evaluating models for slide-level tasks in pathology. 1. Lu et al. 2023, Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images. CVPR 2023. 2. Huang et al. 2023, Leveraging medical Twitter to build a visual–language foundation model for pathology AI. bioRxiv 2023.

Weaknesses

While the study presents interesting ideas which the reviewer believes to be valuable to the machine learning and computational pathology community. the reviewer has several concerns regarding the evaluation framework and ablation studies. 1. In the methodological details of this work, because the results in the few-shot settings can be highly variable, therefore for each shot, 5 randomly sampled sets of training bags are used, and the highest performance for each model is reported. The reviewer find this practice somewhat questionable given that stability/consistency of the algorithm should be an important factor in deciding which algorithms works well in the few-shot setting. By only reporting the highest performing run on the test set, we can have the scenario where Algorithm 1 obtains consistently reasonable, above chance performance (e.g. [0.7, 0.7, 0.7, 0.7, 0.7]) while Algorithm 2 obtains performance below chance but a single lucky run (e.g. [0.4, 0.4, 0.4, 0.4, 0.75]) and the conclusion will be that Algorithm 2 outperforms Algorithm 1 by a wide margin of 5% - which is not a reasonable conclusion. In the few-shot setting reported in the paper, if the SD of the 5 runs can be as high as 10% - how can the reviewer be certain the proposed model in fact does outperform the other methods consistently when only the highest performing run is reported? I would encourage this work to instead use the median for comparing the different models instead of the max, which is more robust to outliers compared to the mean. Additionally, the exact numbers of all 5-runs should be reported as a box plot in the supplement, and other metrics besides AUC (e.g. balanced accuracy) should be reported as well. 2. This work does not make comprehensive comparison to other methods/in the ablation experiments. An important concern being that the proposed method uses both learnable prefix tokens at the instance level and at the slide-level compared to CoOP, therefore effectively doubling the number of learnable parameters. (a) What happens when the proposed method uses the same number of learnable parameters as CoOP? (b) What happens if we cut the number of prefix tokens in half at both the slide-level and the instance-level relative to CoOP such that the total parameter count is the same? 3. Some design choices in the proposed method do not seem to be properly ablated, for instance. During training, an auxiliary loss (equation 9) is used to encourage de-correlation of different prompt prototypes, but the effectiveness of this choice does not seem to be ablated.

Questions

Summarizing the concerns above: 1. What is the performance when using the median versus the max? 2. Effect of reducing the # of prefix tokens so that the total # of parameters is the same? 3. Ablation study concerning the auxiliary loss?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

3 good

Contribution

3 good

Limitations

- Limitations discussed in this work include the effectiveness of the prompt depending on the quality of visual representations.

Summary

This paper introduces an innovative problem setting known as Few-shot Weakly Supervised WSI Classification (FSWC), wherein the availability of labeled WSIs is severely limited. In order to address this challenge, the authors propose a novel Two-level Prompt Learning MIL framework, named TOP, which leverages the power of the VL model and GPT-4 to enhance the learning process of the model.

Strengths

1. This paper demonstrates exceptional writing quality, distinguished by its clear and well-organized structure. The authors adeptly utilize intuitive figures and well-crafted paragraphs to effectively summarize their methods and contributions. Furthermore, the inclusion of extensive experiments and detailed ablation studies serves as a robust validation for their proposed method. 2. The proposed instance prompt guided pooling is a captivating and fitting approach for the vision language model task. It exhibits an intriguing method that holds potential for improving the performance of the model in this context.

Weaknesses

1. The problem addressed in this paper is not entirely novel within the field of WSIs, as previous studies like MI-Zero[1] have also recognized the challenge of limited labeled WSIs. 2. The utilization of pathology language prior knowledge derived from GPT-4 is indeed an interesting aspect of this paper. However, it is essential to address the concerns and questions raised regarding the validity and importance of this knowledge in the task at hand. To assess the correctness and relevance of the knowledge obtained from GPT-4, the paper should ideally outline an evaluation method or standards based on the output of GPT-4. This would help establish the reliability and usefulness of the information derived from the model. Additionally, the authors should clarify their rationale for choosing GPT-4's knowledge over manually designed and professionally validated visual descriptions. It would be beneficial to discuss how they compared the performance and efficacy of the two approaches and why GPT-4-based knowledge was deemed more appropriate. Regarding the generation of questions for GPT-4, it is crucial to provide insights into the process of selecting and designing these questions. The paper should clarify whether there were any specific criteria or standards used to ensure the validity and reasonableness of the questions. Addressing potential biases introduced by the question selection process is also important. Considering that GPT-4 may undergo updates over time, it is essential to acknowledge the potential impact on the answers generated by the model. The authors should discuss the implications of model updates on the overall performance and reproducibility of their proposed framework. In summary, addressing these questions and providing more detailed explanations would enhance the clarity and credibility of the paper, particularly concerning the validity and reliability of the pathology language prior knowledge derived from GPT-4. 3. The technical novelty of the paper is acknowledged to be limited. Although the paper introduces the instance prompt guided pooling method, which is derived from prototype learning and presents some novelty, it may not be considered sufficient to significantly differentiate it from existing approaches. Furthermore, it appears that the use of a Vision Language (VL) model to extract features and the utilization of learnable prompts from CoOp are commonly employed techniques within the field[1]. These aspects may not contribute significantly to the novelty of the proposed framework. Considering these limitations, it would be valuable for the authors to address the relative novelty of their contributions more explicitly. It could be helpful to discuss how their approach builds upon or improves existing methods, and to identify the specific aspects that differentiate their work from prior research in the field. 4. I have a concern regarding the comparison and reporting of results in the paper. Randomly training the network multiple times with different labeled bags and reporting the highest performance of each method may introduce some ambiguity and make the results appear misleading or tricky. To ensure a more robust and fair comparison, it would be preferable to adopt a standardized evaluation methodology. This could involve training and testing the network multiple times using consistent sets of labeled bags for each method, and then reporting the average performance across these trials. This approach would provide a more reliable representation of the methods' benefits and allow for a more accurate comparison between them. By employing a standardized evaluation methodology, the authors would enhance the credibility and trustworthiness of their reported results, addressing concerns about potential biases or inconsistencies introduced by the current approach.

Questions

It would be greatly appreciated if the author could consider making the code, trained models, and specific train-validation-test data splits publicly available for the associated public datasets used in the different methods. This step is crucial for ensuring the reproducibility of the results. Access to these resources would significantly facilitate the verification and replication of the finding. Other questions please refer to the weekness part.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Please refer to the weekness part.

Response to "Comparison to MI-Zero" (Part 1/2)

Thanks for your comments. We sincerely appreciate your recognition of the innovations in our work, which greatly encourages us. We are happy to take this opportunity to further clarify the relationship between our method and MI-Zero. We will first explain why there is not a direct comparison between our method and MI-Zero and then give some quantitative comparison between the performance of these two methods. Finally, we will explain how to combine our method with MI-Zero. **1) We do not make a direct comparison with MI-Zero for the following two reasons:** **a) Our method is orthogonal to MI-Zero, posing a challenge for direct comparison.** Though both our method and MI-Zero are related to large vision-language models, the objectives of the two methods are different. The objective of MI-Zero is to develop a large vision-language model in a specific domain (digital pathology domain), while the objective of our method is to develop new prompt-learning strategies based on an existing large vision-language model for few-shot learning. Our method can be combined with any large vision-language models (including the model developed in MI-Zero) for few-shot WSI classification, not limited to the CLIP model used in the paper. Like many studies on developing large vision-language models, MI-Zero focuses on the model’s performance on zero-shot learning and they used the developed model to achieve slide-level classification through simple instance-level prompts and top-k aggregation (without a parameter learning process). In contrast, our research focuses on designing instance-level and bag-level prompt learning to better utilize existing large vision-language models (such as CLIP) to accomplish few-shot learning with minimal annotations. No few-shot learning experiments are conducted in the MI-Zero paper. The MI-Zero model cannot perform few-shot learning by itself and cannot directly compare with our method under the few-shot experimental setting. **b) Technically, our method can use any large vision-language models including MI-Zero as backbones, but MI-Zero is released after the submission of this paper.** At the time of our manuscript completion (May), there were no available large vision-language models in the field of digital pathology, so we employed the well-known model CLIP in the natural image domain for our experiments. MI-Zero released their paper and model after June (they only released the model trained on public datasets, while the model trained on both public and private datasets mentioned in the original paper has not been made public yet). This makes it difficult for us to use the model trained by MI-Zero as our backbone. Nevertheless, our proposed two-level prompt learning approach can still be easily integrated with the MI-Zero model for few-shot learning and we are performing this experiment now. This will be further discussed in the last part of this response. Combining our method with a domain-specific model like MI-Zero has the potential of achieving higher performance than combining our method with the general model CLIP, but no matter what the performance is, we think the result will not weaken our contribution in this paper.

Mean AUC and STD on the TCGA and Cervical Cancer datasets

As promised in our first round of response, we conducted further experiments and report the results of five runs on *the TCGA dataset* and *the Cervical Cancer dataset* using non-fixed bags, reporting Mean (left panel) and STD (right panel) in Table 1 (*the TCGA dataset*) and Table 2 (*the Cervical Cancer dataset*). We also supplemented five rounds of training using fixed labeled bags on *the TCGA dataset* and *the Cervical Cancer dataset*, employing the same bag in each run for training. Mean (left panel) and STD (right panel) are presented in Table 3 (*the TCGA dataset*) and Table 4 (*the Cervical Cancer dataset*). We can draw similar conclusions as those obtained on the Camelyon16 dataset, and our approach demonstrated the best performance across all datasets. All of these results will be included in the camera-ready version. Table 1: Mean bag AUC using **non-fixed labeled bags** on *the TCGA dataset* |Method|16-shot|8-shot|4-shot|2-shot|1-shot|16-shot|8-shot|4-shot|2-shot|1-shot| |:-------------------------------------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:| |Linear-Probe (Mean-pooling)| 0.5899| 0.5126| 0.4843| 0.4645| 0.4527| 0.0451| 0.0263| 0.0282| 0.0217| 0.0265| |Linear-Probe (Max-pooling)| 0.5951| 0.5245| 0.5037| 0.4731| 0.4615| 0.0512| 0.0563| 0.0647| 0.0365| 0.0174| |Linear-Probe (Attention-pooling)| 0.6868| 0.6223| 0.5954| 0.5569| 0.5524| 0.0279| 0.0516| 0.0699| 0.0492| 0.0596| |CoOp (Attention-pooling)| 0.7545| 0.6700| 0.6662| 0.6416| 0.6306| 0.0280| 0.0512| 0.0636| 0.0751| 0.0860| | Bag Prompt + Prompt guided pooling (**Ours**) |**0.8039**|**0.7685**|**0.7279**|**0.6984**|**0.6879**|0.0265|0.0465|0.0641|0.0743|0.0852| Table 2: Mean bag AUC using **non-fixed labeled bags** on *the Cervical Cancer dataset* |Method|16-shot| 8-shot| 4-shot|2-shot|1-shot|16-shot|8-shot|4-shot|2-shot|1-shot| |:-------------------------------------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:| |Linear-Probe (Mean-pooling)| 0.6514 | 0.6348 | 0.5839 | 0.5740 | 0.5615 | 0.0556 | 0.0696 | 0.0781 | 0.0670 | 0.0634 | |Linear-Probe (Max-pooling)| 0.6003 | 0.5889 | 0.5593 | 0.5525 | 0.5343 | 0.0597 | 0.0635 | 0.0660 | 0.0612 | 0.0436 | |Linear-Probe (Attention-pooling)| 0.7052 | 0.6690 | 0.6524 | 0.6496 | 0.5178 | 0.0364 | 0.0624 | 0.0814 | 0.0871 | 0.0564 | |CoOp (Attention-pooling)| 0.7041 | 0.6606 | 0.6568 | 0.6417 | 0.6049 | 0.0303 | 0.0658 | 0.0775 | 0.0855 | 0.0931 | |Bag Prompt + Prompt guided pooling (**Ours**) | **0.8080** | **0.7855** | **0.7468** | **0.7123** | **0.6652** | 0.0296 | 0.0549 | 0.0733 | 0.0806 | 0.0957 | Table 3: Mean bag AUC using **fixed labeled bags** on *the TCGA dataset* | Method | 16-shot | 8-shot | 4-shot | 2-shot | 1-shot | 16-shot | 8-shot | 4-shot | 2-shot | 1-shot | |:-------------------------------------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:| | Linear-Probe (Mean-pooling) | 0.6022 | 0.5418 | 0.4934 | 0.4908 | 0.4646 | 0.0067 | 0.0072 | 0.0084 | 0.0092 | 0.0126 | | Linear-Probe (Max-pooling) | 0.6227 | 0.5547 | 0.5155 | 0.4985 | 0.4876 | 0.0065 | 0.0072 | 0.0089 | 0.0090 | 0.0137 | | Linear-Probe (Attention-pooling) | 0.7178 | 0.6539 | 0.6248 | 0.5832 | 0.5713 | 0.0067 | 0.0071 | 0.0072 | 0.0082 | 0.0107 | | CoOp (Attention-pooling) | 0.7840 | 0.6824 | 0.6811 | 0.6772 | 0.6801 | 0.0064 | 0.0065 | 0.0084 | 0.0095 | 0.0105 | | Bag Prompt + Prompt guided pooling (**Ours**) | **0.8235** | **0.8059** | **0.7531** | **0.7245** | **0.7123** | 0.0059 | 0.0062 | 0.0083 | 0.0092 | 0.0103 | Table 4: Mean bag AUC using **fixed labeled bags** on *the Cervical Cancer dataset* |Method| 16-shot | 8-shot | 4-shot | 2-shot | 1-shot | 16-shot | 8-shot | 4-shot | 2-shot | 1-shot | |:-------------------------------------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:| | Linear-Probe (Mean-pooling) | 0.6756 | 0.6684 | 0.6593 | 0.6246 | 0.6011 | 0.0098 | 0.0105 | 0.0104 | 0.0113 | 0.0147 | | Linear-Probe (Max-pooling) | 0.6322 | 0.6249 | 0.6038 | 0.5884 | 0.5869 | 0.0103 | 0.1120 | 0.0106 | 0.0114 | 0.0124 | | Linear-Probe (Attention-pooling) | 0.7345 | 0.7282 | 0.7155 | 0.6873 | 0.6137 | 0.0095 | 0.0105 | 0.0108 | 0.0169 | 0.0155 | | CoOp (Attention-pooling) | 0.7565 | 0.7349 | 0.7271 | 0.6927 | 0.6484 | 0.0091 | 0.0095 | 0.0097 | 0.0115 | 0.0117 | | Bag Prompt + Prompt guided pooling (**Ours**) | **0.8189** | **0.8007** | **0.7869** | **0.7618** | **0.7052** | 0.0088 | 0.0089 | 0.0088 | 0.0104 | 0.0101 |