Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement

Summary

The paper proposes a way to enhance image features from VLM pathology foundation models for downstream tasks by aligning them with task-specific text prompts. The authors use two modules -- one which extracts an image representation aligned with task concepts and other which measures similarity of representation with tasks, to enrich image representations for downstream tasks. The authors show results for the cancer subtyping task across 3 datasets indicating the usefulness of the proposed approach.

Strengths

**Clarity and Quality**: The paper is well written and the key ideas are explained clearly. The experiments are well setup, the results and ablations provide good evidence of the proposed method for the specific problems and the limitations are discussed. The code is shared and the supplementary section provides more details around ablations and the theoretical assumptions. **Originality** There has been prior work in using task anchors to improve MIL using learnt or clustered task prototypes (ProtoMIL, TPMIL, etc) The idea of using textual prompts to create task anchors and use these to filter patches and enrich patch representations is interesting. The predictive information maximization (PIM) objective is interesting and is similar to applying CLIP on-the-fly with fixed text embeddings, to extract more task-specific image features. The SIM objective seems useful to suppress task-irrelevant information. **Significance** The proposed approach is a simple addition to MIL and can be applied in conjunction with existing MIL methods.

Weaknesses

The main weaknesses are around some of the assumptions made in the paper listed below which limit the applicability of the approach. **Aligning patches with tasks** In WSIs the labels are only available at slide-level and not patch-level and for many problems only a fraction of the patches in the slide have task-relevant info and for many others the information may need to be aggregated from multiple patches. Even though there is a filtering step to filter patches, the step of aligning patches to tasks is noisy when many patches dont have task info and it may not even be appropriate for problems which need aggregation of information across patches (as the approach homogenizes representations of all patches belonging to the same class). The authors show results on only one type of task -- subtyping where most patches on the slide have the sub-type information and is also typically a simpler MIL task indicated by the strong results on the benchmarks for most methods. Instead of aligning patch representations, it would be better to align slide-level representations with task-concepts, which addresses both the drawbacks listed above. There is also evidence from prior works around usefulness of this like SC-MIL (https://arxiv.org/abs/2303.13405 which align representations of slides belonging to same-class) and they also show aligning patch-level representations (patch SC-MIL) does poorly compared to aligning at the slide-level **Unclear Improvements** It is unclear how useful the task-alignment objective with the information bottleneck layer is. The key part seems like the filtering step which extracts relevant patches for the task and I suspect if thats contributing to majority of the improvements. Once filtering is done, is the alignment step really needed as it seems like a dimensionality reduction step to extract task specific features. A sufficiently strong model should be able to able to extract task relevant features from the embedding dimensions with cross-entropy loss. The alignment step might also be unsuitable as it homogenizes patch representations which may not be good for many MIL tasks and directly optimizing with cross entropy on task-labels might be more flexible and general. Its also unclear if the improvements are coming from a bigger model due to the CIB and CFI modules. Scaling ABMIL to similar number of params would provide more context on this. **Task Concept Alignment** The other key assumption is using text prompt embeddings as task anchors which is very dependent on factors like the quality of the pre-trained VLM, the textual prompts and descriptions used and the idea that it is possible to write morphological descriptors for tasks which is typically not possible for many MIL tasks like survival or molecular signature prediction. The sufficiency and consistency constraints imposed could be limiting specially when any of the factors above break. The SIM objective specifically could be problematic when text doesnt fully capture the required features for the task and other relevant image features could be suppressed.

Questions

Its unclear how much improvements are coming from the patch filtering step and the increased number of parameters and if those alone are sufficient enough. I think its worth trying the alignment on the slide-level representations instead of patch-level representations (similar to SC-MIL) which makes the approach more generic and limits the drawbacks with using slide-level labels for patches.

Rating

5

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

Highlighted the limitations in weaknesses. Provided some suggestions and improvements in the questions. Given the weaknesses and the lack of evidence and limited applicability of the approach to more complex MIL problems like scoring (gleason grading, HER2 grading) which need aggregation of information across patches or other analytical tasks like survival prediction, molecular signature prediction where there might not be possible to write text descriptions, I am basing my decision

Summary

The authors introduce CATE, a novel approach designed to enhance the generalizability of histopathology models by leveraging task-specific concepts derived from the text endoder of pathology vision language models(VLM). CATE includes two modules: CIB and CIF, which work synergistically to improve model robustness. The CIB module identifies and retains features that contribute positively to predicting the task outcome, while it eliminates features that contain irrelevant or superfluous information. Additionally, CIF specializes in creating task-specific features by leveraging similarities between the enhanced image features the and predefined concept anchors. The authors demonstrate the effectiveness of CATE across an extensive series of histopathology datasets, showcasing superior results in model performance and generalization.

Strengths

- The concept introduced by CATE is highly relevant as it addresses a major challenge in histopathology: ensuring models generalize effectively across diverse datasets and clinical scenarios. In computational pathology, the lack of external validation cohorts often leads to uncertainty about whether models have learned genuine signals or have simply overfitted to confounding factors or batch effects. To validate their approach, the authors split existing datasets into indoor and outdoor test sets, demonstrating CATE's robustness across different conditions. - The paper is well-presented and clearly written. - The strength of the paper lies in its innovative approach of repurposing general-purpose vision language models to extract rich, meaningful representations for downsream tasks.

Weaknesses

- CATE's performance hinges largely on the quality of its concept anchors, which in turn depends on domain expertise and the quality of the pre-trained pathology VLM, a weakness that is also highlighted by the authors.

Questions

- Why did you use accuracy as an evaluation metric? Accuracy is straightforward to interpret, but it can be misleading in imbalanced datasets. Metrics like F1-score, weighted accuracy, precision, and recall offer more nuanced insights into model performance, particularly in contexts where class distribution is skewed such as the BRCA dataset. - What strategy was used to split the dataset into in-domain and out-of-domain test sets? Understanding this process ensures transparency in evaluating the model's generalizability across different clinical settings or data sources. - How do you define the class-agnostic concepts? - What is the intuition behind maziming the distance between the enhanced featrures and the class-agnostic concepts? When features are overly tailored to the training samples by maximizing their separation from class-agnostic concepts, the model may become too specialized and prone to overfitting.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

- Exploring the potential of concept anchors for applications beyond subtyping, such as mutation predictio could be an intersting avenue to explore.

Summary

The authors introduce a new tool called Concept Anchor-guided Task-specific Feature Enhancement (CATE) for analysis of whole slide images. The authors are taking advantage of the open source weight now available for the CONCH pathology vision language model trained on paired captions of images from journals and other publicly available image-caption pairs. The high level conceptulization is that language prompts related to a task at the whole slide image level can be enrich for the features most relevant to the downstream task. The title cleverly asserts this is a free lunch when using a vision language encoder for feature extraction the expertly curated concepts can "bring foward" the most relevant features when aggregating at the whole slide level. Interestingly, they split up the CATE task into Predictive Information Maximization to enhance features associated with related concepts and Superfluous Information Minimization is used to supress the concepts likely to be associated with unimportant features. The outputs of PIM and SIM are used during aggregation task to determine weighting that MIL should use for a given feature vector derived from the tile. Finally a module called concept-feature interference is used calibrate the CIB scores. These module provide enhanced features to the aggregation function that provides relative weighting based on the relationship between the semantic concepts relationship to the downstream task. CATE and weights are not backpropogated through the feature embedding tasks and they are static in the MIL function (to the best of my understanding). The authors chose RCC, NSCLC, and BRCA tasks from the TCGA to demonstrate performance

Strengths

The model is build such that one can introduce to any MIL-like framework that uses a VLM encoder. It is a clever idea to guide the aggregation function with the concepts one can get from VLM encoder. The paper is clearly written and claims are not over stated. On the chosen tasks, it is reasonably clear that a performance boost is acheived consistently and in some cases by a dramatic margin. Given that is is a bolt-on method these are very good apples to apples comparisons. The ablations study suggests that the 3 novel modules (PIM, SIM, and CFI and parts are additive to peformance, at least on these tasks.

Weaknesses

I agree with the weaknesses the authors have put forth. The model scope is ~mostly~ limited to downstream task for the the concepts are part of the VLM training. The tasks that are chosen for benchmarking are actually tasks that are likely well represented in VLM training sets with robust descriptions of SCC vs. LUAD, etc. For tasks that source material does not contain common descriptions, say more challenging tasks like predicting mutations, it is unclear to what extent, this work would provide advantage over other approaches. My presumption is that VLM encoders are less robust for feature extraction than the now available large foundation models trained with 10^6 plus whole slide images. If one is not able to use these models and must use an inferior model, then one might argue it is not a "free lunch".

Questions

Out of distribution (OOD) is a term that means many different things to different people. Will you please spare a paragraph to better explain what you mean by OOD. If you find that OOD is not the best term for what you mean, I think it would be better to use a less loaded term. Have you considered a hybrid approach where the vision feature vectors can be extracted with a different foundation model but the VLM is use for the concept guidance? How the normal distribution of SIM in equation (8) align with Bernoulli distribution of ABMIL? Are there alternative distributions that could be used here? (may be Poisson distribution to define random occurance of a non-informative patch?) Minor comments for improvement in readability: 1) In figure 2a, the image encoder cartoon is larger on the side of image which implies the feature vectors are larger than the original patches. 2) Equation (6) LHS should be I(x; alpha | c)

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations section is reasonable. I have provided further feedback in the weaknesses section.

Summary

The authors propose an approach to enhance the performance of Multiple Instance Learning (MIL) models by incorporating concept prompts within the context of pathological Vision-Language Models (VLMs). Two modules, i.e., the Concept-guided Information Bottleneck (CIB) module and the Concept-Feature Interference (CFI) module, are designed to calibrate and inject similarity characteristics between features and concepts. Experiments are conducted on three Whole Slide Image (WSI) datasets, yielding interesting results. The paper includes qualitative and ablation analyses.

Strengths

Technically Sound Method: The proposed approach demonstrates a well-founded methodology, leveraging concept prompts within pathological Vision-Language Models for enhancing Multiple Instance Learning models. Well-Written Paper: The paper is clearly and coherently written, providing a logical flow and structure that makes the complex ideas accessible and easy to follow. Well-Illustrated Figures: The figures are well-designed and effectively illustrate key modules, significantly aiding in the understanding of the methodology.

Weaknesses

Dependence on Text Prompts: The performance heavily relies on the quality of text prompts/concepts. A more principled approach to generating expert-designed prompts would be beneficial. Only the hard-crafted ones make the "free lunch" not that free. Experimental Settings: The choice of the number of IND and OOD sites for different datasets is unclear. How to choose the number of IND and OOD sites? It would be helpful to see performance under more traditional settings on NSCLC and RCC. What's more, it is noticed that the gain of CATE decreases with an increasing number of sites, raising questions about the practicality of the main experimental settings in real-world WSI analysis. Attention Maps: The attention maps are not that convincing to me. Since CATE-MIL provides almost identical attention maps to the baseline ABMIL except for intensity. If ABMIL predicts incorrect attention, CATE-MIL could potentially worsen the situation. Underperformance on NSCLC and RCC: The authors claim that the underperformance of CATE-MIL on NSCLC and RCC due to the elimination of site-specific patterns is concerning. Clarification on why site-specific information for the IID test is not considered task-relevant is needed, especially as this issue does not appear with BRCA.

Questions

The paper is technically sound, but I am not convinced by the provided experimental results. 1) Could the authors suggest more principled ways to obtain expert-designed prompts? Relying on hand-crafted prompts may not be scalable or generalizable. 2) Please refer to the weakness and clarify the experimental settings. More detailed qualitative and quantitative analyses are needed. 3) Lack of comparison with ref [1], which also uses the Information Bottleneck theory to improve the feature quality of MIL. [1] Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classification. CVPR 2023.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

N/A