LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching

Summary

This research paper focuses on LVM-Med, a self-supervised learning (SSL) technique designed for medical imaging tasks. Based on a second-order graph matching strategy, LVM-Med is trained on a large-scale medical imaging dataset. The researchers found that the method significantly improves performance on a variety of downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on large quantities of image-text data. These findings were consistent across two different architectures: ResNet-50 and Vision Transformer (ViT).

Strengths

- The LVM-Med method was evaluated on various tasks, including segmentation, object detection, and image classification. The results were compared to foundational models like Clip, Align, Flava, and SAM. It performed particularly well on eight medical segmentation tasks, outperforming both 2D SSL methods trained on the same dataset and foundational models. - The training of LVM-Med on a large-scale medical imaging dataset indicates its capacity to handle and learn from large amounts of data, which is often a requirement in the field of medical imaging. - The LVM-Med model outperforms both supervised learning methods and foundation models trained on hundreds of millions of image-text instances. This includes a variety of popular models such as Clip, Align, Flava, and SAM. - The results show that LVM-Med performs well in in-out-distribution settings, implying a certain level of robustness and generalizability. - LVM-Med provides benefits in both end-to-end and prompt-based segmentation tasks. This flexibility can be particularly useful in real-world applications, where various segmentation scenarios may be encountered. - The authors also conducted an ablation study, experimenting with variations in LVM-Med's configuration to assess the importance of various components in the overall performance. It was concluded that all factors contribute to the final performance, with the second-order graph matching and Gumbel noise being the most significant.

Weaknesses

- Only 2D backbone is conducted: The LVM-Med model focuses primarily on 2D backbone. Author should provide more discussion about the challenge for applying this framework in 3D backbone. - The LVM-Med model is trained on a large-scale dataset, how can author endure the testing dataset is not leaky in training set. Moreover, the use of a single dataset could potentially limit its generalizability. Ensuring robust performance across diverse datasets from different sources is critical. - The model's performance benefits seem to be tied to its training on a large-scale medical imaging dataset. If such a dataset is not available, the performance of the model might be significantly diminished.

Questions

Please address the comment in weakness part. In addition, though the ViT architecture has more total parameters, in some cases, it is less effective than LVM-Med ResNet-50. More research and discussion would be needed to fully understand this performance discrepancy.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No limitation on limitations and broader societal impacts.

Summary

This paper proposes a self-supervised pre-training strategy for medical imaging using a graph matching approach. Each unlabeled image is transformed via a pair of data augmentations and then processed via an encoder network. The augmented pair of images become vertices in a pair of graphs, with vertex features being the encoder outputs and edge connections selected via k-nn. A graph convolutional network is trained for vertex-to-vertex matching of the extracted pair of graphs. The training objective incorporates global and local similarity learning over spatial features along with a second-order edge similarity cost. Due to the combinatorial nature of the objective, gradients for backpropagation are approximated via Implicit MLE. Pre-training is performed over a large scale dataset comprising 55 publicly available datasets and used for multiple downstream fine-tuning tasks including segmentation, detection and classification.

Strengths

The proposed graph matching technique for self-supervised learning is a novel and significant contribution. Abundant experiments demonstrate the generalizability over downstream tasks, with error bars also included.

Weaknesses

Some related works on graph matching in computer vision are missing in Section 2.3. Doi et al. Detecting Object-Level Scene Changes in Images with Viewpoint Differences Using Graph Matching 2022 Bian et al. Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching 2022 Wu et al. Unsupervised Visible-Infrared Person Re-Identification via Progressive Graph Matching and Alternate Learning 2023 Liu et al. Self-supervised Learning of Visual Graph Matching 2022 Peng et al. GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysis 2022 The authors should discuss and contrast with the graph matching objectives and applications in these works to highlight their novelty in self-supervised learning. Introduction can also be improved to better highlight the contributions. Rather than starting with discussing image-text datasets, the story should highlight the value of self-supervised learning in medical imaging and related works > proposed self-supervised learning method via graph matching > large-scale dataset collected for implementing this method > experiments on downstream tasks.

Questions

Can the authors clarify how they use pre-trained ResNet-50 for downstream segmentation via U-Net? Does this mean that the encoder of U-Net has the same architecture as ResNet-50? This is not clear as the vanilla U-Net architecture (Cicek et al. 2015) is different.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

Limitations are discussed.

Summary

This paper collects a large medical imaging dataset, and it also shows that a self-supervised learning technique based on second-order graph-matching enhances performance in various downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on image-text instances. The evaluation also considers two different architectures: ResNet-50 and ViT backbones.

Strengths

- Creating such a large medical imaging dataset is a commendable feat, and is needed by the community. - The proposed self-supervised task is interesting. - The evaluation results are comprehensive and thorough.

Weaknesses

None to report.

Questions

None.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

These are included in the paper.

Summary

The paper proposes a set of networks called LVM-Med which are trained on large-scale medical datasets. The authors collected more than a million medical images from more than 50 publicly available datasets of diverse modalities and structures of interest (e.g. CT, MRI, Ultrasound...). In the work, several self-supervised algorithms are benchmarked on the large dataset. Furthermore, this work proposes a self-supervised contrastive learning algorithm based on a second-order graph-matching formulation.

Strengths

- the paper combines an incredibly large number of medical image modalities and images - I like the formulation of contrastive learning as a graph matching objective - the method section is comprehensive and the contributions are formalized

Weaknesses

1) Some aspects of the experimentation are unclear to me. From how I understand the text, the authors aim to compare to a large number of other datasets, baselines ("In 2D settings, we also compare with 2D supervised architectures, such as U-Net, U-Net++, Attention U-Net, etc."), and tasks across 2D and 3D. What I do not understand is how the authors choose their baselines and what they present in the tables. For example, in Table 2. for the Drive segmentation dataset, the authors report 2D supervised Methods (e.g., UNet) with Dice scores ranging from 59 to 65. Clearly, this is not a performance on par with scores reported in other works. From the literature, the state of the art in supervised DRIVE segmentation should be way higher. A fully supervised segmentation baseline on the DRIVE dataset should have 80+ Dice (https://paperswithcode.com/sota/retinal-vessel-segmentation-on-drive). Similarly, the IoU performance of BRATS baselines should be higher https://arxiv.org/pdf/1811.02629.pdf Potentially I misunderstand what kind of comparisons the authors provide here. Can the authors please explain the choice of their baselines and their experimental settings? 2) The clarity of the writing in some sections should be improved, e.g., in the experimentation. 3) The reproducibility of the results and methods is a concern. I have not seen the code. Furthermore, the sheer amount of computing required makes reproducibility challenging.

Questions

1) The work has been run on a large set of data. Can the authors provide more detail on the overall computing required to replicate their experiments? 2) The work uses the reparameterization trick to create a complex discrete distribution. Can the authors explain the effect of the backpropagation and what this implies for the learning signal in more detail? 3) Why are there no baselines for alternative ways to define a self-supervised contrastive loss in the presented setting? I would like to stress that I have an overall positive impression of the work and would be willing to reconsider my rating based on the rebuttal.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

-

Summary

The paper presents a large scale medical imaging dataset consisting of 1.3M medical images from 55 publicly available datasets along with a new contrastive learning framework based on graph matching. Specifically, the model is firstly pre-trained on the collected dataset, and then finetuned towards different downstream tasks, improvement is observed a cross different datasets and settings.

Strengths

+ A large scale *medical* dataset for pre-training purposes + Well-written and easy to follow + Extensive experiments and good results

Weaknesses

- The paper did not provide enough details regarding the proposed datasets, especially in terms of the usage of the dataset, which, in my humble opinion, is pretty important to provide guidance on training medical models on large-scale datasets, e.g., 1. how are 3D data used, are they sliced into 2D data firstly? 2. The data comes from different datasets and maybe in different modalities, any balancing strategy during pre-training? 3. Any augmentations besides multi-crop? (e.g., flip, rotate, color jittering etc). In short, I expect the author provide more details regarding how they utilize the dataset - I appreciate the author provided dataset statistics details in the supplementary, yet I am curious how are these datasets selected and have the author tried any filtering/curation? As data curation has been considered very important in modern large-scale model training [1], it would be great to have some insights in the medical area. [1] DataComp: In search of the next generation of multimodal datasets

Questions

see weakness

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

see weakness