G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training

Summary

This paper proposes G2D, a novel vision-language pre-training (VLP) framework for medical imaging that aims to learn both global and dense visual representations from radiography images and their associated radiology reports. The key innovation is a pretext task called Pseudo Segmentation (PS), which uses a pseudo mask derived from attention maps to guide the learning of dense visual features during pre-training. The authors demonstrate that G2D outperforms existing medical VLP approaches on various downstream tasks including classification, segmentation, object detection, and zero-shot visual grounding across multiple medical imaging datasets. Notably, G2D shows strong performance on segmentation tasks even when fine-tuned on very limited data.

Strengths

Novel approach: The paper introduces an innovative method for learning dense visual representations in medical VLP without requiring pixel-level annotations, addressing a key limitation of existing approaches. Well-motivated: The authors provide a clear rationale for why learning dense representations is important for medical imaging tasks and why existing VLP methods struggle with this. Comprehensive evaluation: The method is evaluated on a wide range of downstream tasks and datasets, demonstrating its versatility and effectiveness across different medical imaging applications. Strong results: G2D consistently outperforms existing methods, especially on segmentation tasks where it achieves impressive results with very limited fine-tuning data. Ablation studies: The paper includes thorough ablation experiments to validate key design choices and components of the method. Potential impact: The proposed approach could significantly reduce the need for large annotated datasets in medical imaging, which is a major bottleneck in the field.

Weaknesses

Limited theoretical analysis: While the method is empirically strong, there is little theoretical justification for why the pseudo segmentation task leads to improved dense representations. Complexity of the approach: The method involves several components and processing steps, which may make it challenging to implement and potentially limit its adoption. Computational resources: The pre-training process appears to be computationally intensive (16 A100 GPUs), which could be a barrier for researchers with limited resources. Generalization to other domains: While the focus on medical imaging is valuable, it's unclear how well this approach would generalize to other vision-language domains. Comparison to more recent baselines: Some of the baselines used for comparison (e.g., ConVIRT, GLoRIA) are somewhat older. Comparison to more recent medical VLP methods would strengthen the evaluation.

Questions

Major concerns: My primary concern revolves around the authors' claim that current medical VLP methods primarily align images with entire text reports. This assertion appears to be inconsistent with the facts, as evidenced by several papers that have employed local alignment between image regions and text. This factual contradiction significantly undermines the novelty of the present work. For instance: GLoRIA (Huang et al., ICCV 2021): "Global-Local Representation Alignment for Improved Visual Recognition in Medical Imaging" This paper introduced a global-local alignment approach, learning finer-grained representations by aligning image patches with text tokens. MGCA (Wang et al., arXiv 2022): "Multi-Granularity Cross-Modal Alignment for Generalized Medical Visual Representation Learning" This method employed a multi-granularity alignment strategy, including global, local, and fine-grained levels of alignment. BioViL (Boecking et al., ECCV 2022): "Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing" This work proposed a method to improve biomedical vision-language processing by leveraging text semantics, which includes local alignment strategies. MedKLIP (Wu et al., medRxiv 2023): "Medical Knowledge Enhanced Language-Image Pre-training" This approach utilized external knowledge bases to enhance local alignment, achieving more fine-grained image-text matching. Given these existing works, the authors' characterization of the current state of medical VLP appears inaccurate. This misrepresentation significantly weakens the claimed novelty of their approach. The authors should provide a more accurate description of existing methods and clearly articulate how their approach differs from or improves upon these established local alignment strategies. Other minor concerns: Have you explored the quality of the learned representations at different levels of the network? Are there significant differences in the quality of features at different scales? How sensitive is the method to the choice of threshold used in pseudo mask construction? The ablation shows results for a few values, but is there a principled way to choose this threshold? Have you investigated the potential of using the pseudo masks generated during pre-training for weakly supervised segmentation tasks? How does the performance of G2D change as the amount of pre-training data is varied? Is there a clear relationship between pre-training data volume and downstream task performance? Given the computational requirements for pre-training, have you explored any techniques for making the approach more efficient, such as progressive training or curriculum learning?

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors provide a brief discussion of limitations in the appendix, acknowledging potential issues with the weak supervision signal from pseudo masks and the need for further research on regional visual representations. They also touch on broader impacts, mentioning both potential benefits for healthcare and risks associated with sensitive medical data. While these discussions are valuable, they could be expanded to provide more specific insights into the limitations of the current approach and potential mitigation strategies for the identified risks.

Summary

This manuscript describes a medical vision-language pre-training framework called Global to Dense level representation learning (G2D), that learns global and dense visual features simultaneously with only image-text pairs, by exploiting the aggregated attention map from the vision encoder for a pseudo segmentation pretext task. The improved (frozen) vision encoder is then utilized as part of the model pipeline for a number of downstream tasks (e.g. segmentation, classification)

Strengths

- Pseudo segmentation pretext task enables dense segmentation during pre-training, and avoids external resources as for alignment-based methods, and limitations on high-level semantic representations in reconstruction-based methods - Importance of associating semantic meaning verified via experiment

Weaknesses

- Unclear if specific sentence/phrase to individual image region alignment is achieved, for dense learning - Lack of fine-grained pixel-level evaluation of masks

Questions

1. The accuracy of the initial aggregated attention map appears possibly non-optimal, given that additional thresholding by body mask is required. As such, it might be considered to quantify the accuracy of these maps, possibly against segmentation ground truth. 2. In Section 3.2, it is stated that a threshold is applied (at 85%) to transform the aggregated attention map into a binary mask, before smoothing. It might be clarified if the need for smoothing (and related smoothing parameters) was empirically determined. 3. In Section 3.3, it is stated that "This decoder takes visual feature V_i as input and utilises the pseudo mask ˜M_i as the supervisory signal for the pretext task". It might be clarified as to whether and how specific text can be matched to specific (separate) image regions, as in Figure 4 of Section A.7. In other words, while Figure 4 shows specific text descriptions corresponding to specific image regions, were these correspondences/alignments indicated by the proposed G2D model, or are they external manual observations? A.1 suggests no, but this might be explicitly stated. 4. In Section 4, the choice of ResNet-50 as the encoder over other plausible choices (e.g. U-Net encoder) might be briefly explained. 5. For Table 1, it might be clarified as to what "encoder-decoder" refers to - the updating of both encoder and decoder?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper proposes an encoder-decoder medical VLP approach for global-to-dense visual representation learning. Pseudo segmentation is adopted for dense level learning. Rich experiments validate the effectiveness of the proposed method.

Strengths

1. The motivation behind the work is clear. Pseudo-segmentation supervision is effective, which is validated by experiments. 2. The experiments are rich and ablation analysis shows the contributions of each component and design. 3. The illustrations are clear and easy to understand. 4. The improvements are consistent and sometimes substantial.

Weaknesses

1. The comparisons with MGCA and MRM in the CXR14 dataset are not included in Table 3, but Table 4 includes the comparisons with MGCA and MRM. What are the reasons behind this? 2. Transformer-based vision encoder is not analyzed. 3. The balance between VLA and PA losses is not analyzed.

Questions

Is it not applicable to compare with MGCA and MRM in the CXR14 dataset?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

N/A

Summary

The paper proposes a new medical vision-language model, G2D, which employs vision-language alignment (VLA) and pixel alignment (PA) strategies, combined with a pseudo segmentation (PS) pre-training task, to learn global and dense visual representations from medical images. The VLA strategy is used to learn global representations of images and texts, while the PS task constructs pseudo masks through a parameter-free mechanism to facilitate the learning of dense representations. The method is comprehensively validated across five downstream tasks (image segmentation, object detection, zero-shot image visual grounding, zero-shot image classification, and fine-tuned image classification), demonstrating its effectiveness in handling both unimodal and cross-modal tasks.

Strengths

+ The paper is well-written, with the motivation, method, and results clearly presented. A minor concern is the reference format; it should be [1] instead of (1) according to the NeurIPS template. + A significant concern with most existing works is that they operate primarily at the Image-Text Retrieval level, similar to the perceptual level of CLIP, and do not effectively capture dense features between modalities. The G2D model addresses this issue by integrating Vision-Language Alignment (VLA) and Pseudo Segmentation (PS) tasks to facilitate simultaneous learning of global and dense visual features. This multi-level feature learning significantly enhances the model's performance in tasks requiring dense feature perception, such as segmentation. + During pre-training, the G2D method utilizes only image-text pairs without the need for additional annotated data. By generating pseudo masks on the fly through the PS task, it reduces the cost and complexity associated with data annotation. + The G2D method is novel, and the experiments are robust. Experimental results on five medical imaging tasks involving 25 diseases demonstrate that the G2D model outperforms existing models, even with minimal fine-tuning data. Notably, in segmentation tasks requiring dense visual features, G2D achieves excellent results with just 1% of the training data for fine-tuning.

Weaknesses

Major concerns: - The attention maps could introduce errors in pseudo mask, and these errors may propagate throughout the training process. To address this, a clear validation strategy needs to be outlined. For instance, in Figure 2, aggregated attention map might incorrectly highlight irrelevant regions. It is essential to establish methods for **detecting** and **measuring** these errors to ensure the reliability of the model. I hope the authors could quantify the errors in aggregated attention map and pseudo mask during the rebuttal period. Minor concerns: - The training and validation of the model rely on specific datasets, which may introduce biases and potentially affect the model's generalizability to different datasets. - It is uncertain whether the method can be effectively extended to vision-language tasks involving 3D imaging (e.g., CT and MRI), presenting a limitation in its current scope of application.

Questions

- How do you detect and correct the errors made by aggregated attention map?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations were discussed in Section A.1

The rebuttal addresses my concerns

I thank authors for their efforts in addressing my concerns. After reading the rebuttal, my concern has been addressed. And I will update the final rating.