Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias

Summary

This paper proposes a vision language pretraining method that focusing on tackling the bias caused by different languages. The Text Alignment Regularization (CTR) is proposed to unify cross-lingual semantic representations of medical reports. The experiments show that the proposed CTR can effectively eliminate the bias between English and Spanish medical report.

Strengths

1. This paper address an important and interesting issue of the bias caused by different languages in medical visual language pretraining tasks. 2. The experiments show the performance superiority of the proposed method on both normal medical recognition tasks and cross-linguistic tasks.

Weaknesses

1. The design of the proposed visual language model is closed to existing MLM and CLIP based VLP methods with a incremental improvement of CTR. More analysis is needed for the difference between existing methods and existing VLP methods. 2. Seems only zero-shot classification tasks are evaluated under the cross-linguistic setting. Is it possible to evaluate other downstream task under this setting？ 3. Can you provide more analysis on why the proposed method outperform other medical VLP methods? It seems like the main difference of this method (CTR) does not really have a strong positive impact on single language recognition tasks.

Questions

Please refer to the concerns in the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations

Summary

This paper presents a unified framework for Cross-Lingual Medical Vision-Language Pre-Training (Med-UniC), integrating multimodal medical data from different languages (e.g., English and Spanish). A Cross-lingual Text Alignment Regularization (CTR) is proposed to explicitly unify cross-lingual semantic representations of medical reports originating from diverse language communities. It reaches superior performance across 5 medical image tasks and 10 datasets encompassing over 30 diseases.

Strengths

1. Practical and interesting problem setting, which attempts to unifying the Medical Vision-Language Pre-Training for multiple languages. 2. The method is straight-forward and simple, making the paper easy to understand. 3. The Cross-lingual Alignment loss seems reasonable.

Weaknesses

1. The experimental setting is unclear. For example, the implementation and the training data of the SOTA methods are not introduced. 2. The ablation study seems incomplete, only four settings are shown. It is hard to tell how much gain actually comes from the key design of the paper, i.e., the CTR loss. The biggest gain seems to come from the component "MLM", which is not introduced in the paper. I would also suggest the authors to provide ablation showing how much gain is from the increase in data quantity.

Questions

1. How are the SOTA methods implemented? What data set are they trained on? I wonder whether the comparison is fair if the SOTA methods and the proposed method are trained with different set of data. If the SOTA methods are only trained with one language, I wonder how much gain comes from the additional data. 2. Table 1. Why for CXP500, most SOTA methods show a decrease in performance when the prompts transition from English to Spanish on CXP500, but an increase on PDC? The proposed method shows a consistent decrease in performance when the prompts transition from English to Spanish on both data sets. 3. What does the learning objective “MLM” mean in Table 4?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

No discussion on limitations

Summary

One common challenge in performing medical vision-language pre-training (VLP) is data scarcity, especially in languages other than English. This challenge can be addressed by combining datasets from various languages to train language-agnostic models, but the authors empirically show that each language community (especially non-English ones) induces distinct linguistic biases in their data, even in language-agnostic models. The authors therefore introduce Med-UniC, which leverages cross-lingual text alignment regularization (CTR) (experimented with English and Spanish) to mitigate these biases and achieve SOTA results using chest X-ray scans and reports on many uni-modal visual tasks.

Strengths

• Appears to be novel, the idea of cross-lingual text alignment regularization (CTR) seems to effectively address a significant per-language bias problem in multilingual models based on self-vision, vision-language, and cross-lingual alignment strategies. • Clear architecture explanation with hyperparameters with comprehensive experiments. • Strong results on all experiments. • Generally well-written

Weaknesses

The bias analysis section seems brief given how much attention it was given in the abstract/intro. The authors state that more analysis is in the appendix, but I would have wanted to see more in the main paper – I’m curious whether they attempted to identify the sources of the bias, or it would have been neat to see some samples with similar (language-agnostic) content but different embeddings due to this bias.

Questions

see above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper aims to address community bias caused by having data in multiple languages in medical vision-language pre-training (VLP). Specifically, it introduces the Unifying Cross-Lingual Medical Vision-Language Pre-Training framework to integrate multi-model data from English and Spanish and proposes a Criss-lingual Text Alignment Regularization (CTR) objective to unify cross-lingual semantic representations of medical reports from different languages. The paper shows superior performance compared to other methods on 5 tasks spanning 10 datasets and provides evidence of community bias in cross-lingual VLP through experimental findings.

Strengths

- Mitigating cultural bias in cross-lingual visual-language datasets in English/Spanish is a significant task with potential applications in the clinical field - The findings are novel. In particular, the separation between the learned representations in Med-Unic w/o CTR and the clear closer alignment in embedding space in Med-Unic with CTR is very interesting as it shows clearly that the representational spaces of the datasets in English and Spanish were separated without CTR and are better integrated now, which shows the alignment power of the CTR objective - Results - the paper obtains significant improvement over SOA techniques. - The CTR loss is an innovative method to address community bias in multi-lingual pre-training methodologies in a mathematically interesting way

Weaknesses

- Looking at Figure 4, it is clear that the pre-training methodology proposed by the authors has significant improvements in bringing the learned representations from the two languages closer in the latent space. However, the two representations are still not perfectly aligned in the latent space, which means that the embeddings are still not perfectly integrated. Full integration would show Figure 4a where the two representations are fully overlapped, whereas now there is still separation between the two, with minimal overlap. While this is still a significant result, the model could probably be improved to better integrate and align the datasets. - I would like to understand if the authors think that with further training, these two representations can be further aligned, or if this is the maximum representational power capacity of the proposed pre-training methodology.

Questions

- How was the similarity in Figure 2 between medical reports obtained? What does the X-axis represent?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Authors have adequately addressed limitations of their work.