Summary
The paper introduces GMAI-VL-5.5M, a comprehensive multimodal medical dataset comprising image-text pairs. Additionally, it presents GMAI-VL, a robust general medical vision-language model designed to effectively process multimodal data for clinical decision-making. Experimental results across four datasets demonstrate the model’s effectiveness and potential impact.
Strengths
- The paper is well-written and easy to follow, with visualizations that effectively illustrate the architectural concepts.
- It introduces an open-source dataset encompassing multiple modalities and tasks, though questions remain regarding its overall usefulness.
Weaknesses
Overall, I am concerned about the novelty of this work. Although the authors introduce a large-scale, open-source dataset, it has several notable limitations:
- All labels, masks, and instructions were generated by GPT-4, potentially restricting performance in downstream tasks. Notably, this dataset was not used for validation, which is unexpected.
- The dataset lacks a normalization or standardization process, suggesting that distributions may vary significantly across sources. This variability could have a substantial impact on model training.
- No clinical validation was conducted to confirm the dataset’s practical utility.
Additional Observations:
- In Section 3.1, given that GPT-4 was used to curate the dataset, one would expect downstream performance to be constrained by GPT-4’s limitations. It would be helpful to see GPT-4’s performance on this dataset as a baseline for comparison.
- Section 3 lacks critical details regarding data preparation. It is unclear how images from different institutions were standardized—specifically, the types of scanners used, the consistency in imaging perspectives, any handling of motion blur, and how such issues were managed. Additionally, information on dataset distribution, potential variability, and any applied thresholds is missing. These details are essential to determine the dataset’s suitability for models operating on diverse distributions.
- Although the authors prepared and open-sourced the GMAI-VL-5.5M dataset, it is surprising that metrics from the proposed benchmark are not reported.
- In Line 347, it is mentioned: “We utilize a large-scale medical image-text dataset comprising approximately 11.7 million image-text pairs.” Clarification is needed on whether this dataset is part of GMAI-VL-5.5M and how these data were sourced.
- In Line 362, regarding instruction tuning, the authors note: “The multimodal instruction tuning data is primarily derived from the training data in previous stages by filtering high-quality and more suitable data for fine-tuning.” However, it is unclear how this filtering process was conducted and what inclusion or exclusion criteria were applied.
- Line 320 raises another question: Why was InternLM2.5-7B chosen as the model base instead of a similarly scaled alternative, such as LLaMA-3-8B, which also functions as a generalist? Additionally, has Bio-LLM been considered for the language model?
- In Table 2, the superiority of the proposed method over other baselines, such as HuatouGPT-Vision-7B, remains unconvincing.
- No ablation study has been conducted to evaluate the contribution of each module within the proposed network.
Ethics concerns
In line 348, the authors state, “In the shallow alignment phase, we utilize a large-scale medical image-text dataset comprising approximately 11.7 million image-text pairs, sourced from a combination of publicly accessible datasets and proprietary in-house data.” However, the in-house data is not mentioned elsewhere in the work, raising concerns about ethical compliance and data transparency.