Summary
The work presented aims to evaluate the medical capabilities of OpenAI's latest `o1` model using 37 medical benchmarks, including tasks in concept recognition, clinical decision support and knowledge QA. The authors use a different set of metrics depending on the benchmarks, such as accuracy, f-score, BLEU/ROUGE and AlignScore. The authors compare the results with previous models `GPT-4`, `GPT-3.5` and open-source models `Meditron-70b` and `Llama3 8b`. The results demonstrate overall higher raw scores compared to previous models. Additional results demonstrate that using CoT prompting techniques on the older models improve their performance. The authors briefly discuss the limitations of the metrics used and the need for new evaluation methods. Based on these results, the authors conclude that these scores demonstrate that the `o1` model is closing the gap with human physicians.
Strengths
The paper addresses a timely and important topic: the evaluation of the medical capabilities of OpenAI's latest o1 model. This focus is particularly relevant given the increasing deployment of these models in clinical settings. The paper is generally well-written and structured, which aids in navigating its content.
The paper uses a wide range of benchmarks, uses different evaluation metrics and introduces two new benchmarks which contributes to demonstrating that the improvements of o1 over previous models is not limited to a few benchmarks.
The analysis in section 4.3 and the discussion provide valuable insights. These sections accurately describe the limitations of the metrics used and highlight the need for new evaluation methodologies that more accurately assess medical capabilities. This critical perspective on current evaluation methods is the paper's most significant contribution.
The appendix contains useful additional data, including the prompts used and decoding times, which adds a degree of transparency to the study. The inclusion of case studies, while limited, provides concrete examples that complement the main text.
Weaknesses
While the paper addresses a gap in literature, the methodology used, the reported results and especially their interpretation raise major concerns.
## Evaluation pipeline
1. The use of BLEU and ROUGE metrics for assessing tasks such as summarization is questionable, given that these metrics primarily measure n-gram overlap. This concern is further amplified by the results in Tables 3 and 4, which demonstrate no correlation between AlignScore and the BLEU/ROUGE results. This discrepancy raises doubts about the interpretation of results for these tasks.
2. The model selection appears limited in scope. A more comprehensive comparison would include comparable models from competitors, such as Claude 3.5 Sonnet. Additionally, the decision to compare OpenAI's latest and most powerful model to the 8b parameter version of Llama 3 is questionable. A more appropriate comparison would involve evaluating Llama 3 with 405b parameters.
## Experiments
3. There are significant inconsistencies in the reported results, raising concerns about their validity. For instance:
- GPT-4 and GPT-3.5 are reported to achieve 52.8% and 25.4% on PubMedQA, respectively. However, previously reported results by Microsoft and OpenAI were 75.2% and 71.6% in zero-shot settings [1].
- Inconsistencies exist between tables, such as the apparent inversion of MedQA and MedMCQA results in Table 6 compared to Table 2.
- The NEJMQA results are not whole numbers, which is puzzling given that only 100 questions were used for evaluation.
The lack of reproducible code exacerbates these concerns.
4. The results presented in Table 2 are incomplete, with Meditron and Llama 3 lacking results for MedBullets, LancetQA, NEJMQA, and MedCalc-Bench.
5. The reported results, while adequately formatted, lack statistical significance or confidence intervals. This omission is particularly problematic for benchmarks that were subsampled due to budget constraints, such as NEJMQA (evaluated with only 100 questions). A t-test comparing the results of o1 and GPT-4 on this benchmark yields a p-value > 0.05, indicating no statistically significant difference.
6. The main conclusion (Section 4.2) stating "Yes! WE ARE ONE STEP CLOSER TO AN AI DOCTOR" is not sufficiently supported by the evidence presented. The paper does not demonstrate that higher performance on these benchmarks correlates with medical proficiency. Modern models of clinical reasoning emphasize that medical decision-making does not rely solely on internal cognition [2]. Even with proper evaluations of internal cognitive processes, additional assessments addressing the various cognitive processes involved in clinical reasoning would be necessary to support such a conclusion.
## Case study
7. The case study presented in Figure 6 is incomplete and potentially misleading. The NEJM question referenced was originally an image challenge [3]. It appears that the models were not provided with the necessary images, as evidenced by GPT-4's response mentioning "radiographic findings without seeing the images." This suggests that 'o1' may have hallucinated the radiographic findings ("isolated ulnar fracture") and arrived at the correct answer for incorrect reasons.
## References
[1] Nori, H., King, N., McKinney, S.M., Carignan, D., & Horvitz, E. (2023). Capabilities of GPT-4 on Medical Challenge Problems. ArXiv, abs/2303.13375.
[2] Parsons, A. S., Wijesekera, T. P., Olson, A. P. J., Torre, D., Durning, S. J., & Daniel, M. (2024). Beyond thinking fast and slow: Implications of a transtheoretical model of clinical reasoning and error on teaching, assessment, and research. Medical Teacher, 1–12. https://doi.org/10.1080/0142159X.2024.2359963
[3] NEJM Challenge 2023/04/20, https://www.nejm.org/image-challenge?ci=20230420
Questions
While the paper in its current form is not suitable for publication, it contains potentially valuable insights that could significantly contribute to the field. The most interesting finding appears to be the inadequacy of current testing methodologies, particularly ROUGE/BLEU, in evaluating free-form text medical capabilities. For critical domains such as medicine, it is paramount that the BioNLP community uses appropriate methodologies to ensure patient safety.
## Reframing the paper
The paper could be substantially improved by reframing its focus on the limitations of current free-form text evaluations in medical AI. This reframing should include a comprehensive review of existing evaluation methods, a detailed analysis of why ROUGE/BLEU metrics are insufficient for medical text evaluation, and proposals for more appropriate evaluation techniques. By shifting the focus, the paper could make a significant contribution to the ongoing discussion about responsible AI development in healthcare.
## Methodology
1. The statistical rigor needs enhancement. This includes conducting and reporting appropriate statistical analyses to compare model results, including confidence intervals or p-values for reported metrics.
2. The benchmark selection and reporting require justification and improvement. The inclusion of benchmarks with small sample sizes, such as NEJMQA with only 100 samples, raises concerns about result reliability. The authors should either exclude these or discuss their limitations extensively.
3. For custom benchmarks like LancetQA and NEJMQA, a detailed methodology is needed. This should explain how questions with images or non-textual media were handled, the process for removing or substituting images with textual findings, and the potential impacts of these modifications on benchmark validity.
4. Incorporating human evaluators for at least a portion of the free-form text outputs would significantly enhance the paper's credibility. This should include a description of the human evaluation methodology, rater qualifications, and inter-rater reliability measures.
5. The results could be strengthened by evaluating more appropriate models such as Claude 3.5 Sonnet and Llama3.1 405b in addition to the existing evaluations.
6. A thorough review of the benchmarking code is necessary to address discrepancies with previous work. Making the evaluation code publicly available would allow independent verification of results, enhancing transparency and reproducibility.
## Results
7. The authors should revise their interpretation of results, avoiding direct links between BLEU/ROUGE metrics and "real-world clinical understanding" without supporting evidence. A more nuanced discussion of what the results might indicate, acknowledging the limitations of the metrics used, would be more appropriate.
Ethics concerns
The paper's methodological flaws and overreaching conclusions raise concerns, but it's unclear if they warrant a formal ethics review. While primarily scientific issues, there's a potential risk of indirect harm from overconfidence in the evaluated AI system. The paper highlights the need for more rigorous evaluation in medical AI but falls in a gray area regarding ethical implications. Standard peer review may suffice, but the decision is not straightforward.