Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking

Large language models (LLMs) are increasingly applied in healthcare, yet their evaluation relies predominantly on static benchmarks that are costly, contamination-prone, and lack calibrated measurement properties for fine-grained performance tracking. We developed and validated a computerized adaptive testing (CAT) framework grounded in item response theory to enable scalable, psychometrically rigorous assessment of standardized medical knowledge in LLMs. A two-phase study comprising Monte Carlo simulations and empirical evaluation of 38 LLMs was conducted between July and September 2025. The CAT protocol achieved a near-perfect correlation with full-bank results ( r  = 0.988) using only 1.3% of items. Evaluation time decreased from 6.85 hours to 8.4 min per model, and token usage dropped from 1.77 million to 0.03 million. Model rankings were fully preserved (Spearman’s ρ  = 1.0). At current API pricing, per-model evaluation costs fell from approximately $1,475 to under $5. This adaptive methodology serves as an essential pre- screening and continuous monitoring tool under a standardized testing protocol. Crucially, it is not a substitute for real-world clinical validation or safety-oriented prospective studies; rather, it enables developers and healthcare institutions to implement rigorous, high-frequency, evidence-based evaluation of foundational knowledge prior to more resource-intensive downstream testing.