LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs

Artificial intelligence is revolutionizing scientific research, yet its growing integration into laboratory environments presents critical safety challenges. Large language models and vision language models now assist in experiment design and procedural guidance, yet their ‘illusion of understanding’ may lead researchers to overtrust unsafe outputs. Here we show that current models remain far from meeting the reliability needed for safe laboratory operation. We introduce LabSafety Bench, a comprehensive benchmark that evaluates models on hazard identification, risk assessment and consequence prediction across 765 multiple-choice questions and 404 realistic laboratory scenarios, encompassing 3,128 open-ended tasks. Evaluations on 19 advanced large language models and vision language models show that no model evaluated on hazard identification surpasses 70% accuracy. While proprietary models perform well on structured assessments, they do not show a clear advantage in open-ended reasoning. These results underscore the urgent need for specialized safety evaluation frameworks before deploying artificial intelligence systems in real laboratory settings. Large language models are starting to be used in safety-critical tasks such as controlling robots. Zhou et al. present LabSafety Bench, a benchmark evaluating the ability of large language models to identify hazards and assess laboratory risks.