Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions

We evaluate LLM performance and robustness for materials science Q&A and property prediction. Prompt sensitivity and mode collapse reveal reliability concerns, providing informed skepticism and guiding more cautious adoption in scientific research.

cs.CL cond-mat.mtrl-sci cs.AI cs.LG Materials Science Computer Science Physics

View graph Source

Paper

Are LLMs Ready for Real-World Materials Discovery?Santiago Miret, N. Krishnan2024 · arXiv.org · 62 citations In Library

LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property PredictionAndre Niyongabo Rubungo, Kangming Li, J. Hattrick-Simpers et al.2024 · Machine Learning: Science and Technology · 40 citations In Library

MatTools: Benchmarking Large Language Models for Materials Science ToolsSiyu Liu, Jiamin Xu, Beilin Ye et al.2025 · arXiv.org · 12 citations In Library

Regression with Large Language Models for Materials and Molecular Property PredictionR. Jacobs, Maciej P. Polak, Lane E. Schultz et al.2024 · arXiv.org · 13 citations In Library

MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and KnowledgeJerry Junyang Cheung, Shiyao Shen, Yuchen Zhuang et al.2025 · arXiv.org · 5 citations In Library

LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and DistillationChiang Yuan, Chia-Hong Chou, Janosh Riebesell2024 · Conference on Empirical Methods in Natural Language Processing · 42 citations In Library

34 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific DiscoveryYoel Zimmermann, Adib Bazgir, Alexander Al-Feghali et al.2025 · arXiv.org · 8 citations In Library

Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and CredibilityWentao Ye, Mingfeng Ou, Tianyi Li et al.2023 · arXiv (Cornell University) · 49 citations In Library

What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt EngineeringFederico Errica, G. Siracusano, D. Sanvito et al.2024 · North American Chapter of the Association for Computational Linguistics · 120 citations In Library

Patents near this paper

PREDICTIVE DESIGN SPACE METRICS FOR MATERIALS DEVELOPMENTUS 10,657,300 · CITRINE INFORMATICS, INC. · 2020 APPLICATION FOR MANAGING MODEL-BASED MATERIAL PROPERTIES FOR SIMULATION-BASED ENGINEERINGUS 7,499,841 · SANDIA NATIONAL LABORATORIES · 2009 SYSTEMS AND METHODS FOR PREDICTING MATERIALS PROPERTIESUS 7,292,958 · MASSACHUSETTS INSTITUTE OF TECHNOLOGY · 2007 Hypothesis-Driven Classification of Materials Using Nuclear Magnetic Resonance RelaxometryUS 9,411,031 · LOS ALAMOS NATIONAL SECURITY, LLC · 2016 GENERATING ROBUST MACHINE LEARNING PREDICTIONS FOR SEMICONDUCTOR MANUFACTURING PROCESSESUS 11,029,673 · PDF SOLUTIONS, INC. · 2021

Source of record

arXiv

Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions

Paper

Similar papers

Patents near this paper

Welcome.

Patents near this paper