From Untestable to Testable: Metamorphic Testing in the Age of LLMs

This article discusses the challenges of testing software systems with increasingly integrated AI and LLM functionalities. LLMs are powerful but unreliable, and labeled ground truth for testing rarely scales. Metamorphic Testing solves this by turning relations among multiple test executions into executable test oracles.

Software engineering cs.SE cs.AI Computer Science

View graph Source

Paper

References (3)

01Metamorphic Model-Based Testing Applied on NASA DAT -- An Experience ReportM. Lindvall, D. Ganesan, Ragnar Ardal et al.2015 · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · 74 citations

02“Metamorphic testing: A review of challenges and opportunities,”2018 · ACM Computing Surveys

03Automated Metamorphic Testing of Large Language Models”Proc. Int. Conf. on Automated Software Engineering (ASE) , 2025,

Similar papers

Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic SurveyZheng Zheng, Zenghui Zhou, Yinwang Xu et al.2026 · arXiv.org · 0 citations In Library

Testing the Untestable? An Empirical Study on the Testing Process of LLM-Powered Software SystemsCleyton Magalhaes, Ítalo Santos, Brody Stuart-Verner et al.2025 · IEEE Working Conference on Source Code Analysis and Manipulation · 1 citation In Library

Metamorphic Testing of Large Language Models for Natural Language ProcessingSteven Cho, Stefano Ruberto, Valerio Terragni2025 · IEEE International Conference on Software Maintenance and Evolution · 12 citations In Library

Metamorphic Testing of Large Language Models for Natural Language ProcessingS. B. Cho, Stefano Ruberto, Valerio Terragni2025 In Library

Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction ProtocolWei Ma, Yixiao Yang, Qiang Hu et al.2025 · arXiv.org · 4 citations In Library

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMsZenghui Zhou, Man Li, Xiaoke Fang et al.2026 · Knowledge-Based Systems · 0 citations In Library

LLMORPH: Automated Metamorphic Testing of Large Language ModelsSteven Cho, Stefano Ruberto, Valerio Terragni2025 · International Conference on Automated Software Engineering · 2 citations In Library

Test Oracle Automation in the Era of LLMsF. Molina, Alessandra Gorla, Marcelo d’Amorim2024 · ACM Transactions on Software Engineering and Methodology · 48 citations In Library

Towards Generating Executable Metamorphic Relations Using Large Language ModelsSeung Yeob Shin, Fabrizio Pastore, Domenico Bianculli et al.2024 · arXiv (Cornell University) · 13 citations In Library

Understanding LLM-Driven Test Oracle GenerationAdam Bodicoat, Gunel Jahangirova, Valerio Terragni2025 · 2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware) · 6 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal