From High School Final Exam to LLMs: Evaluation of Machine Translations from Latin to Italian

Machine translation represents one of the earliest domains addressed by computational linguistics, with experiments dating back to the 1950s. In recent years, the quality of machine translation has significantly improved thanks to the introduction of increasingly advanced technologies, ranging from deep learning algorithms (as used in Google Translate) to the more recent Large Language Models (LLMs).n this contribution, we propose an evaluation of the quality of machine translations from Latin into Italian of all Italian high school final examination texts, produced by a selection of LLMs chosen to represent the main currently available model types. Both large-scale commercial and proprietary models and smaller open-source models will be analyzed. LLMs have already been evaluated for translation from Latin into several languages, including English, German, and Spanish; however, a systematic analysis of the Latin–Italian translation pair is still lacking.The quality of the translations produced by the LLMs will be assessed through an integrated approach. First, the automatic metric COMET will be applied; this metric has been shown to correlate well with human judgments, as it evaluates the contextual meaning of translations, thereby overcoming the limitations of traditional metrics based on simple word overlap. In addition, a qualitative evaluation will be conducted in two complementary phases: (i) assessment by upper secondary school teachers, who will evaluate the translations according to criteria analogous to those adopted in school practice; and (ii) identification and classification of errors according to the MQM (Multidimensional Quality Metrics) framework. The aim of this study is to provide an overview of the capabilities and limitations of LLMs in handling a linguistically complex task such as translation from Latin.