Library

LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2025.

Law & regulationArtificial Intelligence in Law
Source

Paper

The full text of this publication is not hosted on 44B due to licensing.

Read it at OpenAlex

Similar papers

LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation TasksA. Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi et al.2024 · Annual Meeting of the Association for Computational Linguistics · 292 citations In Library
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation TasksAnna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi et al.2024 · Repository for Publications and Research Data (ETH Zurich) In Library
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation MethodsHaitao Li, Qian Dong, Junjie Chen et al.2024 · arXiv.org · 537 citations In Library
The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMsNitay Calderon, Roi Reichart, Rotem Dror2025 · arXiv (Cornell University) · 86 citations In Library
Humans or LLMs as the Judge? A Study on Judgement BiasesGuiming Hardy Chen, Shunian Chen, Ziche Liu et al.2024 · Conference on Empirical Methods in Natural Language Processing · 350 citations In Library
Neither Valid nor Reliable? Investigating the Use of LLMs as JudgesKhaoula Chehbouni, Mohammed Haddou, J. Cheung et al.2025 · Neural Information Processing Systems · 32 citations In Library
Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human AgreementSteve Han, Gilberto Titericz Junior, Tom Balough et al.2025 · arXiv.org · 23 citations In Library
LLM Evaluations: A Survey of Programmatic, Human, and LLM-as-Judge ApproachesS. Krishnan2025 · International Journal of Innovative Research in Science Engineering and Technology · 1 citation In Library
Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM JudgmentsYuran Li, Jama Hussein Mohamud, Sun, Chongren et al.2025 · arXiv (Cornell University) · 18 citations In Library
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgeDawei Li, Bohan Jiang, Liangjie Huang et al.2024 · Conference on Empirical Methods in Natural Language Processing · 579 citations In Library
© 2026 NYSGPT2525 LLC