Large Language Model Relevance Assessors Agree With One Another More Than With Human Assessors

Abstract

Relevance judgments can differ between assessors, but previous work has shown that such disagreements have little impact on the effectiveness rankings of retrieval systems. This applies to disagreements between humans as well as between human and large language model (LLM) assessors. However, the agreement between different LLMassessors has not yet been systematically investigated. To close this gap, we compare eight LLMassessors on the TREC DL tracks and the retrieval task of the RAG track with each other and with human assessors. We find that the agreement between LLMassessors is higher than between LLMs and humans and, importantly, that LLMassessors favor retrieval systems that use LLMs in their ranking decisions: our analyses with 30-50 retrieval systems show that the system rankings obtained by LLMassessors overestimate LLM-based re-rankers by 9to 17~positions on average.