FACE: A Fine-Grained Reference-Free Evaluator for Conversational Information Access

A systematic, reliable, and low-cost evaluation of Conversational Information Access (CIA) systems remains an open challenge. Existing reference-based evaluation methods have proven insufficient for evaluating the dynamic nature of information access conversations, while existing LLM-based reference-free methods suffer from evaluation bias and limited generalizability. We propose FACE: a Fine-grained, Aspect-based Conversation Evaluator that evaluates diverse turn- and dialogue-level aspects of conversations. FACE leverages beam search and bandit optimization to select optimized LLM instructions for each evaluation aspect. It assigns scores to atomic information units (particles) using the selected instructions and then aggregates them into a single score. We show that FACE achieves strong correlations with human judgments, outperforming state-of-the-art conversation evaluation methods by a large margin. We further demonstrate that its optimized instructions are transferable across various LLMs and datasets. Additionally, unlike existing LLM-based methods that provide single uninterpretable scores, FACE provides insights into the system performance and enables identifying and locating problems within conversations.