Rank, Don't Generate: Statement-level Ranking for Explainable Recommendation

Textual explanations, generated with large language models (LLMs), are increasingly used to justify recommendations. Yet, evaluating these explanations remains a critical challenge. Lexical n-gram metrics struggle with paraphrases; semantic metrics reward similarity without ensuring factual grounding; LLM-based evaluators are highly prompt-dependent, often proprietary, and hard to replicate, while current evaluation protocols provide little insight into which explanatory factors are faithful and relevant. Meanwhile, proposed models tend to hallucinate unsupported claims or default to generic rationales. We advocate a shift in objective: rank, don't generate. We formalize explainable recommendation as a statement-level ranking problem, where systems rank candidate explanatory statements derived from reviews and return the top-k as explanation. This formulation mitigates hallucination by construction and enables fine-grained factual analysis. It also models factor importance through relevance scores and supports standardized, reproducible evaluation with established ranking metrics. Meaningful assessment, however, requires each statement to be explanatory (item facts affecting user experience), atomic (one opinion about one aspect), and unique (paraphrases consolidated), which is challenging to obtain from noisy reviews. We address this with (i) an LLM-based extraction pipeline producing explanatory and atomic statements, and (ii) a scalable, semantic clustering method consolidating paraphrases to enforce uniqueness. Building on this pipeline, we introduce StaR, a benchmark for statement ranking in explainable recommendation, constructed from four Amazon Reviews 2014 product categories. We evaluate popularity-based baselines and state-of-the-art models under global-level (all statements) and item-level (target item statements) ranking. Popularity baselines are competitive in global-level ranking but outperform state-of-the-art models on average under item-level ranking, exposing critical limitations in personalized explanation ranking. Code, data, and supplementary materials are released to support reproducible research and future work: https://github.com/BenKabongo25/Statement_Ranking.