LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena

ABSTRACT LLM safety and ethical alignment are widely discussed, but the impact of refusals on user satisfaction remains underexplored. In particular, little is known about how users react when models refuse to answer, a primary mechanism for enforcing ethical boundaries in LLMs. We address this gap by analyzing nearly 50,000 model comparisons from Chatbot Arena, where users select their preferred LLM response in pairwise matchups. Using a novel RoBERTa-based classifier fine-tuned on hand-labelled data, we distinguish between refusals due to ethical concerns and technical limitations. Our results reveal a notable refusal penalty: ethical refusals receive significantly lower win rates than standard responses and even technical refusals, indicating that users are especially dissatisfied when models decline for ethical reasons. However, this penalty is not uniform. Refusals receive more favourable evaluations when the underlying prompt is highly sensitive (e.g. involving illegal content), and when the refusal is phrased in a detailed and contextually aligned manner. These findings underscore a core tension in LLM design: safety-aligned behaviours may conflict with user expectations, calling for more adaptive moderation strategies that account for context and presentation.