Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems

The lack of time-efficient and reliable evaluation methods hamper the\ndevelopment of conversational dialogue systems (chatbots). Evaluations\nrequiring humans to converse with chatbots are time and cost-intensive, put\nhigh cognitive demands on the human judges, and yield low-quality results. In\nthis work, we introduce \\emph{Spot The Bot}, a cost-efficient and robust\nevaluation framework that replaces human-bot conversations with conversations\nbetween bots. Human judges then only annotate for each entity in a conversation\nwhether they think it is human or not (assuming there are humans participants\nin these conversations). These annotations then allow us to rank chatbots\nregarding their ability to mimic the conversational behavior of humans. Since\nwe expect that all bots are eventually recognized as such, we incorporate a\nmetric that measures which chatbot can uphold human-like behavior the longest,\ni.e., \\emph{Survival Analysis}. This metric has the ability to correlate a\nbot's performance to certain of its characteristics (e.g., \\ fluency or\nsensibleness), yielding interpretable results. The comparably low cost of our\nframework allows for frequent evaluations of chatbots during their evaluation\ncycle. We empirically validate our claims by applying \\emph{Spot The Bot} to\nthree domains, evaluating several state-of-the-art chatbots, and drawing\ncomparisons to related work. The framework is released as a ready-to-use tool.\n