Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining

There is an increasing focus on model-based dialog evaluation metrics such as\nADEM, RUBER, and the more recent BERT-based metrics. These models aim to assign\na high score to all relevant responses and a low score to all irrelevant\nresponses. Ideally, such models should be trained using multiple relevant and\nirrelevant responses for any given context. However, no such data is publicly\navailable, and hence existing models are usually trained using a single\nrelevant response and multiple randomly selected responses from other contexts\n(random negatives). To allow for better training and robust evaluation of\nmodel-based metrics, we introduce the DailyDialog++ dataset, consisting of (i)\nfive relevant responses for each context and (ii) five adversarially crafted\nirrelevant responses for each context. Using this dataset, we first show that\neven in the presence of multiple correct references, n-gram based metrics and\nembedding based metrics do not perform well at separating relevant responses\nfrom even random negatives. While model-based metrics perform better than\nn-gram and embedding based metrics on random negatives, their performance drops\nsubstantially when evaluated on adversarial examples. To check if large scale\npretraining could help, we propose a new BERT-based evaluation metric called\nDEB, which is pretrained on 727M Reddit conversations and then finetuned on our\ndataset. DEB significantly outperforms existing models, showing better\ncorrelation with human judgements and better performance on random negatives\n(88.27% accuracy). However, its performance again drops substantially, when\nevaluated on adversarial responses, thereby highlighting that even large-scale\npretrained evaluation models are not robust to the adversarial examples in our\ndataset. The dataset and code are publicly available.\n