USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation

The lack of meaningful automatic evaluation metrics for dialog has impeded\nopen-domain dialog research. Standard language generation metrics have been\nshown to be ineffective for evaluating dialog models. To this end, this paper\npresents USR, an UnSupervised and Reference-free evaluation metric for dialog.\nUSR is a reference-free metric that trains unsupervised models to measure\nseveral desirable qualities of dialog. USR is shown to strongly correlate with\nhuman judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and\nPersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces\ninterpretable measures for several desirable properties of dialog.\n