The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialogues

When training a model on referential dialogue guessing games, the best model\nis usually chosen based on its task success. We show that in the popular\nend-to-end approach, this choice prevents the model from learning to generate\nlinguistically richer dialogues, since the acquisition of language proficiency\ntakes longer than learning the guessing task. By comparing models playing\ndifferent games (GuessWhat, GuessWhich, and Mutual Friends), we show that this\ndiscrepancy is model- and task-agnostic. We investigate whether and when better\nlanguage quality could lead to higher task success. We show that in GuessWhat,\nmodels could increase their accuracy if they learn to ground, encode, and\ndecode also words that do not occur frequently in the training set.\n