Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills

Being engaging, knowledgeable, and empathetic are all desirable general\nqualities in a conversational agent. Previous work has introduced tasks and\ndatasets that aim to help agents to learn those qualities in isolation and\ngauge how well they can express them. But rather than being specialized in one\nsingle quality, a good open-domain conversational agent should be able to\nseamlessly blend them all into one cohesive conversational flow. In this work,\nwe investigate several ways to combine models trained towards isolated\ncapabilities, ranging from simple model aggregation schemes that require\nminimal additional training, to various forms of multi-task training that\nencompass several skills at all training stages. We further propose a new\ndataset, BlendedSkillTalk, to analyze how these capabilities would mesh\ntogether in a natural conversation, and compare the performance of different\narchitectures and training schemes. Our experiments show that multi-tasking\nover several tasks that focus on particular capabilities results in better\nblended conversation performance compared to models trained on a single skill,\nand that both unified or two-stage approaches perform well if they are\nconstructed to avoid unwanted bias in skill selection or are fine-tuned on our\nnew task.\n