Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents

Turn-level user satisfaction is one of the most important performance metrics\nfor conversational agents. It can be used to monitor the agent's performance\nand provide insights about defective user experiences. Moreover, a powerful\nsatisfaction model can be used as an objective function that a conversational\nagent continuously optimizes for. While end-to-end deep learning has shown\npromising results, having access to a large number of reliable annotated\nsamples required by these methods remains challenging. In a large-scale\nconversational system, there is a growing number of newly developed skills,\nmaking the traditional data collection, annotation, and modeling process\nimpractical due to the required annotation costs as well as the turnaround\ntimes. In this paper, we suggest a self-supervised contrastive learning\napproach that leverages the pool of unlabeled data to learn user-agent\ninteractions. We show that the pre-trained models using the self-supervised\nobjective are transferable to the user satisfaction prediction. In addition, we\npropose a novel few-shot transfer learning approach that ensures better\ntransferability for very small sample sizes. The suggested few-shot method does\nnot require any inner loop optimization process and is scalable to very large\ndatasets and complex models. Based on our experiments using real-world data\nfrom a large-scale commercial system, the suggested approach is able to\nsignificantly reduce the required number of annotations, while improving the\ngeneralization on unseen out-of-domain skills.\n