End-to-End Continuous Speech Emotion Recognition in Real-life Customer Service Call Center Conversations

Speech Emotion recognition (SER) in call center conversations has emerged as a valuable tool for assessing the quality of interactions between clients and agents. In contrast to controlled laboratory environments, real-life conversations take place under uncontrolled conditions and are subject to contextual factors that influence the expression of emotions. In this paper, we present our approach to constructing a large-scale real-life dataset (CusEmo) for continuous SER in customer service call center conversations. We adopted the dimensional approach for continuous emotion annotation while incorporating contextual information, such as the metadata of the interlocutors and the context of the call. The study also addresses the challenges encountered during the application of the End-to-End (E2E) SER system to the dataset, including determining the appropriate label sampling rate and input segment length, as well as integrating contextual information (interlocutor’s gender and empathy level) with different weights using multi-task learning. The result shows that incorporating the empathy level information can slightly improve the performance of the E2E SER model.