Enhancing Speech Emotion Recognition through Segmental Average Pooling of Self-Supervised Learning Features

Speech Emotion Recognition (SER) involves analyzing human emotions conveyed through vocal expressions. Recently, self-supervised learning (SSL) has emerged as a powerful tool for SER by enabling the extraction of meaningful representations from large-scale unlabeled audio datasets, significantly reducing the need for manual labeling. However, most SSLbased SER methods employ Global Average Pooling (GAP) to derive a single audio representation, which unintentionally treats all segments, including both verbal and non-verbal, as equally important. This approach can result in the dilution of crucial emotional cues by irrelevant non-verbal segments, thereby limiting the effectiveness of the model. To address this challenge, we introduce Segmental Average Pooling (SAP), a novel pooling technique designed to selectively emphasize informative verbal segments while disregarding non-verbal parts. By combining both GAP and SAP for SSL features, our method is able to integrate the overall context from GAP with more focused segmental information from SAP, thereby capturing a richer representation of emotional cues. This dual-pooling strategy leads to significant improvements in SER performance, as demonstrated by our experiments, which achieved state-of-the-art results on the IEMOCAP dataset for English, and superior performance on the KEMDy19 dataset for Korean, in terms of both unweighted and weighted accuracies.