FAF: A novel multimodal emotion recognition approach integrating face, body and text

: Multimodal emotion analysis performed better in emotion recognition depending on more comprehensive emotional clues and multimodal emotion dataset. In this paper, we developed a large multimodal emotion dataset, named “HED” dataset, to facilitate the emotion recognition task, and accordingly propose a multimodal emotion recognition method. Specifically, the “HED” dataset contains happy, sad, disgust, angry and scared emotion-aligned face, body and text samples, which are much larger than existing datasets. Moreover, the emotion labels were correspondingly attached to those samples by strictly following a standard psychological paradigm. To promote recognition accuracy, “Feature After Feature” framework was used to explore crucial emotional information from the aligned face-body-text samples. For the images, a residual network was used for feature extraction. To understand the text, we use the BERT word vector for feature selection. To fuse all the emotion clues, the image features were initially fused, and the fused feature vectors were stitched with text features to form the combined features. Then, the combined features are further explored by using convolutional layers to explore the high-level complementary information among the multimodal information, and the attention mechanism was introduced to give different weights and improve the performance of emotion recognition in the fused modality. We employ various benchmarks to evaluate the “HED” dataset and compare the performance with our method. The results show that the five-classification accuracy of the proposed multimodal fusion method is about 83.75%, and the performance is improved by 1.83%, 9.38%, and 21.62% respectively compared with that of individual modalities. The complementarity between each channel is effectively used to improve the performance of emotion recognition. We had also established a multimodal online emotion prediction platform, aiming to provide free emotion prediction to more users.