Multi-Branch Deep Radial Basis Function Networks for Facial Emotion Recognition

Emotion recognition (ER) from facial images is one of the landmark tasks in affective computing with major developments in the last decade. Initial efforts on ER relied on handcrafted features that were used to characterize facial images and then feed to standard predictive models. Recent methodologies comprise end-to-end trainable deep learning methods that simultaneously learn both, features and predictive model, where the most successful models are based on convolutional neural networks (CNNs). While these models have excelled at this task, they still fail at capturing local patterns that emerge in the learning process. We hypothesize these patterns could be captured by variants based on locally weighted learning. Specifically, in this paper we propose a CNN-based architecture enhanced with multiple branches formed by radial basis function (RBF) units that aims at exploiting local information at the final stage of the learning process (i.e., in the layers close to the output layer). Intuitively, these RBF units capture local patterns shared by similar instances using an intermediate representation, then the outputs of the RBFs are feed to a softmax layer that exploits this information to improve the predictive performance of the model. This feature could be particularly advantageous in ER as cultural / ethnicity differences may be potentially identified by the local units. We evaluate the proposed method in several ER datasets and show the proposed methodology achieves state-of-the-art performance in some of them, even when we adopt a pre-trained VGG-Face model as backbone. We show the proposed method outperforms consistently CNNs that do not have the proposed local learning component. Moreover, we found the proposed model is advantageous when training datasets are reduced and when merging images coming from different distributions (e.g., combining two ER datasets). We show it is the incorporation of local information what makes the proposed model competitive.