Design and Analysis of an Automatic UI Adaptation Framework from Multimodal Emotion Recognition using an RGB-D Sensor

Abstract

Abstract This work presents the design and analysis of an Automatic adaptive user interface (AUI) that uses a novel solution for the recognition of the emotional state of a user through both facial expressions and body posture from an RGB-D sensor. Six basic emotions are recognized through facial expressions in addition to the physiological state recognized through the body posture. The facial expressions and body posture are acquired in real-time from a Kinect sensor. A scoring system is used to improve the recognition by minimizing the confusion between the different emotions. The implemented solution achieves an accuracy rate of above 90%. The recognized emotion is then used to derive an Automatic AUI where the user is provided the help automatically and can use speech commands to modify the User Interface (UI). A comprehensive user study is performed to compare the usability of the Automatic AUI with a manual system. Results show that even though the Automatic AUI is quantitatively slower and thus not as efficient, it results in a similar effectiveness and error safety compared to a manual system. In addition, the Automatic AUI results in a significantly positive user experience compared to a manual system.