Augmenting the Social Presence of Interactive Characters Using Real-time Speech Recognition

Recently, in addition to games, interaction with real virtual agents has been used for job training. Furthermore, multimodal dialog using both voice and gesture recognition is used. However, there is a challenge with conventional batch processing speech recognition: it is difficult to make the virtual agent respond while the user is speaking. Therefore, in this study, we confirm the possibility of enhancing social presence by generating responses using real-time speech recognition in spoken dialog with characters. While the user speaks, by extracting keywords from the speech content in real-time, we generate the emotional response of the virtual agent. The experimental results demonstrate that our approach improves social presence compared with conventional spoken dialog.