Development of a Multimodal Dialogue System in a VR Environment

We investigated the impact of displaying a conversational agent in three dimensions within a virtual reality (VR) space for multimodal dialogue systems using generative AI. Conventional systems present the agent on a 2D screen, which imposes limits such as unnatural interpersonal distance and a lack of depth, preventing the interaction from approaching human-human dialogue. To address these issues, we built a life-sized agent in a Unity-based VR environment where the user wearing a head-mounted display (HMD). A comparative experiment between the VR system and a baseline 2D system showed that the VR system achieved higher scores for human-likeness and intention to continue use.