Unmasking Communication Partners: A Low-Cost AI Solution for Digitally Removing Head-Mounted Displays in VR-Based Telepresence

Face-to-face conversation in Virtual Reality (VR) is a challenge when\nparticipants wear head-mounted displays (HMD). A significant portion of a\nparticipant's face is hidden and facial expressions are difficult to perceive.\nPast research has shown that high-fidelity face reconstruction with personal\navatars in VR is possible under laboratory conditions with high-cost hardware.\nIn this paper, we propose one of the first low-cost systems for this task which\nuses only open source, free software and affordable hardware. Our approach is\nto track the user's face underneath the HMD utilizing a Convolutional Neural\nNetwork (CNN) and generate corresponding expressions with Generative\nAdversarial Networks (GAN) for producing RGBD images of the person's face. We\nuse commodity hardware with low-cost extensions such as 3D-printed mounts and\nminiature cameras. Our approach learns end-to-end without manual intervention,\nruns in real time, and can be trained and executed on an ordinary gaming\ncomputer. We report evaluation results showing that our low-cost system does\nnot achieve the same fidelity of research prototypes using high-end hardware\nand closed source software, but it is capable of creating individual facial\navatars with person-specific characteristics in movements and expressions.\n