Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation

Standard registration algorithms need to be independently applied to each\nsurface to register, following careful pre-processing and hand-tuning.\nRecently, learning-based approaches have emerged that reduce the registration\nof new scans to running inference with a previously-trained model. In this\npaper, we cast the registration task as a surface-to-surface translation\nproblem, and design a model to reliably capture the latent geometric\ninformation directly from raw 3D face scans. We introduce Shape-My-Face (SMF),\na powerful encoder-decoder architecture based on an improved point cloud\nencoder, a novel visual attention mechanism, graph convolutional decoders with\nskip connections, and a specialized mouth model that we smoothly integrate with\nthe mesh convolutions. Compared to the previous state-of-the-art learning\nalgorithms for non-rigid registration of face scans, SMF only requires the raw\ndata to be rigidly aligned (with scaling) with a pre-defined face template.\nAdditionally, our model provides topologically-sound meshes with minimal\nsupervision, offers faster training time, has orders of magnitude fewer\ntrainable parameters, is more robust to noise, and can generalize to previously\nunseen datasets. We extensively evaluate the quality of our registrations on\ndiverse data. We demonstrate the robustness and generalizability of our model\nwith in-the-wild face scans across different modalities, sensor types, and\nresolutions. Finally, we show that, by learning to register scans, SMF produces\na hybrid linear and non-linear morphable model. Manipulation of the latent\nspace of SMF allows for shape generation, and morphing applications such as\nexpression transfer in-the-wild. We train SMF on a dataset of human faces\ncomprising 9 large-scale databases on commodity hardware.\n