High Resolution Zero-Shot Domain Adaptation of Synthetically Rendered Face Images

Generating photorealistic images of human faces at scale remains a\nprohibitively difficult task using computer graphics approaches. This is\nbecause these require the simulation of light to be photorealistic, which in\nturn requires physically accurate modelling of geometry, materials, and light\nsources, for both the head and the surrounding scene. Non-photorealistic\nrenders however are increasingly easy to produce. In contrast to computer\ngraphics approaches, generative models learned from more readily available 2D\nimage data have been shown to produce samples of human faces that are hard to\ndistinguish from real data. The process of learning usually corresponds to a\nloss of control over the shape and appearance of the generated images. For\ninstance, even simple disentangling tasks such as modifying the hair\nindependently of the face, which is trivial to accomplish in a computer\ngraphics approach, remains an open research question. In this work, we propose\nan algorithm that matches a non-photorealistic, synthetically generated image\nto a latent vector of a pretrained StyleGAN2 model which, in turn, maps the\nvector to a photorealistic image of a person of the same pose, expression,\nhair, and lighting. In contrast to most previous work, we require no synthetic\ntraining data. To the best of our knowledge, this is the first algorithm of its\nkind to work at a resolution of 1K and represents a significant leap forward in\nvisual realism.\n