Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

Natural images are projections of 3D objects on a 2D image plane. While\nstate-of-the-art 2D generative models like GANs show unprecedented quality in\nmodeling the natural image manifold, it is unclear whether they implicitly\ncapture the underlying 3D object structures. And if so, how could we exploit\nsuch knowledge to recover the 3D shapes of objects in the images? To answer\nthese questions, in this work, we present the first attempt to directly mine 3D\ngeometric cues from an off-the-shelf 2D GAN that is trained on RGB images only.\nThrough our investigation, we found that such a pre-trained GAN indeed contains\nrich 3D knowledge and thus can be used to recover 3D shape from a single 2D\nimage in an unsupervised manner. The core of our framework is an iterative\nstrategy that explores and exploits diverse viewpoint and lighting variations\nin the GAN image manifold. The framework does not require 2D keypoint or 3D\nannotations, or strong assumptions on object shapes (e.g. shapes are\nsymmetric), yet it successfully recovers 3D shapes with high precision for\nhuman faces, cats, cars, and buildings. The recovered 3D shapes immediately\nallow high-quality image editing like relighting and object rotation. We\nquantitatively demonstrate the effectiveness of our approach compared to\nprevious methods in both 3D shape reconstruction and face rotation. Our code is\navailable at https://github.com/XingangPan/GAN2Shape.\n