Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

Differentiable rendering has paved the way to training neural networks to\nperform "inverse graphics" tasks such as predicting 3D geometry from monocular\nphotographs. To train high performing models, most of the current approaches\nrely on multi-view imagery which are not readily available in practice. Recent\nGenerative Adversarial Networks (GANs) that synthesize images, in contrast,\nseem to acquire 3D knowledge implicitly during training: object viewpoints can\nbe manipulated by simply manipulating the latent codes. However, these latent\ncodes often lack further physical interpretation and thus GANs cannot easily be\ninverted to perform explicit 3D reasoning. In this paper, we aim to extract and\ndisentangle 3D knowledge learned by generative models by utilizing\ndifferentiable renderers. Key to our approach is to exploit GANs as a\nmulti-view data generator to train an inverse graphics network using an\noff-the-shelf differentiable renderer, and the trained inverse graphics network\nas a teacher to disentangle the GAN's latent code into interpretable 3D\nproperties. The entire architecture is trained iteratively using cycle\nconsistency losses. We show that our approach significantly outperforms\nstate-of-the-art inverse graphics networks trained on existing datasets, both\nquantitatively and via user studies. We further showcase the disentangled GAN\nas a controllable 3D "neural renderer", complementing traditional graphics\nrenderers.\n