Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild

We present a novel 3D pose refinement approach based on differentiable\nrendering for objects of arbitrary categories in the wild. In contrast to\nprevious methods, we make two main contributions: First, instead of comparing\nreal-world images and synthetic renderings in the RGB or mask space, we compare\nthem in a feature space optimized for 3D pose refinement. Second, we introduce\na novel differentiable renderer that learns to approximate the rasterization\nbackward pass from data instead of relying on a hand-crafted algorithm. For\nthis purpose, we predict deep cross-domain correspondences between RGB images\nand 3D model renderings in the form of what we call geometric correspondence\nfields. These correspondence fields serve as pixel-level gradients which are\nanalytically propagated backward through the rendering pipeline to perform a\ngradient-based optimization directly on the 3D pose. In this way, we precisely\nalign 3D models to objects in RGB images which results in significantly\nimproved 3D pose estimates. We evaluate our approach on the challenging Pix3D\ndataset and achieve up to 55% relative improvement compared to state-of-the-art\nrefinement methods in multiple metrics.\n