Reconstruct, Rasterize and Backprop: Dense shape and pose estimation from a single image

This paper presents a new system to obtain dense object reconstructions along\nwith 6-DoF poses from a single image. Geared towards high fidelity\nreconstruction, several recent approaches leverage implicit surface\nrepresentations and deep neural networks to estimate a 3D mesh of an object,\ngiven a single image. However, all such approaches recover only the shape of an\nobject; the reconstruction is often in a canonical frame, unsuitable for\ndownstream robotics tasks. To this end, we leverage recent advances in\ndifferentiable rendering (in particular, rasterization) to close the loop with\n3D reconstruction in camera frame. We demonstrate that our approach---dubbed\nreconstruct, rasterize and backprop (RRB) achieves significantly lower pose\nestimation errors compared to prior art, and is able to recover dense object\nshapes and poses from imagery. We further extend our results to an (offline)\nsetup, where we demonstrate a dense monocular object-centric egomotion\nestimation system.\n