Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization

Blind Perspective-n-Point (PnP) is the problem of estimating the position and\norientation of a camera relative to a scene, given 2D image points and 3D scene\npoints, without prior knowledge of the 2D-3D correspondences. Solving for pose\nand correspondences simultaneously is extremely challenging since the search\nspace is very large. Fortunately it is a coupled problem: the pose can be found\neasily given the correspondences and vice versa. Existing approaches assume\nthat noisy correspondences are provided, that a good pose prior is available,\nor that the problem size is small. We instead propose the first fully\nend-to-end trainable network for solving the blind PnP problem efficiently and\nglobally, that is, without the need for pose priors. We make use of recent\nresults in differentiating optimization problems to incorporate geometric model\nfitting into an end-to-end learning framework, including Sinkhorn, RANSAC and\nPnP algorithms. Our proposed approach significantly outperforms other methods\non synthetic and real data.\n