Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

The goal of perception for autonomous vehicles is to extract semantic\nrepresentations from multiple sensors and fuse these representations into a\nsingle "bird's-eye-view" coordinate frame for consumption by motion planning.\nWe propose a new end-to-end architecture that directly extracts a\nbird's-eye-view representation of a scene given image data from an arbitrary\nnumber of cameras. The core idea behind our approach is to "lift" each image\nindividually into a frustum of features for each camera, then "splat" all\nfrustums into a rasterized bird's-eye-view grid. By training on the entire\ncamera rig, we provide evidence that our model is able to learn not only how to\nrepresent images but how to fuse predictions from all cameras into a single\ncohesive representation of the scene while being robust to calibration error.\nOn standard bird's-eye-view tasks such as object segmentation and map\nsegmentation, our model outperforms all baselines and prior work. In pursuit of\nthe goal of learning dense representations for motion planning, we show that\nthe representations inferred by our model enable interpretable end-to-end\nmotion planning by "shooting" template trajectories into a bird's-eye-view cost\nmap output by our network. We benchmark our approach against models that use\noracle depth from lidar. Project page with code:\nhttps://nv-tlabs.github.io/lift-splat-shoot .\n