Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients

In this work we are the first to present an offline policy gradient method\nfor learning imitative policies for complex urban driving from a large corpus\nof real-world demonstrations. This is achieved by building a differentiable\ndata-driven simulator on top of perception outputs and high-fidelity HD maps of\nthe area. It allows us to synthesize new driving experiences from existing\ndemonstrations using mid-level representations. Using this simulator we then\ntrain a policy network in closed-loop employing policy gradients. We train our\nproposed method on 100 hours of expert demonstrations on urban roads and show\nthat it learns complex driving policies that generalize well and can perform a\nvariety of driving maneuvers. We demonstrate this in simulation as well as\ndeploy our model to self-driving vehicles in the real-world. Our method\noutperforms previously demonstrated state-of-the-art for urban driving\nscenarios -- all this without the need for complex state perturbations or\ncollecting additional on-policy data during training. We make code and data\npublicly available.\n