SimNet: Learning Reactive Self-driving Simulations from Real-world Observations

In this work, we present a simple end-to-end trainable machine learning\nsystem capable of realistically simulating driving experiences. This can be\nused for the verification of self-driving system performance without relying on\nexpensive and time-consuming road testing. In particular, we frame the\nsimulation problem as a Markov Process, leveraging deep neural networks to\nmodel both state distribution and transition function. These are trainable\ndirectly from the existing raw observations without the need for any\nhandcrafting in the form of plant or kinematic models. All that is needed is a\ndataset of historical traffic episodes. Our formulation allows the system to\nconstruct never seen scenes that unfold realistically reacting to the\nself-driving car's behaviour. We train our system directly from 1,000 hours of\ndriving logs and measure both realism, reactivity of the simulation as the two\nkey properties of the simulation. At the same time, we apply the method to\nevaluate the performance of a recently proposed state-of-the-art ML planning\nsystem trained from human driving logs. We discover this planning system is\nprone to previously unreported causal confusion issues that are difficult to\ntest by non-reactive simulation. To the best of our knowledge, this is the\nfirst work that directly merges highly realistic data-driven simulations with a\nclosed-loop evaluation for self-driving vehicles. We make the data, code, and\npre-trained models publicly available to further stimulate simulation\ndevelopment.\n