We develop a deep generative model built on a fully differentiable simulator\nfor multi-agent trajectory prediction. Agents are modeled with conditional\nrecurrent variational neural networks (CVRNNs), which take as input an\nego-centric birdview image representing the current state of the world and\noutput an action, consisting of steering and acceleration, which is used to\nderive the subsequent agent state using a kinematic bicycle model. The full\nsimulation state is then differentiably rendered for each agent, initiating the\nnext time step. We achieve state-of-the-art results on the INTERACTION dataset,\nusing standard neural architectures and a standard variational training\nobjective, producing realistic multi-modal predictions without any ad-hoc\ndiversity-inducing losses. We conduct ablation studies to examine individual\ncomponents of the simulator, finding that both the kinematic bicycle model and\nthe continuous feedback from the birdview image are crucial for achieving this\nlevel of performance. We name our model ITRA, for "Imagining the Road Ahead".\n