While supervised learning is widely used for perception modules in\nconventional autonomous driving solutions, scalability is hindered by the huge\namount of data labeling needed. In contrast, while end-to-end architectures do\nnot require labeled data and are potentially more scalable, interpretability is\nsacrificed. We introduce a novel architecture that is trained in a fully\nself-supervised fashion for simultaneous multi-step prediction of space-time\ncost map and road dynamics. Our solution replaces the manually designed cost\nfunction for motion planning with a learned high dimensional cost map that is\nnaturally interpretable and allows diverse contextual information to be\nintegrated without manual data labeling. Experiments on real world driving data\nshow that our solution leads to lower number of collisions and road violations\nin long planning horizons in comparison to baselines, demonstrating the\nfeasibility of fully self-supervised prediction without sacrificing either\nscalability or interpretability.\n