Over the last few years, we have witnessed tremendous progress on many\nsubtasks of autonomous driving, including perception, motion forecasting, and\nmotion planning. However, these systems often assume that the car is accurately\nlocalized against a high-definition map. In this paper we question this\nassumption, and investigate the issues that arise in state-of-the-art autonomy\nstacks under localization error. Based on our observations, we design a system\nthat jointly performs perception, prediction, and localization. Our\narchitecture is able to reuse computation between both tasks, and is thus able\nto correct localization errors efficiently. We show experiments on a\nlarge-scale autonomy dataset, demonstrating the efficiency and accuracy of our\nproposed approach.\n