Scene information plays a crucial role in trajectory forecasting systems for autonomous driving by providing semantic clues and constraints on potential future paths of traffic agents. Prevalent trajectory prediction techniques often take High-Definition maps (HD maps) as part of the inputs to provide scene knowledge. Although HD maps offer accurate road information, they may suffer from the high cost of annotation or restrictions of law, which limits their widespread use. Therefore, it is crucial for trajectory prediction methods to generate reliable prediction results in mapless scenarios. In this paper, we tackle the problem of improving the consistency of predicted trajectories and the scene road topology when map information is unavailable during the test phase. To achieve this, we propose a universal knowledge distillation (KD) framework. This KD framework trains a map-based teacher network on samples with annotated HD maps and subsequently transfers the knowledge to a student mapless predictor through a two-fold knowledge distillation process. Experimental results show that our method stably improves prediction performance in test-time mapless situations on many widely used trajectory prediction baselines, and achieves state-of-the-art mapless prediction performances. Qualitative visualization results demonstrate that our approach helps infer unseen map information. Our solution is generalizable for common trajectory prediction networks and datasets.