Multi-agent trajectory forecasting in autonomous driving requires an agent to\naccurately anticipate the behaviors of the surrounding vehicles and\npedestrians, for safe and reliable decision-making. Due to partial\nobservability in these dynamical scenes, directly obtaining the posterior\ndistribution over future agent trajectories remains a challenging problem. In\nrealistic embodied environments, each agent's future trajectories should be\nboth diverse since multiple plausible sequences of actions can be used to reach\nits intended goals, and admissible since they must obey physical constraints\nand stay in drivable areas. In this paper, we propose a model that synthesizes\nmultiple input signals from the multimodal world|the environment's scene\ncontext and interactions between multiple surrounding agents|to best model all\ndiverse and admissible trajectories. We compare our model with strong baselines\nand ablations across two public datasets and show a significant performance\nimprovement over previous state-of-the-art methods. Lastly, we offer new\nmetrics incorporating admissibility criteria to further study and evaluate the\ndiversity of predictions. Codes are at: https://github.com/kami93/CMU-DATF.\n