Predicting the motion of multiple agents is necessary for planning in dynamic\nenvironments. This task is challenging for autonomous driving since agents\n(e.g. vehicles and pedestrians) and their associated behaviors may be diverse\nand influence one another. Most prior work have focused on predicting\nindependent futures for each agent based on all past motion, and planning\nagainst these independent predictions. However, planning against independent\npredictions can make it challenging to represent the future interaction\npossibilities between different agents, leading to sub-optimal planning. In\nthis work, we formulate a model for predicting the behavior of all agents\njointly, producing consistent futures that account for interactions between\nagents. Inspired by recent language modeling approaches, we use a masking\nstrategy as the query to our model, enabling one to invoke a single model to\npredict agent behavior in many ways, such as potentially conditioned on the\ngoal or full future trajectory of the autonomous vehicle or the behavior of\nother agents in the environment. Our model architecture employs attention to\ncombine features across road elements, agent interactions, and time steps. We\nevaluate our approach on autonomous driving datasets for both marginal and\njoint motion prediction, and achieve state of the art performance across two\npopular datasets. Through combining a scene-centric approach, agent permutation\nequivariant model, and a sequence masking strategy, we show that our model can\nunify a variety of motion prediction tasks from joint motion predictions to\nconditioned prediction.\n