Predicting the future trajectory of a moving agent can be easy when the past\ntrajectory continues smoothly but is challenging when complex interactions with\nother agents are involved. Recent deep learning approaches for trajectory\nprediction show promising performance and partially attribute this to\nsuccessful reasoning about agent-agent interactions. However, it remains\nunclear which features such black-box models actually learn to use for making\npredictions. This paper proposes a procedure that quantifies the contributions\nof different cues to model performance based on a variant of Shapley values.\nApplying this procedure to state-of-the-art trajectory prediction methods on\nstandard benchmark datasets shows that they are, in fact, unable to reason\nabout interactions. Instead, the past trajectory of the target is the only\nfeature used for predicting its future. For a task with richer social\ninteraction patterns, on the other hand, the tested models do pick up such\ninteractions to a certain extent, as quantified by our feature attribution\nmethod. We discuss the limits of the proposed method and its links to causality\n