Forecasting long-term human motion is a challenging task due to the\nnon-linearity, multi-modality and inherent uncertainty in future trajectories.\nThe underlying scene and past motion of agents can provide useful cues to\npredict their future motion. However, the heterogeneity of the two inputs poses\na challenge for learning a joint representation of the scene and past\ntrajectories. To address this challenge, we propose a model based on grid\nrepresentations to forecast agent trajectories. We represent the past\ntrajectories of agents using binary 2-D grids, and the underlying scene as a\nRGB birds-eye view (BEV) image, with an agent-centric frame of reference. We\nencode the scene and past trajectories using convolutional layers and generate\ntrajectory forecasts using a Convolutional LSTM (ConvLSTM) decoder. Results on\nthe publicly available Stanford Drone Dataset (SDD) show that our model\noutperforms prior approaches and outputs realistic future trajectories that\ncomply with scene structure and past motion.\n