Although reinforcement learning has been successfully applied in many domains\nin recent years, we still lack agents that can systematically generalize. While\nrelational inductive biases that fit a task can improve generalization of RL\nagents, these biases are commonly hard-coded directly in the agent's neural\narchitecture. In this work, we show that we can incorporate relational\ninductive biases, encoded in the form of relational graphs, into agents. Based\non this insight, we propose Grid-to-Graph (GTG), a mapping from grid structures\nto relational graphs that carry useful spatial relational inductive biases when\nprocessed through a Relational Graph Convolution Network (R-GCN). We show that,\nwith GTG, R-GCNs generalize better both in terms of in-distribution and\nout-of-distribution compared to baselines based on Convolutional Neural\nNetworks and Neural Logic Machines on challenging procedurally generated\nenvironments and MinAtar. Furthermore, we show that GTG produces agents that\ncan jointly reason over observations and environment dynamics encoded in\nknowledge bases.\n