Graph neural networks (GNNs) have become a popular approach to integrating\nstructural inductive biases into NLP models. However, there has been little\nwork on interpreting them, and specifically on understanding which parts of the\ngraphs (e.g. syntactic trees or co-reference structures) contribute to a\nprediction. In this work, we introduce a post-hoc method for interpreting the\npredictions of GNNs which identifies unnecessary edges. Given a trained GNN\nmodel, we learn a simple classifier that, for every edge in every layer,\npredicts if that edge can be dropped. We demonstrate that such a classifier can\nbe trained in a fully differentiable fashion, employing stochastic gates and\nencouraging sparsity through the expected $L_0$ norm. We use our technique as\nan attribution method to analyze GNN models for two tasks -- question answering\nand semantic role labeling -- providing insights into the information flow in\nthese models. We show that we can drop a large proportion of edges without\ndeteriorating the performance of the model, while we can analyse the remaining\nedges for interpreting model predictions.\n