How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking

Attribution methods assess the contribution of inputs to the model\nprediction. One way to do so is erasure: a subset of inputs is considered\nirrelevant if it can be removed without affecting the prediction. Though\nconceptually simple, erasure's objective is intractable and approximate search\nremains expensive with modern deep NLP models. Erasure is also susceptible to\nthe hindsight bias: the fact that an input can be dropped does not mean that\nthe model `knows' it can be dropped. The resulting pruning is over-aggressive\nand does not reflect how the model arrives at the prediction. To deal with\nthese challenges, we introduce Differentiable Masking. DiffMask learns to\nmask-out subsets of the input while maintaining differentiability. The decision\nto include or disregard an input token is made with a simple model based on\nintermediate hidden layers of the analyzed model. First, this makes the\napproach efficient because we predict rather than search. Second, as with\nprobing classifiers, this reveals what the network `knows' at the corresponding\nlayers. This lets us not only plot attribution heatmaps but also analyze how\ndecisions are formed across network layers. We use DiffMask to study BERT\nmodels on sentiment classification and question answering.\n