Improved knowledge distillation by utilizing backward pass knowledge in neural networks

Knowledge distillation (KD) is one of the prominent techniques for model\ncompression. In this method, the knowledge of a large network (teacher) is\ndistilled into a model (student) with usually significantly fewer parameters.\nKD tries to better-match the output of the student model to that of the teacher\nmodel based on the knowledge extracts from the forward pass of the teacher\nnetwork. Although conventional KD is effective for matching the two networks\nover the given data points, there is no guarantee that these models would match\nin other areas for which we do not have enough training samples. In this work,\nwe address that problem by generating new auxiliary training samples based on\nextracting knowledge from the backward pass of the teacher in the areas where\nthe student diverges greatly from the teacher. We compute the difference\nbetween the teacher and the student and generate new data samples that maximize\nthe divergence. This is done by perturbing data samples in the direction of the\ngradient of the difference between the student and the teacher. Augmenting the\ntraining set by adding this auxiliary improves the performance of KD\nsignificantly and leads to a closer match between the student and the teacher.\nUsing this approach, when data samples come from a discrete domain, such as\napplications of natural language processing (NLP) and language understanding,\nis not trivial. However, we show how this technique can be used successfully in\nsuch applications. We evaluated the performance of our method on various tasks\nin computer vision and NLP domains and got promising results.\n