Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitation

Knowledge Distillation (KD) is a widely-used technology to inherit\ninformation from cumbersome teacher models to compact student models,\nconsequently realizing model compression and acceleration. Compared with image\nclassification, object detection is a more complex task, and designing specific\nKD methods for object detection is non-trivial. In this work, we elaborately\nstudy the behaviour difference between the teacher and student detection\nmodels, and obtain two intriguing observations: First, the teacher and student\nrank their detected candidate boxes quite differently, which results in their\nprecision discrepancy. Second, there is a considerable gap between the feature\nresponse differences and prediction differences between teacher and student,\nindicating that equally imitating all the feature maps of the teacher is the\nsub-optimal choice for improving the student's accuracy. Based on the two\nobservations, we propose Rank Mimicking (RM) and Prediction-guided Feature\nImitation (PFI) for distilling one-stage detectors, respectively. RM takes the\nrank of candidate boxes from teachers as a new form of knowledge to distill,\nwhich consistently outperforms the traditional soft label distillation. PFI\nattempts to correlate feature differences with prediction differences, making\nfeature imitation directly help to improve the student's accuracy. On MS COCO\nand PASCAL VOC benchmarks, extensive experiments are conducted on various\ndetectors with different backbones to validate the effectiveness of our method.\nSpecifically, RetinaNet with ResNet50 achieves 40.4% mAP in MS COCO, which is\n3.5% higher than its baseline, and also outperforms previous KD methods.\n