ScaleKD: Strong Vision Transformers Could Be Excellent Teachers

Summary

This paper introduces a novel knowledge distillation method called ScaleKD. The method aims to leverage well pre-trained vision transformer models as teacher models for a variety of student model architectures.the authors first adopt a cross attention projector to align student features with the teacher's. Then, a dual-view feature mimicking module and a teacher parameter perception module are used to achieve better knowledge transfer. Extensive experiments demonstrate the effectiveness of ScaleKD across various tasks and model types.

Strengths

1. The paper is well-written and well-structured. 2. The authors provide extensive experimental results and analyses that demonstrate the effectiveness of the proposed method.

Weaknesses

1. The paper's motivation could be strengthened. As highlighted in previous research [1], a stronger teacher model does not always equate to a better teacher. The necessity of adapting a ViT teacher for training a CNN student needs further justification. Including a comparison between ViT teachers and CNN teachers would provide better support. 2. The teacher parameter perception (TPP) module's cross-architecture KD paradigm is similar to techniques used in existing research [2,3]. While this does not diminish the novelty of other aspects of the paper, these related works should be properly discussed. 3. The authors critique the focus on "evaluation on small datasets with non-mainstream student models" in existing works. Some relevant papers about these issues should also be discussed, such as [4,5]. [1] Mirzadeh, Seyed Iman, et al. "Improved knowledge distillation via teacher assistant." Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020. [2] Wang, Jiabao, et al. "CrossKD: Cross-head knowledge distillation for object detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. [3] Bai, Haoli, et al. "Few shot network compression via cross distillation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020. [4] Hao, Zhiwei, et al. "Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale." arXiv preprint arXiv:2305.15781 (2023). [5] Stanton, Samuel, et al. "Does knowledge distillation really work?." Advances in Neural Information Processing Systems 34 (2021): 6906-6919.

Questions

In the dual-view feature mimicking module, the direct component is omitted during the alternative feature mimicking process. However, the remaining non-direct features are duplicated in the normal matching process (as shown in the upper path in Figure 2b). How does the removal of the non-direct component in the upper path affect performance?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper presents a new knowledge distillation method, named ScaleKD. Previous works mostly use CNNs to distill vision transformers. However, how to use vision transformers to distill CNNs is less explored. This paper shows that pretrained vision transformers are good teachers for other types of student models. The cores of the proposed method are three main components, including the cross-attention projector, dual view feature mimicking, and teacher parameter perception. The motivation of this paper is clear and the proposed method is also interesting. The authors propose to do model distillation from three aspects, which have been proven useful in the experiment section.

Strengths

- The idea of this paper is interesting and the novelty is significant. Unlike previous KD methods, this paper proposes a new way to do knowledge distillation. - The presentation of this paper is also good. It seems that the proposed method is easy to follow. - Experimental results are good. Compared to previous knowledge distillation methods, the results shown in this paper improve them. In addition, the authors also provide detailed analysis on the importance of each component.

Weaknesses

- Reading this paper is too tedious. The paragraph is too long in the introduction section. It is difficult to capture the important content. - The proposed method consists of three parts, which make it look complicated. - From Table 2, it seems that when the teacher models' scale increases, there is improvement. However, according to my knowledge, previous KD methods mostly fail to do this. Can the authors explain why the proposed method can achieve this? I think this is important for the KD community to design better KD methods. - Though exploring how to use vision transformers as teachers is important, I am also curious about another thing. Have the authors used CNNs as teacher models? This is what most previous works did.

Questions

I think the authors should elaborate more on why scaled teacher models help in the proposed method. I would like to see some analysis on this.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have included limitations in the main paper.

Summary

This paper focuses on whether the pre-trained vision transformer models could be used as teachers to distilling knowledges to heterogeneous neural network architectures. The proposed ScaleKD aims to solve three problems including 1) feature computing paradigm different, 2) model scale differences, and 3) knowledge density differences. Extensive experiments are conducted with different neural networks, including CNN, ViT, and MLP on image classification task. This paper also shows that when scaling up the size of teacher models or their pre-training datasets, ScaleKD showcases larger gains to the student models.

Strengths

1. The difficulties of transferring knowledge between different architectures are well summarized, including the differences in feature computing paradigm, differences in model scale, and differences in knowledge density. 2. All the figures, as well as the writing are clear and easy to follow. 3. The extensive experiments show the effectiveness of the proposed ScaleKD.

Weaknesses

1. The related work section should be improved to discuss the differences between the proposed ScaleKD and existing works. The CAP, DFM, and TPP should be compared with existing works to show the originality and novelty. The references include: [1] ViTKD: Feature-based Knowledge Distillation for Vision Transformers, CVPR 2024 [2] Prefallkd: Pre-impact fall detection via cnn-vit knowledge distillation, ICASSP 2023 [3] Distilling efficient vision transformers from cnns for semantic segmentation, Arxiv 2023 [4] A good student is cooperative and reliable: CNN-transformer collaborative learning for semantic segmentation, ICCV 2023 2. The experiments are conducted with ViT, SwinT, ResNet, MobileNet, ConvNeXt, MLP-Mixer. There are several types of CNN, however the claims in Abstract mentions that "Our method can train student backbones that span across a variety of CNN, MLP and ViT." More backbones should be included, or the description in abstract is over-claimed. 3. A question is the ScaleKD includes the MLP and ViT layers in CAP, does it mean the improved KD performance comes from such architecture update? Is the proposed method achieve KD with adding such layers to augment the student model? This is the main concern, please give more discussion.

Questions

Please refer to the weakness.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper concentrates on the distillation of knowledge from a large-scale, pre-trained, ViT-based teacher model to heterogeneous architectures. It incorporates three distinct designs: a) a Cross Attention Projector (CAP), which serves as the fundamental design that bridges the structural disparity between a non-ViT model and the ViT teacher; b) a Dual-View Feature Mimicking and a Teacher Parameter Perception module, both of which are constructed on top of the CAP to facilitate the distillation process. The effectiveness of the proposed methodology is validated through extensive experimentation.

Strengths

- The proposed distillation technique is effective for heterogeneous architectures with a ViT-based teacher. - The conducted experiments are comprehensive, providing solid validation for the effectiveness of the method. - The proposed CAP structure is a good contribution which is feasible for various student architectures.

Weaknesses

__Presentation__: The connection between DFM and TPP and their respective motivations is unclear. Specifically, it’s unclear how the modifications of DFM and TPP specifically address the issues of model scale and knowledge density. __Effects of CAP__: While the introduced CAP appears to be a promising design, its effectiveness is only demonstrated within the context of the ScaleKD framework. It would be interesting to see if this architecture could be integrated with other heterogeneous architecture methods, such as OFA, to replace traditional projectors. __Effects of DFM__: The ablation study of DFM merely indicates that the filtering operation is beneficial to CAP-based distillation. It raises the question of whether this strategy is also compatible with other feature alignment architectures, such as linear head or convolutional head? __Effects of TPP__: The design of TPP seems to be at odds with its stated purpose. Given that different network architectures, such as CNNs or ViTs, are known to have different knowledge preferences, it’s questionable whether aligning the student’s parameters with the teacher’s under a heterogeneous architecture is an effective approach. Furthermore, it’s unclear why using the last layer of the teacher as an additional head function for distillation would aid in alignment in the parameter space. Empirical evidence (such as CKA visualization) or theoretical justification is needed to demonstrate whether TPP can encourage different network architectures to exhibit similar learning behavior. __Experiments__: Some details of the experiments, such as the teacher information in Tables 6-8, are missing from the presented tables.

Questions

please refer to the weakness part.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The limitations are properly discussed.

Final rating

Thanks for the reponses to my concerns. Basically, the authors have solved my concerns. In addition, all the other reviewers also recognize the contributions this paper made. I would like to keep my original rating unchanged.

Post rebuttal comment

Thanks for the response, which addresses my concerns. I will increase my rating accordingly. However, I still think the statement of 'parameter space' is somewhat ambiguous. I suggest the authors include more clarification in the final version.

Thanks for the Recognition of Our Rebuttal

Thank you so much for the recognition of our responses. We are glad to see that you tend to accept our paper. We will make more efforts to improve our paper further. Many thanks for your constructive comments, time and patience.

Thanks for Your Support of Our Paper

Thank you so much for recognizing our rebuttal and setting the final rate as "Accept" to support our paper. We will make more efforts to improve our paper further. Many thanks for your constructive comments, time and patience.