One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation

Summary

In this paper, the authors propose a new heterogeneous knowledge distillation approach. The core idea is to map the intermediate layer features of the network to a unified logit space to eliminate feature mismatches caused by different structures. The author conducts thorough experiments on distilling between CNN, ViT, and MLP networks. According to the experimental results in the paper, the proposed approach yields promising results.

Strengths

1. The experiments in this paper are comprehensive, considering distillation between various networks with different structures, and conducting experiments on both CIFAR-100 and ImageNet datasets. 2. The proposed method is reasonable, as projecting network features onto a latent space to avoid the alignment issue of distillation between networks with different structures may indeed lead to better results. 3. According to the experimental results, the proposed method achieves good performance in various experiments.

Weaknesses

1. The newly proposed method is very similar to deep supervision in that both involve adding an auxiliary head to the intermediate layer to learn the final output. The only difference is that deep supervision previously learned hard labels, while the proposed method learns soft labels from the teacher. However, this paper does not discuss the differences between this method and deep supervision, including theoretical and experimental results. 2. In some experimental settings, the improvement brought by the proposed method is very limited. And the comparison is not comprehensive. For example, in Table 1 and Table 2, OFD, Review, and CRD's results are missing. 3. In addition, in Table 1, some experiments are combined with FitNet while others are not. Although the authors have provided an explanation, it still feels strange that we cannot conclude that ResNet50's features are more applicable just because it is the most commonly used network. And why ResNet50 is not adopted as the teacher in Table 2?

Questions

Please address the problem in Weakness

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Potential negative societal is not applicable.

Summary

This paper introduces a new method to distill knowledge between heterogeneous models named OFD-KD. This paper proposes to project the intermediate features into logits for distillation. A new loss function is also introduced in this paper to adaptively enhance the target information. Extensive experiments verify the effectiveness of this method.

Strengths

1. This paper is easy to understand and clearly written. 2. This paper uses CKA to visualize the differences between CNN, VIT, and MLP. 3. Extensive experiments are conducted to verify the effectiveness of this method.

Weaknesses

1. The improvement is relatively minor in ImageNet-1K. The most improvement is around 0.1%-0.3% in Tables 1, 5, and 6. This method does not exhibit a clear advantage over other techniques. 2. The architecture employed in this work has been extensively explored. This form of intermediate logit supervision has been widely used in BYOT, DCM, DKS, and other methods. This study does not offer any significant novelty in the context of distillation. 3. Table 1 fails to compare the latest feature-based distillation method, particularly SemCKD, which is a method dedicated to heterogeneous distillation. Thus, the experimental comparisons presented in this paper lack meaningfulness. The authors should compare with the recent state-of-the-art hint-based methods. 4. The author's use of CKA comparison is unnecessary, given the obvious architectural differences between CNN, MLP, and VIT. The differences between these architectures have been discussed in many works. 5. Please provide the results of OFA during distillation between heterogeneous networks such as VGG, ResNet, ShuffleNet, and MobileNet.

Questions

1. In Tables 1 and 2, the results of many distillation methods are not as high as the baseline results, please explain your implementation details and why this is the case. 2. Is there any quantitative indicator to prove that the method proposed in this paper really bridges the gap of heterogeneous distillation?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

See weaknesses and questions.

Summary

This paper tackles the problem of cross-architecture distillation, that is, the teacher and the student in KD are of different model architectures. By using centered kernel alignment, the authors observe that features learned by models of different architectures shows significant feature divergence, indicating that previous hint-based methods are not suit for this task. To bridge this gap, the authors propose a simple yet effective one-for-all KD framework called OFA-KD. Specifically, they project intermediate features into an aligned latent space to discard architecture-specific information. And an adaptive target enhancement scheme is proposed to prevent the student from being disturbed by irrelevant information. The authors conduct experiments on CIFAR-100 and ImageNet-1k benchmarks with CNN, ViT and MLP architectures to demonstrate the effectiveness of the proposed method.

Strengths

Motivation: The motivation is clear. Cross-architecture distillation provides more feasible options for teacher models, as it may not always be possible to find a superior teacher model with a homogeneous architecture. Originality: Learning in an aligned latent space is the first application in cross-architecture distillation, and the adaptive target information enhancement loss is novel. Quality: Written of the paper is good. Sufficient experiments and ablation studies with other methods and the proposed variants are implemented. Clarity: The paper consists of text explanations and an illustration of the OFA-KD framework and the proposed loss. Significance: The paper solves the problem of cross-architecture distillation, expanding feasible options for teacher models in practice, and improves the accuracy of distilled student models.

Weaknesses

1. In the CKA analysis, it seems that when comparing features of models of the same architecture, the authors just using features of one model in both x-axis and y-axis, as the corresponding heatmaps are symmetry. If using two models, such as ResNet18 vs. ResNet34 or two ResNet18 trained with different initialization, would the results be different? 2. The authors propose to using an aligned latent space for cross-architecture distillation, and adopting the logits space as a special instance. I wonder whether there are any other possible choices of the space, and what will happen when using them. 3. Introduction of the adaptive target information enhancement loss is a bit complicated. Maybe it is possible to simplify the notations. 4. What is the principle of designing the branches, such as its architecture and layer number? Have the authors tried other branch designs? Additionally, there are some grammar mistakes and typos: - line 192, "slow" -> "slowly" - line 193, "enhance" -> "enhancement" - line 255, "additional" -> "addition" - line 257, "reports" -> "report" - line 299, "strengthen" -> "strength" - line 311, "methods" -> "method" - line 339, "disitllation" -> "distillation"

Questions

Please refer to "Weaknesses"

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes. Limitations and border impact are discussed in the conclusion.

Summary

This paper first demonstrates that there is significant feature divergence of the learned features between heterogeneous teacher and student models, which is a scenario rarely explored in previous knowledge distillation community. And the authors point out that the hint-based methods are ineffective in this cross-architecture distillation. They propose OFA-KD to improve the distillation performance between heterogeneous architectures. It first projects intermediate features into an aligned latent space (the logits space). In addition, they introduce an adaptive target enhancement scheme to prevent the student from being disturbed by irrelevant information. The extensive experiments with various architectures demonstrate the superiority of the OFA-KD framework.

Strengths

This paper studies KD with different architectures. It is an interesting attempt to build a generic framework for distilling students with arbitrary mainstream model architectures, i.e., CNN, ViT, and MLP. Experiment results in both the main paper and the supplementary material demonstrate the necessity of doing cross-architecture distillation. The performance improvement of OFA is remarkable. On ImageNet-1K, the maximum improvement is 0.8\%, and on CIFAR-100, the maximum improvement is 5.0\%, even compared with the most recent KD baselines, DIST and DKD.

Weaknesses

The additional branches will increase the training cost. I think the authors should give more analyses on this. The reported results of using res50 as the teacher on ImageNet-1K are obtained by using both FitNet and OFA-KD (Table 1). I think this is not a fair comparison. For example, what’s the result of FitNet + DKD/DIST? Missing some references also adopting multi-branch architecture: [A] Be your own teacher: Improve the performance of convolutional neural networks via self distillation, ICCV 2019 [B] Distillation-based training for multi-exit architectures, ICCV 2019 [C] MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks, arXiv 2019

Questions

The OFA result (ViT-B teacher) in Table 6 of the main paper seems inconsistent with that in Figure 5 of the supplementary material. The accuracy gain of the res50 student is 1.19\% in the main paper, while in the supplementary material, the accuracy gain is 1.47\%. Please check it. As there are additional branches introduced during the training procedure, it requires more computational resources to train the same student model than using traditional KD methods. Could the authors provide more details about the branches to illustrate the extra resources consumption? For example, I am interested in the number parameters and the FLOPs of the additional branches. Why does the paper choose to use depth-width convolutional layers for branches in CNN models, while ViT blocks are used in ViTs and MLPs? There are so many hyperparameters such as the scaling factor, the clip grad norm, and the \gamma value. How to choose these hyperparameters in practice?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes. Limitations and border impact are discussed in the conclusion.

Response

Thank you for your response. The rebuttal address most of my concerns. I have raised my score. I would be very glad to see the author release their code if this paper is accepted.