What Knowledge Gets Distilled in Knowledge Distillation?

Summary

This paper argues that distillation implicitly transfers many different behaviors from the teacher to the student, even though these are not explicitly matched via the distillation objective. These include: 1. The saliency map computed via Grad-CAM 2. Specific adversarial examples of the teacher 3. Invariance that the teacher has learned via data augmentations on color jitter, bridghtness, contrast etc., 4. Performance on Out-of-Distribution (OoD) data The distillation approaches studied here are: - the standard K-divergence based loss, - the Hint loss which matches embeddings and - contrastive representation KD The architectures covered are VGG, ResNet, ViT, and Swin (although not all these architectures are used in all experiments). All main experiments are on ImageNet. A few additional experiments are on MNIST. The paper also provides an intuitive argument for why such implicit knowledge is transferred: the distilled student learns the specific decision boundary of the teacher. This is verified for an MNIST setting where the network via some existing techniques for comparing decision boundaries. The paper argues that these findings have practical implications in (a) allowing the student to generalizing to new domains, (b) but also in transferring harmful biases to the student.

Strengths

1. The questions explored in the paper are novel and interesting. This is a new perspective on what type of knowledge may be inadvertently transferred from the teacher to the student. 2. The paper covers various types of knowledge (namely, saliency maps, OOD, adversarial examples, invariances) and there is clear evidence across the board that the student typically imitates the teacher non-trivially. 3. The paper covers three fundamentally different types of distillation losses, which strengthens their claim. 4. The experiments are designed carefully. In all cases, an "independent" student is considered as a baseline so as to judge whether the distilled-student is indeed copying the teacher.

Weaknesses

1. My main concern is that all the main experiments appear to be limited to one (ImageNet) dataset. While this is certainly a non-trivial and sufficiently realistic dataset, one should be wary of drawing observations from a single dataset. Either a language dataset or another image dataset (say, CIFAR100) would have been ideal. But I understand that it would be hard to add these during the rebuttal. 2. The argument for "why does KD transfer such implicit knowledge" is not compelling and appears to be somewhat circular. The argument given is that "KD transfers various implicit behaviors of the teacher because the student inherits the decision boundary". The key puzzle however is why the KD objective is able transfer all of the decision boundary, and that too on OoD and adversarial regions the student isn't trained on. Overall, I find the claim that the paper's argument "explains" the phenomenon to be too strong. ### Summary Overall, this paper presents is an interesting collection of observations, identifying an underlying pattern across various distillation algorithms. Although I have some concerns with (a) the specificity to ImageNet and (b) the paper's claim at explaining these phenomena, I believe future work would definitely find this useful. ### Post-rebuttal update Increasing the score by 1 to acknowledge the authors' new experiments on adversarial vulnerability and domain adaptation that go beyond ImageNet.

Questions

1. Are there any types of knowledge that _do not_ get transferred? The main paper would benefit from a more balanced and scientific discussion of any such types knowledge. 2. One of the implications laid out in the paper, namely that the teacher may transfer biases to the student, may have also been captured in other work, which may have been prior to this paper [1]. [1] _Teacher's pet: understanding and mitigating biases in distillation_, Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, Sanjiv Kumar https://arxiv.org/abs/2106.10494

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

I couldn't find a limitation section in the paper. I'd appreciate explicitly listing out some limitations e.g., there is lack of clarity on 1. how this generalizes to other datasets 2. why these types of knowledge get transferred 3. what types of knowledge do/do not get transferred

Summary

The paper presents an empirical study to analyze inner workings of knowledge distillation (KD) methods. The authors try to understand the nature of the knowledge "distilled" from a teacher to a student by studying how various properties get transferred in this learning. The properties include (a) localization information, (b) adversarial vulnerability, (c) invariance to data transformations and (d) domain adaptation. There is further analysis about some other properties in the supplement. The paper analyzes three broad distillation methods on ImageNet with multiple CNN and transformer architectures. Lastly, the authors present positive/negative applications of this understanding for specific tasks (domain adaptation, bias transfer).

Strengths

Strengths: (1) The paper is nicely organized and very well written. The motivations to study the topic are clear. To my knowledge, this analysis is quite original. (2) The authors investigate some relevant and interesting directions in a modular way to study KD. (3) Multiple different architectures and distillation techniques are carefully analyzed for various properties and generally demonstrate consistent results.

Weaknesses

Weaknesses: The weaknesses I see stem from the purely empirical nature of the study. (1) The generalizability of results to other types of datasets, tasks, domains remains unknown. (Major) (2) The selection of properties studied seems a little arbitrary. (Minor) (3) There is lack of any theoretical analysis to characterize the distilled knowledge. (Minor) Overall, I like the paper but can't favour stronger acceptance yet. I enjoyed reading through it. It is structured well, the experiments are quite thorough (except in one aspect!) and give some interesting insights about KD. Among the weaknesses I highlighted, I won't hold the last two weaknesses strongly against the paper as (3) is a difficult topic in itself and the studied set of properties could still be regarded as reasonably comprehensive in terms of popular ML topics today. However, considering the purely empirical nature, weakness (1) is major for me that should be tackled before I can more clearly favor acceptance.

Questions

Questions: (1) **Other datasets/tasks**: How well do the results generalize over other datasets and different tasks (small/large scale images, fine-grained recognition, scene classification etc.). I understand that the number of possible experiments is too large and the paper can't cover everything. However, the same way the paper deserves credit for coverage in terms of distillation methods and architecture selection of teacher and student, it also needs to confirm/study if these results hold for other settings. At least a couple more settings should be included. This is specially important as there are some non-intuitive results and demand further analysis to be confirmed. Also, it would be great if you could illustrate domain adaptation application for some non-MNIST data too. (2) **Properties selected**: While I am assuming the current set of properties were selected as these are some popular topics of study in ML, I would like to know your rationale behind selecting them. For example did you also consider assessing if the uncertainty of teacher gets transferred to student? (3) **More questions about experiments**: Does a mixture of simultaneous KL + Hint or other mixtures provide stronger distillation of the properties? Is the localization transfer for transformer -> CNN poor? You could use some other saliency map as localization proxy if grad-CAM is not suitable.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

4 excellent

Contribution

3 good

Limitations

The authors mention "We do not have a separate section for limitation, but since ours is an analysis paper, we are discussing the open questions all throughout the paper." While I understand their reasoning and thought process, I also believe they should summarize the limtiations of their experimental setup in the conclusion or a separate section. It is an experimental study and it can't possibly conduct all experiments as part of it. The open questions raised throughout the paper I believe or more related to depth of the analysis. The authors should certainly add the limitations regarding breadth of their analysis.

Summary

The paper explores different ways in which a student network can inherit implicit properties from a teacher network beyond just improving task performance. The authors study various distillation methods, including output-based, feature-based, and contrastive-based techniques, and show that these methods can indirectly distill properties. The paper also investigates why knowledge distillation works and discusses its practical implications. Overall, the paper aims to shed light on the "dark knowledge" of knowledge distillation and motivate further investigation into the distillation process.

Strengths

1. This paper sheds light on the dark knowledge of knowledge distillation (KD) by investigating a series of questions in KD and gives interesting findings. 2. Each sub-section in experiments is well-organized, and the question is answered with clearly designed experiments and supportive results, followed by insightful discussions. 3. The paper is well-written and the presentation is decent.

Weaknesses

1. The major weakness of this paper is the setup appears to be more exploratory in nature. While exploratory experiments can certainly provide valuable insights and help to understand the findings, they do not substitute for a thorough evaluation that generalizes across diverse datasets. In this paper, the use of only a few datasets may not present a comprehensive picture and could yield misleading results. 2. The selection of benchmark methods in this paper, while representative of some standard approaches, is notably restricted. The experimental comparison lacks several recent knowledge distillation (KD) methods, leaving the evaluation somewhat incomplete. The inclusion of these recent advances in KD techniques would not only enhance the comprehensiveness of the evaluation but also deepen our understanding of the KD process. (Ren et al., 2022) Better Supervisory Signals by Observing Learning Paths. (Chen et al., 2021) Distilling Knowledge via Knowledge Review. (Zhao et al., 2022) Decoupled Knowledge Distillation.

Questions

See above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above.

Summary

The paper designed a series of experiments to fundamentally dissect and understand knowledge distillation. The key questions include, Does localization knowledge get distilled? Does adversarial vulnerability get distilled? Does invariance to data transformations get distilled? Does knowledge about unseen domains get distilled? Can students inherit harmful biases from the teacher? and the main content of this paper is to design experiments to answer these questions empirically. At last, the paper provides some intuition about why knowledge distillation works in this way, and concludes with open questions on how to leverage such findings.

Strengths

The paper proposed to comprehensively study the critical, interesting and fundamental problem about deep understanding of knowledge distillation, from an angle of what properties of the teacher are distilled into students, which is a novel angle. The paper extracted the key questions in order to understand knowledge distillation, where those questions are interesting and essential to help the problem better. Also, for each question, the paper designed corresponding experiments on the representative models with standard datasets. The paper is easy to follow.

Weaknesses

Even though the paper proposed to study a critical fundamental problem, the paper focused more on designing the experiments to answer the potential questions that could help understand the questions. However, many answers towards the questions are not that surprising, which falls in the expectation. Purely relying on some empirical results to facilitate the understanding of the questions limits the contribution of this paper. If the paper could have some theoretical results to demonstrate why it works and when it works, it could solidify the contribution of this paper. For each experiment, it’s unclear whether the conclusion could still hold for various architectures or datasets. So it would be great that the paper could include more representative architectures and broader datasets to make results more convincing. The contributions of this paper would be more solid if there are some attempts to answer some of the open questions in the conclusion part, e.g. how to design algorithms to control which properties to distill or not distill.

Questions

The intuition presented in section 6 could be formalized lowering variance as in the following paper? Menon et al, “A Statistical Perspective on Distillation”, ICML 2021.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

yes

Thanks for the response!

Thanks for your rebuttal! I'm raising my score by 1 point to acknowledge your new experiments. Reg. 2: I still believe that this discussion is still circular in the non-linear neural network) regime. In this regime, it is not clear how many points one would need to uniquely define a decision boundary. Your experiments merely show that the decision boundary somehow gets transferred, but it does not shed insight on the "how?". So to re-emphasize, I think this section would be much more interesting and rigorous, if the "explanation" claim is removed, and if an open "how?" question was posed. Reg. 3: I'd highly encourage bringing this section in the main paper as it presents a more balanced view of what distillation does or does not do.

Rebuttal acknowledgement

I want to thank the authors for the rebuttal. The additional experiments strengthen the support for their observations and expand the coverage of analysis, which was much needed. They certainly address my concerns to an extent that I won't consider them as major anymore. Consequently, I am increasing my score from **5 to 6**, assuming incorporation of additional modifications the authors have committed to.