Summary
This paper studies relational knowledge distillation (RKD) on semi-supervised learning setting. This is different from the previous theoretical results on knowledge distillation, which mostly focus on feature matching (not relational). They introduce a new notion called low clustering error, which quantifies the difference betweeen the predicted and ground truth clusterings, and show that RKD can provably produce low clustering error. They also have a few other results, including the label efficiency theorm on a cluster-aware semi-supervised learning framework with low clustering errors, and unifying data augmentation consistency regularization with RKD, showing that consistency reuglarization focuses on the local perspective, while RKD improves over the global perspective.
Strengths
Originality: Since we already have seen a few papers on connecting spectral clustering with contrastive learning or other learning methods, the spectral clustering part of this paper is not particularly surprising. However, I do like the relational part, which focuses on the relationships between the features, rather than the feature matching itself. I think this is a very interesting and important direction to pursue. Moreover, the authors also investigate the benefit of clustering awareness in the semi-supervised learning scenario, which is novel.
Quality: I think this is a well written paper with rigorous definitions, theorems and proofs.
Clarity: Since this is a theory paper, I will not say it is straightforward to read. However, I think the authors did a very good job in the introduction to demonstrate the main contributions of this paper, and present the technical theorems and assumptions in a structured way. Therefore I would say this paper is well written.
Significance: I think this paper is indeed interesting, because it provides an interesting connection between relational knowledge distillation learning and spectral clustering, and also provide theoretical characterization on various aspects.
Weaknesses
It is a bit difficult for me to understand the intuition of Assumption 4.2, and the toy example after assumption 4.2 is too special, which does not explain much about the complicated definitions. It would be nice to further elaborate on the intuitions, and perhaps give a non-trivial example of this assumption, and why this assumption is a mild assumption.
Personally I like Theorem 4.1 better, and I do not feel it is really necessary to consider the limited unlabeled samples setting (because usually we may assume the unlabeled samples are cheap). Therefore, it might improve the presentation of this paper, if the author can add more motivations on Sec 4.2. (if there is no such motivation, it's also fine, I think it is a minor spot.)
The current connections between different results are weak. Specifically, I cannot easily understand how the last DAC result connects with the previous two. I might be wrong, but I initially felt that the third result says the combination of DAC and RKD can improve the performance. However, the statement of Theorem 6.2 seems only contains DAC, not RKD. Therefore, it seems to me that the first and third results are in parallel: both of them are providing upper bounds of clustering error. Is it true?
-- if this is true, I think the presentation of the introduction should be updated, to avoid potential misconceptions.
Questions
I do not have other questions.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Not applicable.