Once again we thank the Reviewer for their comments and feedback.
> Changing to last-epoch evaluations would be an improvement, but I still believe it should be followed by some comments on the improper evaluation scheme. In particular, since I assume the last-epoch evaluation is still based on particular successful experiments, based on the test performance (which should not really be possible beforehand).
We do understand the Reviewer's concern — after all, this is a well-known issue that has been raised within the ML community throughout the years. (Unfortunately, the fact is that papers that use standard academic datasets like CIFAR and Imagenet — including every baseline we compared against — do use this evaluation scheme and so we had to follow the same approach to guarantee a fair comparison.)
To address the Reviewer's concern:
(i) We will add some comments on the evaluation scheme we used that reflect the Reviewer's concern and our discussion above. (We are happy to adopt any suggestions regarding the phrasing of the latter.)
(ii) We will include the ablation-experiments which show that our approach is robust to parameters like the size of the validation datasets, the choice of the regression method for learning a(x) and k(x), errors in the estimates of a(x) and k(x) (please see the general response for more details). Hopefully this mitigates the danger of overfitting to the nuances of the academic datasets chosen for evaluation.
> The suggested change of phrasing would be adequate for me, however, I would also suggest at least modifying the use of "consistently" as a few experiments are not improved and a few more have negligible improvements (<0.02).
We will modify the use of "consistently" and change our phrasing to:
"Our approach performs on par with previous SOTA approaches and, interestingly, it often outperforms them by a large margin in the important cases where (i) only a few labeled examples are available; (ii) one deals with large-scale problems with many classes like the Imagenet dataset".
We are happy to adopt any further suggestions from the reviewers.
> My main concern is that both in the title and the main paper, a lot of emphases is put on the wording "distillation with unlabeled examples", but since the method does require a labeled validation dataset, I believe the naming is misleading.
The term "distillation with unlabeled examples" is not ours — it was introduced in [CKSNH] to describe the very popular distillation-setting where one has access to only a few labeled examples but a lot of unlabeled examples. Note also that using a few of the labeled examples as a validation set is a very common practice in these type of "learning with few labeled examples"-settings and, additionally, as we mention in Lines 274-278 of the main body of the paper, to be fair to methods not using validation data, we have included the validation data in the training dataset of all methods.
Another term that has been used to describe the setting we consider is "semi-supervised distillation", but we find it a bit confusing since in standard semi-supervised learning one does not have access to a teacher model.
Does the reviewer prefer the latter term/have another suggestion?
[CKSNH]: Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton. Big Self-Supervised Models are Strong Semi-Supervised Learners. NeurIPS 2020.