Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate

Summary

The authors proposed a novel regularization approach called Learning from Teaching (LoT) to enhance generalization. The hypothesis that simple correlations are generalizable is the main question for this work. Through a teacher student approach, the authors are capable of provide to the main model more generalizable and imitable

Strengths

- The LoT computation measurement of ‘imitability’ through the student, later used as the regularizer, is the paper main contribution. - Comprehensive experimental section with a broad set of experiments, such as RL, Fine-tuning and Image classification.

Weaknesses

- I would like to see further evidences of the proposed hypothesis.

Questions

- The hypothesis on Section 2.1 have been proposed before? I better discussion on that will help the overall reading and to better position the paper when compared to other approaches. - For all the experiments N (student steps ratio) was 1? If not the authors compared LoT with other methods for the same number of steps? If not the superior number of steps can partially be the cause of the performance gains. - Regarding the choice of metrics the authors chose KL-divergence, did the authors experimented with other losses? - Not sure if it is feasible but benchmark LoT on ImageNet-R [1] and ImageNet-Sketch [2] will be interesting for a broader set of results. [1] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR pp. 8340–8349, 2021. [2] Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning ro- bust global representations by penalizing local predictive power. In NeurIPS, pp. 10506–10518, 2019.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

- The current set of baselines can be expanded. Methods like [3] and [4] can provide a better understanding on how well LoT is positioned. [3] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021 [4] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang, Decoupled Knowledge Distillation, CVPR, 2022.

Summary

This paper proposes the LOT (Learning from Teaching) regularization technique, which employs auxiliary student learners to help the main model capture these more generalizable correlations. The authors hypothesize that generalizable correlations are expected to be easier to imitate, and LOT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The results suggest the effectiveness and efficiency of LOT in identifying generalizable information.

Strengths

(i) This work proposes a concept that generalizable correlations are expected to be easier to imitate. (ii) This work proposes Learning from Teaching (LOT), a novel regularization technique for deep neural networks to enhance generalization. It is to compute a measure of ‘imitability’ for the main model to learn data correlations at the correct scales. (iii) Multiple experiments demonstrate the effectiveness of LOT

Weaknesses

(i) The core argument of this article is that "generalizable correlations are expected to be easier to imitate", but previous studies have shown that more complex data and correlations contain richer information, where models need to obtain better accuracy to learn, making them more helpful for generalization. These two views seem to be contradictory. At the same time, the author constructed their hypothesis based on human cognition, but the relevant theoretical basis is lacking, which makes its reliability questionable. (ii) This distillation is similar to meta-learning, but its support for generalization is confused. As we all know, meta-learning performs great generalization by obtaining general knowledge by distilling task-specific knowledge and then using it to complete various tasks. But in this paper, as the author said, "both the teacher and student models may lack or possess different task-specific knowledge", it is difficult to understand why it improves generalization. I think it seems to be "joint learning", but this learning mechanism and optimization details are missing. I hope the author can further provide LOT's insight. (iii) The description of the training process in this article is a bit hard to follow. For example, the author mentioned that generalization is improved through "joint learning", but in Section 2.2, the teacher-learner and student-learner are optimized alternately. In the introduction, the author mentioned using the student learner as an auxiliary task to improve generalization, but it is not clear how to capture the generalization connection, what this generalization connection is, and how to guide the model learning. More details may be better. In summary, my concerns focus on the credibility of the author's motivation and the soundness of the method. If these issues can be addressed, I will be happy to improve my score.

Questions

Please see the Weaknesses.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

N/A

Summary

The paper introduces Learning from Teaching (LOT), a regularization technique to enhance the generalization capabilities of deep neural networks. LOT uses separate student models trained by inmate the prediction of the teacher model to provide feedback, promoting the capture of generalizable and imitable correlations. The effectiveness of LOT is demonstrated through significant performance improvements in tasks across Computer Vision, Natural Language Processing, and Reinforcement Learning, showcasing better generalization with fewer training steps compared to traditional methods.

Strengths

Originality: LOT is a novel regularization technique. This approach is unique in its hypothesis that generalizable correlations are easier to imitate, leading to the design of a regularization method that leverages the feedback from student models to improve the main model's generalization. This represents a creative combination of existing ideas from cognitive science and machine learning, applied in an innovative way to enhance neural network performance. Quality: The quality of the research is good. The paper have rigorous experimental validation across multiple domains, including Computer Vision, Natural Language Processing, and Reinforcement Learning. The methodology is well-detailed, with clear formulations and algorithms provided for implementing LOT in various learning contexts. Clarity: The paper is well-written and organized. The introduction and motivation for LOT are clearly articulated, providing a strong rationale for the proposed approach. The methodology section is thorough, with detailed explanations and pseudocode for the LOT algorithm. The experimental results are presented with clear figures and tables, effectively illustrating the benefits of LOT.

Weaknesses

1. The paper primarily compares LOT to a baseline teacher-only model. It will be the best to Include more comparisons with other regularization methods, such as dropout, batch normalization, or other recent advances in knowledge distillation, would provide a stronger validation of LOT's relative effectiveness, since it is possible that LOT could be replaced by combination of other much similar regularization. 2. Although the paper mentions the computational efficiency of LOT, a more thorough analysis of the scalability and computational cost, especially for larger models and datasets, would be beneficial. Providing detailed benchmarks on computational resources, memory usage and time required for training with LOT compared to other methods would help clarify its practical applicability in real-world scenarios. 3. The paper uses a fixed set of student models for feedback. Exploring the impact of different types of student models with varying capacities and architectures could provide deeper insights into the robustness and versatility of LOT. 4. While the empirical results are strong, the theoretical foundations behind why generalizable correlations are easier to imitate could be elaborated further.

Questions

1. I want to ask what is the performance of those NLP model or CV model on the validation set of trained datasets? Since it is also a type of generalization. It could be used to show whether the LOT is helpful for preventing over fitting. 2. For "The total training steps for LOT and the Teacher-only approach is the same for fair comparison", my understanding is that the LOT only train half number of epochs comparing to the teacher-only approach? (If we only train with one student model) 3. The paper only reports the performance the teacher model, what is performance for those students model?

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

2

Limitations

I think the paper adequately addressed the limitations.

Official Comment by Reviewer NPF4

Thanks to the author for the feedback, which solved some of my confusion, but the reliability of the algorithm's insights is still not guaranteed since it basically relies on a strong Hypothesis. It would be better if a theoretical analysis could be provided. Despite this, I am still willing to revise my score from 4->5, and hope that the author can further polish and improve this work.