Understanding the Gains from Repeated Self-Distillation

Summary

This paper theoretically investigates the effect of multiple rounds of self-distillation (SD) in linear regression. Under some conditions on the ground truth model $\theta^{\ast}$ and the data matrix $X$, it is shown that multi-step SD can improve the risk bound by a factor of $r = \text{rank}(X)$. Specifically, this happens when the non-zero singular values of $X$ are all distinct and when $\theta^{\ast}$ is *perfectly* parallel to $u_1$, where $u_1$ is the leading singular vector of $X$. The authors show the necessity of such conditions. The improvement yielded by multi-step SD is demonstrated empirically on some simple regression tasks.

Strengths

**1.** This paper seems to be one of the first ones to theoretically characterize *multi-step* SD. **2.** The paper is more or less well-written.

Weaknesses

**1.** The condition of $\theta^{\ast}$ being *perfectly* parallel to $u_1$ is too strong and unrealistic in my opinion. This makes the results of this paper less interesting. A better result would be quantifying the gains of multi-step SD assuming the angle between $\theta^{\ast}$ and $u_1$ is bounded by some quantity, say $\beta$ -- this is a more realistic setting. Then, focus on the special case of $\beta$ being small (and so $\beta = 0$ would be a very special case of these general results). Is it possible to do something like this? **2.** There are no practically relevant insights for multi-step SD in classification problems which are more interesting now. **3.** Overall, I'm not sure about the relevance of the results in this paper due to the above two points.

Questions

Please see Weakness #1.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

1

Limitations

Not discussed in detail.

Summary

The paper analyzes the gains a model can achieve from multi-step self-distillation and presents a solid theory showing that the excess risk does not increase with more self-distillation steps. The synthetic task in the paper effectively proves this analysis.

Strengths

1) The analysis of excess risk for the model trained with multi-step self-distillation is solid. 2) The synthetic task and experiments on the Air Quality, Airfoil, and AEP datasets well support the proposed theory. 3) The paper is well-written and easy to follow.

Weaknesses

1) The study of self-distillation has already been explored in [1,2]. [1] shows that any distillation step can be calculated directly from the initial distillation step. Can the authors compare to [1]? In the experiments of [1], it is shown that accuracy does not always increase with more self-distillation steps. In this paper, the experiments only show results for 2-step self-distillation. How about more steps, like 5 or $\frac{r}{2}$? The datasets used in the experiments do not seem to be widely used. How about results on widely used datasets like CIFAR-10? 2) Assumption 2 is not guaranteed in real tasks. As shown in Figure 2(c), the proposed analysis heavily relies on Assumption 2.2. However, Assumption 2.2 does not seem to be satisfied in real tasks, which weakens the proposed analysis. 3) The experiments are not sufficient to support the proposed theory. [1] Kenneth Borup and Lars N.Andersen. Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation. In Advances in neural information processing systems, 2020 [2] Rudrajit Das, Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In Proceedings of the 40th International Conference on Machine Learning, 2023.

Questions

1) Please address the weaknesses mentioned. 2) What networks are used in the experiments?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, imitations have been discussed.

Summary

The paper tries to provide a theoretical analysis of gains from applying self-distiliation repeatedly, in particular by trying to show that it is important to optimally set the imitation paramter for each step rathen than having a fixed Value. The study is conducted on using ridge estimator for linear regressin. The authors provide a theorm on the excess risk under two assumption when multi-step SD is applied. Experimental results on synthetic data and thre real world datasets. The paper also provides insights on how to set the imitation pramter real datasets.

Strengths

Along with the theoretical analysis, the authors provide experimental results to demonstrate the necessity of the two major assumptions they have for the theoretical gain bounds they show. The analysis on importance of properly choosing imitation factor at each step and how it relates to excess risk is insightful. The problem setting, although simplified, but is straightforward to understand and the idea and analysis is well presented.

Weaknesses

The paper does not provide neither discussion nor experiments on how the proposed analysis can be used in more complex setting (e.g. more expressive models). On the other hand, for the experimental results the author do not provide any insight into why on AEP dataset using multi-step SD degrades the results. Could it be due to violation of the assumptions? In general, the paper provide theoretical analysis for a known technique without providing much insight into how one can use it to improve experimental work, for example it could have been more interesting if the authors could provide more principled techniques for hyperparameter setting for multi-step SD. It would be more helpful if more experimental analysis on either more datasets or in depth were conducted.

Questions

- Can the authors provide any thoughts on how/challenges of expanding the analysis into more complex family of models? - How does the Assumption 2.2 can be interpreted as constraints on more complex models, for example for MLPs? - What could be the reason for drop in performance for AEP?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The theoretical analysis is limited in the scope of models and it is not immediately clear how it can be extended into more expressive models. Furthermore, it is not straightforward to incorporate them on large problems by optimally setting hyperparameters.

Summary

This paper explores self-distillation from a theoretical perspective in the context of linear regression. Distillation is when a model is trained simultaneously to predict training labels and the predictions of another model that has already been trained on the data. Self-distillation is when the trained model has the same architecture as the model being trained. There has been recent work empirically show that self-distillation can result in better models and this paper gives a theoretical understanding of multi-step self-distillation in the context of linear regression. The starting insight of the paper is that repeated self-distillation can be thought of as a pre-conditioner matrix multiplied by the optimal solution to ridge regression. Using this insight, they prove that there is a linear regression problem where self-distillation repeated r times (the rank of the data matrix) can outperform one-step self-distillation and ridge regression by a factor of r. They measure performance in terms of "excess risk" which is the distance to the optimal parameters under a norm weighted by the covariance matrix. Their example with the factor of r gap requires two assumptions: all the singular values of the data matrix are distinct and the optimal parameters are orthogonal to all but one eigenvector. They show theoretically that both assumptions are necessary for such a gap to exist. They also derive an expression for the excess risk and conjecture that using more than r steps doesn't help the self-distillation process (as long as the self-distillation parameters are chosen optimally?). They have some experiments where they show that 2-step self-distillation is robust to the ridge regression hyperparameter lambda. Because finding optimal parameters for self-distillation is difficult on real data, they do not show the performance of 3 or more -step self-distillation.

Strengths

* The paper begins with a nice insight into multi-step self-distillation and leverages it to prove several theoretical results. * The paper shows a very interesting gap in performance between r-step self-distillation and ridge regression and 1-step self-distillation. * The paper shows that two assumptions in the construction are necessary for the gap. * The paper shows in practice that 2-step self-distillation works very well.

Weaknesses

Larger weaknesses: * The paper only shows results in terms of excess risk. I'm not familiar enough with the fixed design setup to determine how useful excess risk is. * The paper shows that r-step self-distillation can be very powerful but they give no way to practically run it i.e., choose the parameters on real data. Smaller points to improve: * I'm confused by the statement of Assumption 2.1. Shouldn't this say that the optimal parameter is orthogonal to every eigenvector *except* the first one? * More generally, I was confused about the way you stated the assumptions. You referred to two necessary assumptions which I assumed would be Assumption 1 and Assumption 2 but they were actually sub-assumptions in Assumption 2. I would ask that you rework what you call these assumptions and how you refer to them for clarity. * In your appendix, you referred back to equations in the main body without reproducing them. This made it annoying to check your work e.g., you compared equation 8 and 26 but I kept having to flip between them (until I eventually copied equation 8). Since there's no page limit in the appendix, I would appreciate if you reproduced all equations when you refer to them. * Your figures are difficult to read when printed out in black and white (and, I assume, for color blind people). Could you please change the marker for each line so they're more distinguishable?

Questions

* You show a gap between 1- and r-step self-distillation. Is there a similar gap between 2-step and r-step self-distillation? Based on your experiments, it looks like 2-step is already very good. Maybe this is a setting where 2-steps basically gives you all the power of r-steps like e.g., load balancing and the "power-of-2 choices". * You justify using excess risk because it "ensures that the signal-to-noise ratio is uniform in all directions." I don't understand this and I don't see a direct relationship between excess risk and MSE. Could you please tell me why I should care about excess risk? And show me where it comes from theoretically?

Rating

7

Confidence

3

Soundness

4

Presentation

4

Contribution

3

Limitations

Yes

Official Comment by Reviewer 2wSn

Thank you for the reply. Most of my concerns have been addressed.