Progressive Ensemble Distillation: Building Ensembles for Efficient Inference

Summary

The paper addresses the problem of obtaining an ensemble of small models suitable for flexible inference requirements and anytime inference, somewhat similar to cascading classifiers. A key contribution is the derivation of a weak learning condition for the distillation of a pre-trained to an ensemble of smaller students as well as an algorithm to obtain such an ensemble. The students are allowed to reuse intermediate activations of other students to efficiently expand the student model hypothesis set. The method is supported by theoretical results on the generalization error, and empirical results on classification tasks for both synthetic, vision, and sensor data.

Strengths

- The paper is well-written, clear, and well-organized. It is mostly easy to follow and understand the argumentation. - The proposed technique is supported both theoretically and empirically, which is a strong feature.

Weaknesses

- The computational requirements during training are unclear. Since Algorithm 2 requires the fitting of potentially multiple models to obtain each student it appears to potentially be quite expensive (especially with large $R$ and/or large `max-search`), but this is not addressed explicitly in the paper - The dimensions of $K_t^-$ and $K_t^+$ are $N \times L$ and especially for large $N$, this could be a bottleneck during training (granted, for a mini-batch this is less severe). Furthermore, while reusing stored activations of previous students in subsequent students might keep the parameter count stable, it also requires additional memory and carefulness in which activations to store. Thus there is some overhead on storing and loading activations and $K$-matrices throughout training. - The empirical results are weak at comparing to other baseline methods, and the method is struggling at the TinyImageNet and ImageNet-1k tasks, where additional inference time is required compared to the teacher. Minor: - L14/15: Claiming distillation is a rigid procedure seems too bold, as a multitude of distillation techniques exists providing options to obtain lots of different students. Granted, most are aimed at obtaining a single student, but distillation in general is very flexible. - L121: "proabability" -> "probability" - Algorithm 1: $R$ is not specified in the algorithm, but needs deduction from Section 3.3 - Inconsistent use of RESCHED and RESHED - Formatting of B-Distill and E-RNN are inconsistent in different places in the paper (e.g. L300-309).

Questions

Generally, my largest concern is the lack of other baseline methods for the empirical results and the unclear computational requirements during training. Additionally: - In Section 5.3 (and Figure 2) it appears that FLOPs are converted to inference time, but it is unclear how, and if this conversion actually holds for an actual implementation. Consider measuring the actual inference time instead. - Clarify the computational requirements during training. What is the overhead on memory and training speed when storing and loading activations and $K$-matrices? - Include common distillation techniques and other methods for early-exit or anytime inference as baselines. E.g. it is unclear how NO-RESHED compares to other distillation schemes with appropriately sized students, or if existing anytime inference techniques surpass the proposed. Minor: - Some dimensions of e.g. $x_i$ and $f(x_i)$ are not clear from the paper (but can be deduced), and e.g. in L119, what indices are $j$ summing over? Consider introducing dimensions more clearly early on. - Figure 2: Since the $x$-axis is the fraction of teacher inference time, the teacher should be marked by a dot and not a line, since the teacher is not able to perform inference at every possible inference time. It should be clear that the teacher is not flexible in inference time.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Sufficiently addressed.

Summary

This paper studies the problem of "progressive distillation": Given a large teacher model, the task is to decompose into smaller student model so that progressively evaluating additional models in this ensemble results into more accurate predictions. Τhe main contributions of this paper are: (i) A principled approach called B-DISTIL for approaching the progressive distillation problem: The authors formulate a two player zero-sum game, from which they derive a weak learning condition. B-DISTIL approximately solves this game. (ii) Theoretical guarantees for the proposed approach under certain assumptions.

Strengths

Principled approach with theoretical guarantees that seems to perform well in real-world settings.

Weaknesses

The proposed approach seems somewhat sophisticated — perhaps not very easy to implement even. By reading the paper, it was not clear to me whether there exists a simpler (but non-idealized) baseline that could be used for comparison — mostly to reassure the reader that the introduced sophistication is actually necessary.

Questions

Is there a simple (maybe standard Knoweledge-Distillation-based) baseline that is meaningful to compare against? For example, in Lines 70-72 the authors mention that "when performing distillation onto a weighted combination of ensembles, it has been observed that adding additional models into the ensemble does not dramatically improve performance over that of a single distilled model". While this could be the case, could such an approach be used as a simple baseline for this setting, so that one could see what are the trade-offs between implementing a simple approach and potentially improving performance by implementing a more sophisticated one like the one proposed by the authors? (I understand if the answer is "there's no simple way to approach this problem", but maybe then perhaps this should be mentioned more explicitly.)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have explained the limitations of their work.

Summary

This paper proposes B-DISTILL, a progressive distillation algorithm that allows for easy trade-off between accuracy and inference-time/latency at runtime. By modeling knowledge distillation as a zero-sum game problem, B-DISTILL utilizes the intermediary connection modules to train and aggregate the sub-student models progressively, resembling the traditional boosting methods. The paper provides mathematical proofs that guarantee the convergence and generalization of B-DISTILL. The experimental results demonstrate the efficiency of B-DISTILL in both anytime inference and early prediction tasks.

Strengths

1. The paper presents a novel perspective by redefining the knowledge distillation problem and effectively applying it to the tasks of anytime inference and early prediction. 2. The paper provides complete mathematical proof and experimental validation to support its claims.

Weaknesses

1. While the method proposed in this paper introduces a novel perspective, its application scope and advantages appear to be quite limited. 2. It seems that some dynamic network structures could potentially be used to address the anytime reference problem. However, it appears that the paper lacks a comparative analysis with relevant methods in terms of results. 3. It might be worth considering modifying the title. B-DISTILL is more like a training method specifically designed for efficient inference rather than a knowledge distillation-related approach.

Questions

1. I would like to know the role of knowledge distillation in this context. Why not directly use ground truth (gt) as the fitting target? 2. What results would be obtained if the teacher model is directly replaced with softened labels?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes.

Summary

The authors describe a new method for knowledge distillation with an ensemble of lower capacity student models, and draw connections between their method and classical boosting approaches. They provide a theoretical analysis of the risk of this method, and demonstrate the benefits of their approach in learning tasks with a limited inference budget, either in terms of computing cost or speed of inference.

Strengths

A compelling idea. Approaching the task of ensemble distillation through a boosting perspective is to my knowledge a novel idea, and the connections that can be made to the boosting literature as a result are quite interesting. At face value, the B-DISTIL algorithm does not seem limited to the distillation setting, and it would be interesting to understand how it compares more generally to other boosting algorithms.

Weaknesses

As an overall weakness, I found the presentation of the paper to be very confusing. In particular: - The relationship between two-player games and boosting needs to be better described in the related work and problem formulation. Schapire and Freund do not simply "show that weak learners can be aggregated to produce strong learners" (lines 90,91) but rather establish key correspondences between the formulation of two player games and boosting algorithms. Language from this correspondence are used throughout section 3.1 ("players,", "minimax value of the game", "ensemble of predictors","weak learners") without a clear description of how two player games and boosting relate to one another, and make the exposition difficult to follow. - Notation needs to be more carefully defined throughout the text. For example, the constants $L,M,N,R$ are all used without explicit reference to what they represent in the main text (see questions below for more notation issues). - It is unclear to me how student models are constructed. Do the configurations in Tables 2-7 provide specifications for the student models used? Or rather for $\mathcal{F}_0$? Are the different rows of the table different base models, or do they somehow relate to the use of intermediate layer connections? How are intermediate connections implemented in each of the specific model architectures described? Overall, I believe that the quality of the paper suffers significantly from issues with presentation, and I am willing to reconsider my score if these issues are addressed. Another weakness of this work is comparison to previous work with respect to experimental findings. It would be useful to know how the results in Figure 4 compare to anytime inference as described in Huang et al. 2018 or Ruiz and Verbeek, 2020 for image classification results.

Questions

- What is an (ordered) set of low inference cost hypothesis classes? (Lines 115-117). Does this correspond to an ensemble with an increasing number of member models? If so it should be clearly stated that this is the case. How does $\mathcal{F}$ relate to $\mathcal{F}_m$? (Line 121). - How does $\mathcal{F}_r$ relate to $\mathcal{F}_0$? How is $\mathcal{F}'_r$ related to $\mathcal{F}_r$? - On first glance, the weak learning condition (Definition 1) appears to me quite different from other weak learning conditions in the traditional boosting literature. Is there an interpretation of this condition that is consistent with other definitions of weak learning? This would be good to know. - The claim "Existing boosting methods for classification treat multi-class settings (L > 1) as L in stances of the binary classification problem (one vs. all)" (Lines 162-166) does not apply to Adaboost.M1. Are the weak learners that you study here unable to meet the weak learning condition for Adaboost.M1? If they meet this condition, it would be a useful baseline against which to compare the performance of this method. - In Figure 3, it would be useful to know the correspondence between the total number of FLOPS required by the ensemble, and the corresponding model accuracy.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

As the authors state, some limitations of their methods include the need to design the class of student models, and the potential additional cost of evaluating models.

Summary

The main focus of this paper is to address a problem in progressive knowledge distillation, which involves approximating a single large teacher model by utilizing an ensemble of multiple smaller student models. The authors propose an algorithm called B-DISTILL to tackle this specific problem. One notable advantage of this methodology is its capability to effectively balance the trade-off between cost and performance by adjusting the ensemble size of the student models.

Strengths

1. The problem formulation of "progressive knowledge distillation" is intriguing and well-motivated. In conventional knowledge distillation approaches for model compression, small student models of fixed sizes are typically employed, resulting in a fixed inference cost. A notable advantage of the proposed methodology is its ability to dynamically adjust the inference costs based on the available resources, which is a clear strength of the approach. 2. While the concept of approximating a function using a combination of multiple functions is not novel (as evident from classical boosting methods mentioned by the authors), this paper provides a distinct contribution by connecting these ideas to the field of knowledge distillation.

Weaknesses

1. The scalability of the proposed methodology appears to be somewhat limited. It was anticipated that B-DISTILL would achieve a similar level of performance as the teacher model while utilizing the same inference cost. However, when applied to TinyImageNet and ImageNet datasets, B-DISTILL falls short of meeting this expectation. 2. One important baseline is missing - deep ensembles using the model structure considered in B-DISTILL. Including this baseline would provide a clear motivation for the progressive formulation adopted in B-DISTILL.

Questions

1. The proposed methodology is in line with the principles of slimmable networks (Yu et al., 2019), as it enables users to control the inference cost. Although there are distinctions in the primary categories of each approach, such as pruning and distillation, they share common properties in terms of model compression. Thus, it would be advantageous to include related works, such as slimmable networks, in the main text to ensure readers have a comprehensive understanding of the topic. 2. Could we adapt the inference cost based on the "difficulty" of the input? Given that there might be instances where accurate predictions can be made without the need for additional student models, the idea of limiting the ensemble size based on the difficulty is highly appealing. __Miscellaneous:__ 1. Typo: "B-DSTILL" in Figure 2. 2. Adjust the legend in Figure 3.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors addressed the limitations.

Thank you.

Thank you for your response. I appreciate the detailed answers to the points mentioned, and I believe my concerns about presentation will be addressed by the revisions suggested by the authors in their rebuttal. I especially support the revisions suggested around the expositions of student models. I have updated my score to reflect these revisions.

Additional questions

Thank you again for taking the time to review our submission. We hope our responses have resolved the reviewer's concerns but we are happy to discuss further if the reviewer has any additional questions.