Transformer models can face practical limitations due to their high computational requirements. At the same time, such models exhibit significant activation sparsity, which can be leveraged to reduce the inference cost by converting parts of the network into equivalent Mixture-of-Experts (MoE) layers. Despite the crucial role played by activation sparsity, its impact on this process remains unexplored. We demonstrate that the efficiency of the conversion can be significantly enhanced by a proper regularization of the activation sparsity of the base model. Moreover, motivated by the high variance of the number of activated neurons for different inputs, we introduce a more effective dynamic-$k$ expert selection rule that adjusts the number of executed experts on a per-token basis. To achieve further savings, we extend this approach to multi-head attention projections. Finally, we develop an efficient implementation that translates these computational savings into actual wall-clock speedup. The proposed method, Dense to Dynamic-$k$ Mixture-of-Experts (D2DMoE), outperforms existing approaches on common NLP and vision tasks, reducing inference cost by up to 60% without significantly impacting performance.
Paper
Similar papers
Peer review
Summary
This paper proposes a more effective dynamic-k expert selection rule that adjusts the number of executed experts on a per-token basis to reduce the high computational cost of the transformer models. They claim their D2DMoE model outperforms existing approaches.
Strengths
1. The motivation of this paper is good. They try to save up the inference cost of the transformer models. 2. They perform experiments on several tasks.
Weaknesses
1. The most important contribution is the dynamic-k routing algorithms. However, the main work is a small modification to MoEfication. The difference of the proposed method with MoE-based methods is limited. 2. The experiments are not conducted well. This paper lacks a comprehensive comparison with other methods. There exist many MoE-based VIT methods, such as [1, 2]. Furthermore, why does the authors does not provide any performance number in the table? This paper claims they outperforms existing approaches on common NLP and vision tasks, however, only two methods are compared. 3. The major advantage of D2DMoE compared with MoEfication is saving inference cost. However, the saved cost is quite limited observed from fig. 4. [1] Mod-squad: Designing mixtures of experts as modular multi-task learners [2] M3 vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design
Questions
1. Training Overhead: Can you provide more details on the additional training time required for sparsity enforcement and router training? How does this overhead compare to the overall training time of the original dense models? 2. Real-world Deployment: What steps or considerations are necessary for deploying D2DMoE models in real-world applications, particularly in terms of latency, hardware compatibility, and integration with existing pipelines? 3. Adversarial Robustness: Have you evaluated the robustness of D2DMoE models against adversarial attacks? If not, do you have any plans to explore this aspect in future work? 4. Comparison with Pruning Techniques: How does D2DMoE compare with other model pruning or compression techniques in terms of performance, computational savings, and implementation complexity? 5. Scalability: Can the proposed method be effectively scaled to extremely large models like GPT-3? Are there any specific challenges or considerations when applying D2DMoE to such models?
Rating
3
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
The paper does not sufficiently address potential challenges in real-world deployment, such as latency variations, hardware compatibility, and integration with existing systems.
We thank the reviewer for the response. While the brevity of their comment suggests that they may have already formed a definitive opinion about our work, we believe it is important to clarify our view to other participants. Apart from addressing the raised concerns, we also want to highlight how they have been presented. In particular, we find it challenging to understand why the **reviewer's response took so long, given that it is brief, somewhat vague, and largely reiterates previously stated points**. We point out other issues in our responses below. **We respectfully ask other participants to carefully consider both the reviewer's comments and our discussion to arrive at their own conclusions regarding the validity of the raised concerns.** > I still think the dynamic-k routing is a slight modification of MOE's top-k routering. **The reviewer seems to disregard our discussion, where we clearly state that dynamic-k is not our only contribution.** We also emphasize that reviewer uQnr praised the novelty of our router training scheme. We have provided a clear justification for the introduction of dynamic-k. This modification is evidently not that obvious given that dynamic-k was not adopted in earlier works. We demonstrate that static top-k gating is inappropriate for MoEs converted from dense models (Figure 2b) and that dynamic-k **enables consistent improvements** (Figure 4, Figure 5a). We also highlight that many influential ideas, such as the GELU activation function [1], mixup augmentation [2], dropout [3], or expert choice routing for MoEs [4] could be seen as __slight modifications__. Straightforward, yet well-motivated methods that provide consistent improvements are crucial for advancing the field, and simplicity often facilitates broader adoption. > Furthuremore, from fig. 4, we cannot observe the performance directly. Using loss value to represent the performance is not a good choice. **The reviewer initially described the cost savings shown in Fig. 4 as minimal. However, following our rebuttal, the reviewer apparently no longer seems to contest the improvement shown in the figure.** Instead, they raise a new concern shortly before the conclusion of the discussion period. We emphasize that the **reviewer’s statement is inaccurate, as we do report accuracy for classification tasks**. We use test loss only for LLM evaluation, as done in multiple other works. In particular, a significant portion of the **industry and research community heavily invests in research on scaling laws [5, 6]. These studies focus on improving model loss**, which indicates that it is widely considered as an important indicator of model performance. To further support our claim, **we provide a downstream evaluation of our Gemma models on the BoolQ dataset**. We take the base model, which achieves 68.40% accuracy, and convert it to MoE with D2DMoE and MoEfication and perform zero-shot evaluation. In the table below, we report **relative accuracy** of the models at different compute budgets. Similarly to the loss in Figure 4d, **our method largely retains the performance** across multiple compute budgets, while the **performance of MoEfication decreases significantly**. | Compute budget | 100% | 90% | 80% | 70% | 60% | 50% | 25% | 10% | |----------------|---------|--------|--------|--------|--------|--------|--------|--------| | D2DMoE | 100.00% | 99.68% | 99.37% | 98.69% | 97.60% | 94.34% | 92.75% | 90.89% | | MoEfication | 100.00% | 92.24% | 92.19% | 92.15% | 88.79% | 75.40% | 86.70% | 77.53% | > Although [1,2] are multi-task methods, they all provide pretrained models. The authors should attempt to compare them because the proposed method is quite similar to [1,2]. We still consider the fact that the referenced works are multi-task learning solutions as a fundamental issue that renders any fair comparison impossible. The reviewer's response and his original review **fail to clarify how the referenced works are similar or relevant to our work**, with the only superficial connection being that they also use the MoE architecture. [7] targets Taskonomy and PASCAL-Context datasets, and [8] targets NYUD-v2 and PASCAL-Context datasets (all of them exclusively multi-task datasets). Neither of these works includes evaluations on baselines that are even remotely similar to ours, nor does our method target multi-task learning. Even though [7,8] might provide pre-trained models, comparison of models trained on different datasets **would mostly reflect performance differences stemming from the use of different training data** instead of those inherent to the method. We include the references in a separate response.
#### References [1] Hendrycks, Dan, and Kevin Gimpel. "Gaussian error linear units (gelus)." arXiv preprint arXiv:1606.08415 (2016). [2] Zhang, Hongyi, et al. "mixup: Beyond empirical risk minimization." arXiv preprint arXiv:1710.09412 (2017). [3] Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." The journal of machine learning research 15.1 (2014): 1929-1958. [4] Zhou, Yanqi, et al. "Mixture-of-experts with expert choice routing." Advances in Neural Information Processing Systems 35 (2022): 7103-7114. [5] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020). [6] Clark, Aidan, et al. "Unified scaling laws for routed language models." International conference on machine learning. PMLR, 2022. [7] Chen, Zitian, et al. "Mod-squad: Designing mixtures of experts as modular multi-task learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [8] Fan, Zhiwen, et al. "M^3vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design." Advances in Neural Information Processing Systems 35 (2022): 28441-28457.
Further clarification on differences between D2DMoE and multi-task methods provided by the reviewer
To further emphasize the key differences between multi-task methods referenced in [7,8] and our work, we outline them below: * Our goal is to reduce inference cost by skipping the redundant computation during the model inference through the MoE framework. In contrast, the referenced works employ MoE primarily to solve the multi-task problem, with **efficiency being only a secondary concern**. * Both referenced methods use a classic formulation of MoE adapted for the multi-task setting (e.g. [7] optimizes mutual information between experts and tasks). **None of these works mentions activation sparsity or anything similar to our router training scheme**. * The cited works do always spend the **same amount of compute for each sample** (classic formulation of MoE), with the differences being present only between tasks for [7]. **In contrast, our dynamic-k gating adjusts the computational effort of the model on a per-sample and per-token basis**. * Our method allows for the **adjustment of computational cost after training** and during deployment through the change of the $tau$ hyperparameter. In contrast, [7,8] must retrain the model each time to target a different budget. While this may be not initially obvious from the FLOPs vs accuracy plots in ours and referenced works, we consider this a vital property of dynamic inference methods. * Finally, **we target scenarios with limited computational resources** for the adaptation of pre-trained model, while the references provided by the reviewer require a **significant compute budget to train models from scratch** (e.g. [7] reports training for 80 hours with 240 NVIDIA V100 GPUs).
Summary
The paper presents Dense to Dynamic-k Mixture-of-Experts (D2DMoE), a method to convert a dense model into a MoE model, that exploits the fact that activations in Transformer models are typically very sparse. The sparsity can be further improved by the addition of square Hoyer regularization during a light fine-tuning phase. Since the sparsity pattern changes over examples, the use of a dynamic selection rule is preferred. The strategy used to convert the dense model into a MLP is almost identical to that of Zhang et al. from 2022, but the loss function used to train the router is different. The router in this paper is trained to predict the norm of the output of the corresponding expert (and the experts with the an expected l2 norms over a certain dynamic threshold are selected). Finally, the present work also MoEfies not only the FFN layers as usual, but also all the linear layers in the MHA (for that, each linear layer is converted into a MLP of the same cost, minimizing the l2 error, similar to distillation). The proposed approach is evaluated on different architectures (ViT, BERT, GPT2, Gemma) and benchmarks (including both vision and language). In all cases the proposed approach offeres a better FLOP-quality trade-off than that of Zhang et al. (2022), and achieves a quality similar to the original dense baseline but at significantly fewer FLOPs.
Strengths
- The paper is very well structured and written. All the steps involved in the proposed method are clearly explained in the main paper, or briefly explained first and then thoroughly documented in the appendix. - Figure 5a shows that the step done to improve sparsification is clearly useful. Although this doesn't show how the quality of the resulting model is affected, figure 4 shows that the quality of the original dense model is matched, or very nearly matched, with significantly fewer FLOPs (and better FLOP-quality trade-off than the MoEfication proposed by Zhang et al. in 2022). - Figure 5b shows that more FLOPs (i.e. experts) are allocated to image regions with higher semantical meaning, as one intuitively would expect. This might not be a strength if one only cares about FLOP vs quality, but it certainly makes the approach more interpretable.
Weaknesses
- The use of MoEs from scratch is criticized because the training difficulties that these have. However, the training recipes for these have improved significantly in the past two or three years, and are now a component of many state-of-the-art models (e.g. [Mixtral](https://mistral.ai/news/mixtral-of-experts/), [DeepSeekMoE](https://arxiv.org/abs/2401.06066), [Gemini 1.5](https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf)). Thus, I believe that these should definitely be considered as a baseline, otherwise the potential impact of this work is more limited. - The proposed method involves many steps. They are well documented (see strengths), but makes the whole approach a bit cumbersome, in my opinion. Alternativelly, I wonder how the dynamic routing, the sparsity-indicuing regularization and other of the components used would perform when training a model from scratch. - As usual, one must be carefull when doing only FLOP vs quality comparisons, since the proposed method, especially the dynamic selection of experts, may be hard to efficiently implement on modern hardware (GPUs, TPUs). I couldn't find this discussed anywhere in the paper, so I would appreciate more clarity on this (perhaps even additional plots showing Runtime vs quality, even if it's only on the appendix).
Questions
See the comments mentioned in the weaknesses.
Rating
6
Confidence
5
Soundness
2
Presentation
4
Contribution
3
Limitations
No particular ethical considerations affecting this work, in my opinion.
Summary
This paper proposes a method to convert a dense pre-trained transformer network into its sparse counterpart. The approach begins by fine-tuning a pre-trained network to sparsify its activation values. Subsequently, the method clusters the neurons in the MLP layers to form distinct experts and introduces a router network. Finally, the router network is fine-tuned to predict the norm of the output of each expert. Additionally, this work explores the possibility of modifying attention layers and clustering MLP layers using the GLU activation function. The results demonstrate that the proposed method is robust to high sparsity and outperforms baseline model
Strengths
1. The computational cost of large models is a significant research problem. This work proposes a promising method that can reduce FLOPs by up to 60% without compromising performance. 2. This study conducts extensive experiments across different modalities, models, and datasets, making the results convincing.
Weaknesses
1. This method introduces several hyperparameters during training and inference, which could potentially make it difficult to reproduce and deploy. 2. I find that Figure 2 does not justify dynamic-k very well, as most layers have a similar percentage of non-zero activations. However, this concern is alleviated after reviewing Figures 12 through 16. 3. Although this work sparsifies at a coarse granularity, which is beneficial for reducing latency, reporting wall-time would be more convincing than solely reporting FLOPs.
Questions
After converting the attention weight matrices, is any router network introduced to determine which head is activated? In the main paper, I only read that the attention matrices are replaced with two-layer networks with comparable parameters and cost. I didn't find any rationale provided for this conversion.
Rating
6
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
1. Currently, this method is only evaluated on small networks, so it is unknown whether the proposed method can be effectively applied to larger networks. 2. Although this method leverages sparsity in activations to accelerate inference, it does not reduce the number of parameters, and may even increase them, unlike other methods such as pruning that simultaneously reduce the parameter count.
We thank the reviewer for the feedback and the discussion.
Summary
This paper improves over MoEfication with four innovations: (i) enforcing higher activation sparsity; (ii) directly predict the norm of the output of each expert; (iii) dynamic-k expert selection scheme; and (iv) generalization to any standalone linear layer. The resulting method achieves significant improvements in terms of cost-vs-performance trade-offs in text classification, image classification, and language modeling.
Strengths
- The proposed method is a nice addition of MoEfication, in terms of both performance and generalization (to any standalone linear layer). It makes this kind of method more usable in practice. - I like the idea of directly predicting the norm so as to achieve dynamic-k expert selection. This dynamic scheme is novel and bridges the fields of dynamic inference and MoE. - The experiments illustrate that the proposed method is applicable in various settings, including text classification, image classification, and language modeling.
Weaknesses
- The baseline seems weak. ZTW only receives a few citations and is not published at a top machine learning conference/journal. - Only FLOPs are reported and it is a poor indicator of practical latency/throughput, especially in the LLM era. - It is not clear if the proposed method could be combined with other methods to show its broader applicability. In particular, quantization is a very general method and is almost adopted as a default in many practical applications. - The writing could be improved. For example, more technical details of MoEfication should be included. - In all figures, the lines are continuous. However, these lines should have been a result of interpolation. The original point values are missing.
Questions
NA
Rating
6
Confidence
4
Soundness
4
Presentation
3
Contribution
4
Limitations
Please see weakness.
We appreciate the reviewer’s involvement and are grateful for a very swift response. > Although the baseline is from a top conference, it seems too old. (...) The extension paper from November 2023 demonstrates that ZTW remains a SOTA early exit method and outperforms other baselines such as GPF [1] (ACL2021) and L2W [2] (ECCV2022). In the response to reviewer 85BP we have also added A-ViT [3] (CVPR2022), with the results available in Figure 3 of the rebuttal PDF. Overall, we compare our method against four baselines (ZTW, MoEfication[4], CoFi[5], A-ViT) and conduct experiments on four datasets from different modalities. This surpasses the evaluation scope of other established works on dynamic architectures published at top conferences, which often evaluate on fewer datasets and baselines or compare against static models instead ([3], CVPR2022; [4], ACL 2022; [5], CVPR2023; [6], EMNLP2023; [7], NeurIPS2021; [8], NeurIPS2022; [9] ICLR2023). > (...) it's better to report both accuracy and latency, when combining your method with quantization. To show that quantization reduces the latency of D2DMoE, we modify our kernels to handle `float16` and `int8` data types. We perform a similar experiment to the one from Figure 1 of the rebuttal PDF: 1) we sample gating decisions from the Bernoulli distribution with probability $p$ 2) measure the execution time of our experts for the three data type variants. We present the latency (ms) in the tables below: ### RTX 4090 | p | 0.0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 | |---------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------| | float32 | 0.005 | 0.009 | 0.013 | 0.018 | 0.023 | 0.028 | 0.033 | 0.038 | 0.042 | 0.047 | 0.052 | | float16 | 0.004 | 0.005 | 0.007 | 0.009 | 0.011 | 0.014 | 0.016 | 0.018 | 0.021 | 0.024 | 0.027 | | int8 | 0.004 | 0.004 | 0.005 | 0.007 | 0.008 | 0.009 | 0.010 | 0.011 | 0.012 | 0.013 | 0.014 | ### A100 | p | 0.0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 | |---------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------| | float32 | 0.006 | 0.009 | 0.012 | 0.015 | 0.019 | 0.022 | 0.025 | 0.028 | 0.031 | 0.035 | 0.038 | | float16 | 0.006 | 0.007 | 0.008 | 0.010 | 0.011 | 0.013 | 0.014 | 0.016 | 0.017 | 0.019 | 0.021 | | int8 | 0.007 | 0.008 | 0.009 | 0.010 | 0.011 | 0.012 | 0.014 | 0.015 | 0.016 | 0.017 | 0.019 | The results show that both the higher activation sparsity (lower $p$) of our method and lower-precision data types are complementary in terms of wall-clock time reduction. While we see a smaller improvement from using `int8` over `float16` on A100, we attribute this to differences between GPU architectures and software support for low-precision arithmetic. ##### References [1] Liao, Kaiyuan, et al. "A global past-future early exit method for accelerating inference of pre-trained language models." Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: Human language technologies. 2021. [2] Han, Yizeng, et al. "Learning to weight samples for dynamic early-exiting networks." European conference on computer vision. Cham: Springer Nature Switzerland, 2022. [3] Yin, Hongxu, et al. "A-vit: Adaptive tokens for efficient vision transformer." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. [4] Zhang, Zhengyan, et al. "MoEfication: Transformer Feed-forward Layers are Mixtures of Experts." Findings of the Association for Computational Linguistics: ACL 2022. 2022. [5] Chen, Xuanyao, et al. "Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. [6] Tan, Shawn, et al. "Sparse Universal Transformer." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. [7] Rao, Yongming, et al. "Dynamicvit: Efficient vision transformers with dynamic token sparsification." Advances in neural information processing systems 34 (2021): 13937-13949. [8] Schuster, Tal, et al. "Confident adaptive language modeling." Advances in Neural Information Processing Systems 35 (2022): 17456-17472. [9] Chataoui, Joud, and Mark Coates. "Jointly-Learned Exit and Inference for a Dynamic Neural Network." The Twelfth International Conference on Learning Representations. 2023.
Thank you for conducting the additional experiments. I don’t see any major concerns that would result in a rejection of this paper. However, after reading the comments from the other reviewers, I think there is still room for improvement in experiments. As a result, I decide to maintain my current score.
We appreciate the reviewer's response, active participation in the discussion, and support for our work. Their feedback has been valuable in helping us improve our paper.
Summary
The paper introduces a method called D2DMoE aimed at enhancing the efficiency of transformer models. D2DMoE implements a dynamic-k routing mechanism that allows the model to select a variable number of experts based on the input. The method leverages the inherent activation sparsity in transformer models to reduce the number of active parameters during inference, leading to significant computational savings—up to 60%—without compromising performance. The approach demonstrates that by converting dense layers into Mixture-of-Experts (MoE) layers, transformer models can achieve better accuracy-sparsity trade-offs, making them more efficient for various NLP and vision tasks.
Strengths
The paper presents a thorough empirical evaluation across multiple tasks (image classification, text classification, language modeling) and model architectures (ViT, BERT, GPT-2, Gemma). The experiments compare against relevant baselines and demonstrate consistent improvements.
Weaknesses
In my view, this paper appears to be primarily a repackaged sparse-activated pruning technique stemming from the MoE concept. Several concerns arise: 1. Limited comparison with alternative sparsification methods: The paper predominantly contrasts with MoEfication, neglecting a thorough analysis against other compression or sparsification strategies beyond early-exit techniques. Consequently, the comparative experiments presented lack depth and fail to be fully convincing. 2. Lack of practical acceleration results: The paper only presents theoretical reductions in FLOPs and parameter counts. Without actual inference acceleration results on real hardware (e.g., V100, H100, GTX-4090Ti), it's impossible to assess the practical benefits of the method. The additional overhead from the gating mechanism could potentially negate some of the theoretical gains. 3. Questionable novelty of Dynamic-k gating: The proposed expert selection based on ℓ2-norm of output is indeed more reminiscent of pruning techniques than traditional MoE approaches. This calls into question the novelty of the method when viewed in the context of pruning literature, and makes the comparisons with MoE methods potentially unfair or irrelevant. 4. Limited novelty and inadequate discussion of related work: Many of the proposed operations bear similarities to existing sparse activation pruning methods. The paper fails to adequately discuss these connections, instead focusing on less relevant work. This omission of crucial related work in pruning literature significantly undermines the claimed novelty of the approach. 5. Lack of discussion of some SOTA sparse methods: such as Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models. ICML2024. 6. Theoretical foundation: The paper lacks a strong theoretical justification for why the proposed methods work better, which becomes even more critical given the concerns about its relationship to existing pruning techniques.
Questions
See weaknesses.
Rating
5
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
NA
We thank the reviewer for the comments. We understand we have addressed all the other concerns. >There are also more advanced approaches to dynamic inference, such as dynamic token sparsity, etc. This work seems to have no advantage against these approaches We refer the reviewer to Figure 3 of the rebuttal PDF, where we add A-ViT [1], a dynamic token sparsity method, as a baseline for the response to reviewer 85BP. The results show that we outperform A-ViT. >The authors highlight that their work is a dynamic inference approach, so are there any comparisons with previous works on dynamic pruning? According to a recent survey [2], at least two groups of works use the term “dynamic pruning”. Dynamic pruning approaches such as [3, 4, 5, 6] differ from static pruning because they allow the network structure to change by pruning or reactivating connections during training. The resulting model is static and therefore should complement dynamic inference methods such as ours (as shown in Section 5.6). A small subset of dynamic inference works do use the “dynamic pruning” term for methods that dynamically select channels for execution in convolutional layers [7, 8, 9]. However, they all target exclusively CNNs, while our work focuses on Transformer models. ##### References [1] Yin, Hongxu, et al. "A-vit: Adaptive tokens for efficient vision transformer." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. [2] He, Yang, and Lingao Xiao. "Structured pruning for deep convolutional neural networks: A survey." IEEE transactions on pattern analysis and machine intelligence (2023). [3] Lin, Tao, et al. "Dynamic Model Pruning with Feedback." ICLR-International Conference on Learning Representations. 2020. [4] Ruan, Xiaofeng, et al. "DPFPS: Dynamic and progressive filter pruning for compressing convolutional neural networks from scratch." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 3. 2021. [5] Lin, Shaohui, et al. "Accelerating Convolutional Networks via Global & Dynamic Filter Pruning." IJCAI. Vol. 2. No. 7. 2018. [6] Chen, Zhiqiang, et al. "Dynamical channel pruning by conditional accuracy change for deep neural networks." IEEE transactions on neural networks and learning systems 32.2 (2020): 799-813. [7] Elkerdawy, Sara, et al. "Fire together wire together: A dynamic pruning approach with self-supervised mask prediction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. [8] Gao, Xitong, et al. "Dynamic Channel Pruning: Feature Boosting and Suppression." International Conference on Learning Representations. [9] Lin, Ji, et al. "Runtime neural pruning." Advances in neural information processing systems 30 (2017).
Thanks for the reply and the additional experiments. Most of my concerns are addressed except two: - Although the baseline is from a top conference, it seems too old. Additional experiments with newer baselines may not be necessary during this rebuttal period, but I highly recommend trying out more recent ones. - If your method does not hurt accuracy, and since quantization usually does not hurt accuracy too much, I'm not surprised that when combining your method with quantization, the accuracy will be maintained. However, this may not be true for latency. Therefore, it's better to report both accuracy and latency, when combining your method with quantization.
About dynamic inference method
The authors highlight that their work is a dynamic inference approach, so are there any comparisons with previous works on dynamic pruning? There are also more advanced approaches to dynamic inference, such as dynamic token sparsity, etc. This work seems to have no advantage against these approaches
I thank the reivewer's time and effort to give me a detailed response. However, I still think the dynamic-k routing is a slight modification of MOE's top-k routering. Furthuremore, from fig. 4, we cannot observe the performance directly. Using loss value to represent the performance is not a good choice. Although [1,2] are multi-task methods, they all provide pretrained models. The authors should attempt to compare them because the proposed method is quite similar to [1,2].
Thank you very much for addressing my concern, as well as those from other reviewers. I am still reluctant to increase my score, since I think that the method is quite cumbersome (as I mentioned) given the goal and what it achieves. For instance, the authors have mentioned during the rebuttal that the comparison with MoEs trained from-scratch is not fair because the goal of this work is "to reduce the inference cost of existing pre-trained models". If so, a really imporant and very popular strategy is missing as a baseline: distillation to a smaller model (dense or MoE).
We thank the reviewer for the response. We understand the reviewer's concerns with distillation as a baseline, but - similarly to pruning - we see it more as a complementary model acceleration technique [1]. However, due to time constraints, we cannot provide any experiments that could demonstrate it to the reviewer and will only be able to include them in the camera-ready version. Additionally, we would like to point out that the CoFi method [2] evaluated in combination with our method in Section 5.6 does use distillation as a part of its training scheme. Our method still allows for computational saving when applied on top of CoFi. Input-dependent activation sparsity is an inherent property in almost all models, and therefore our method should be universally applicable. Finally, while we appreciate the thoroughness of the reviewer, to the best of our knowledge so far no papers have examined the compatibility of dynamic inference with all three major model compression methods—pruning, quantization, and distillation. In this context, our comparisons with pruning and quantization are already particularly comprehensive. We thank the reviewer for the discussion and his suggestions that helped us improve the paper. #### References [1] Han, Yizeng, et al. "Dynamic neural networks: A survey." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.11 (2021): 7436-7456. [2] Xia, Mengzhou, Zexuan Zhong, and Danqi Chen. "Structured pruning learns compact and accurate models." arXiv preprint arXiv:2204.00408 (2022).
Response by Reviewer
Thank the authors for providing further clarification. I have read the rebuttal and other reviews. I decide to maintain my original score of 6.
We thank all the reviewers for the discussion. We have addressed each reviewer's concerns in detail, including the additional experiments and clarifications provided in our initial rebuttal regarding FLOPs vs latency, the A-ViT baseline, and quantization. During the discussion period, we also presented new results with quantization for reviewer uQnr. We also added downstream evaluations for reviewer 85BP, despite receiving their feedback just a day before the end of the discussion period. Our general impression was that the reviewers’ main concerns revolved around the latency of our method and its compatibility with other model acceleration techniques. We have provided an efficient GPU kernel for our method, which we consider a significant technical contribution. We demonstrated that our method provides a 63% wall-clock time reduction for FFN modules and around 30% reduction when evaluated end-to-end. We believe that we have adequately addressed the above-mentioned issues through the discussion. We are pleased to note that the three reviewers who recommended acceptance remain positive about our work after the discussion and were satisfied with the responses we provided. We would like to express our gratitude to all the reviewers for their engagement during the discussion period and constructive feedback that helped us improve our paper.
Decision
Accept (poster)