KernelWarehouse: Rethinking the Design of Dynamic Convolution

Responses to Official Review by Reviewer miVX: Part 1

Thank you so much for the constructive comments, and the recognition of the novelty and the effectiveness of our proposed method. Please see our below responses to your concerns and questions one by one. 1.**To your comment regarding the weakness** “The proposed method claims to achieve the SOTA performance over various vision benchmarks. While it outperforms previous ConvNets, the SOTA models are now mainly transformer-based models. There is no comparison/discussion between the proposed method and transformers.” **Our responses** are: **(1)** We claim that our method achieves SOTA performance over various vision benchmarks **under the condition of** comparing it with existing top-performing dynamic convolution methods such as DY-Conv and ODConv on training the same ConvNet backbones; **(2)** Note dynamic convolution operators are designed to replace normal convolution operators and enhance the capacity of modern ConvNet backbones, and the main motivation of our work is to advance the dynamic convolution research from the perspective of making dynamic convolution operator capable of using a significantly large kernel number ($n>100$ instead of $n<10$) while maintaining parameter efficiency. Therefore, to validate our motivation and method KernelWarehouse, **in experiments we follow popular benchmarking protocols used in dynamic convolution research** and compare our method with existing top-performing dynamic convolution methods on various vision benchmarks by applying them to train different types of ConvNet backbones including lightweight ones and relatively large ones. Under this context, we believe our current experimental comparisons are decent and convincing; **(3)** Yes, the SOTA models are now mainly vision transformer (ViT) models. However, directly comparing ConvNet models trained with our method to ViT models is neither fair nor meaningful due to different architectures and training setups. We think a more convincing way is to validate the generalization ability of our method to ViT backbones. However, because of different feature extraction paradigms (a ConvNet learns local features by applying convolutional layers to image/feature pixels through sliding window strategy and stage-wise down-sampling, while a ViT learns global feature dependencies by adopting a patchify stem where self-attention blocks are applied to non-overlapping same-sized image/feature patches), there is no available research that can apply dynamic convolution to ViTs, to the best of our knowledge. After the paper submission, we further performed pilot experiments on DeiT [1]. Benefiting from kernel partition and warehouse sharing that enable us to apply the attentive mixture learning paradigm to a dense local kernel scale instead of a holistic kernel scale, we find that our method can be well generalized to improve the capacity of seminal ViT architectures by representing each of split cells of weight matrices for “value and MLP” layers as a linear mixture of kernel warehouse shared across multiple multi-head self-attention blocks and MLP blocks, except the “query” and “key” matrix which are already used to compute self-attention. Detailed results are summarized in the below Table. It can be seen that: **(i)** With a small convolutional parameter budget, e.g., $b=3/4\times$, KW can get improved model accuracy while reducing model size of DeiT-Small; **(ii)** With a larger convolutional parameter budget, e.g., $b=4\times$, KW can significantly improve model accuracy, bringing $4.38$%|$2.29$% absolute top-1 accuracy gain to DeiT-Tiny/DeiT-Small; **(iii)** These performance trends are similar to those reported on ConvNets in our original manuscript (see Table 2 and Table 3), demonstrating the appealing generalization ability of our method to different neural network architectures. [1] Hugo Touvron, et al., “Training data-efficient image transformers & distillation through attention”, In ICML,2021. | Models | Parameters | Top-1 Acc (%) | Top-5 Acc (%) | | --- |:---:|:---:|:---:| | DeiT-Tiny | 5.72M | 72.13 | 91.32 | | + KW (1$\times$) | 6.43M | 73.30 | 91.46 | | + KW (4$\times$) | 21.55M | 76.51 | 93.05 | | DeiT-Small | 22.06M | 79.78 | 94.99 | | + KW (3/4$\times$) | 19.23M | 79.94 | 95.05 | | + KW (1$\times$) | 24.36M | 80.63| 95.24 | | + KW (4$\times$) | 78.93M | 82.07 | 95.70 |

Responses to Official Review by Reviewer miVX: Part 2

2.**To your question** “In Table 1, why KW 4x is lower than KW 2x? Any study on this?” **Our responses** are: **(1)** Here, the results are reported for the traditional training strategy which trains each model with 100 epochs and simple data augmentations. Under such a training strategy, it tends to incur overfitting on KW when using a larger convolutional parameter budget $b$ like KW($4\times$) against KW($2\times$). Actually, in Figure 2 of the original manuscript, we provided the training and validation accuracy curves of KW($2\times$), KW($4\times$), DY-Conv($4\times$), ODConv($4\times$) and baseline ResNet18 models to clearly illustrate this overfitting issue. It can be seen that KW($4\times$) brings much higher accuracy (2.79% absolute improvement) on training set yet relatively lower accuracy (0.49% absolute drop) on validation set, compared to KW($2\times$) which already attains better performance than DY-Conv($4\times$) and ODConv($4\times$) in terms of both model size and accuracy. That means ResNet18 with KW($4\times$) likely has better model capacity than ResNet18 with KW($2\times$) but may need strong regularization to get final model accuracy improvement on validation set. Such an overfitting issue is the reason why KW($4\times$) is lower than KW($2\times$) in Table 1, and we actually clarified it in our original manuscript (see the paragraph titled **“Results Comparison on ResNets18 with the Traditional Training Strategy”**); **(2)** This overfitting issue can be well resolved by the advanced training strategy adopted in ConvNeXt [2] which trains each model with a longer training schedule (300 epochs) and more aggressive augmentations, as can be clearly seen from the results shown in Table 2. To better explore the potentials of KW and get improved comparisons of KW with existing top-performing dynamic convolution methods DY-Conv($4\times$) and ODConv($4\times$), we adopt this advanced training strategy for our main experiments on relatively large backbones (including ResNets and ConvNeXt) and **get consistently improved model accuracy to all methods** (Table 2 vs. Table 1). Comparatively, our KW always achieves the best results on all backbones, and increasing the convolutional parameter budget $b$ consistently shows improved model accuracy. We also clarified these points in our original manuscript (see the paragraph titled **“Results Comparison on ResNets and ConvNeXt-Tiny with the Advanced Training Strategy”**). [2] Zhuang Liu, et al., “A convnet for the 2020s”. In CVPR, 2022. **Finally**, regarding more experiments and discussions that we have made during the rebuttal phase, you are referred to our top-level comments titled **“The Summary of Our Responses to All Official Reviews”**, our responses to the other reviewers, and the revised manuscript.

Responses to Official Review by Reviewer AvKF: Part 1

Thank you so much for the constructive comments, and the recognition of the motivation, the novelty and the effectiveness of our proposed method. Please see our below responses to your concerns and questions one by one. 1.**To your comment regarding the first weakness** “The paper writing needs to improve. The introduction only has two major paragraphs and is really hard to parse and digest.” **Thanks for your kind suggestion, and our responses** are: **(1)** Indeed, the Introduction section of our original manuscript has two major paragraphs. In paragraph #1, we first describe the concept difference of dynamic convolution to normal convolution by comparing their definitions with a general formulation, then discuss the major limitation of dynamic convolution on parameter efficiency and existing works to address such a limitation, finally point out the current technical gap and our main motivation (namely rethinking dynamic convolution to explore its performance boundary with a significantly large kernel number (e.g., $n > 100$ instead of $n<10$) while enjoying parameter efficiency). In paragraph #2, we first discuss on two critical cues (parameter dependencies within the same layer and parameter dependencies across successive layers) ignored by existing works in dynamic convolution research, which create two technical pathways that serve as the basis to formulate our KernelWarehouse, then describe the key components of KernelWarehouse, namely kernel partition, assembling kernels at multiple convolutional layers with a shared warehouse and a new attention function, to show how KernelWarehouse redefines three basic concepts of “kernels”, “assembling kernels” and “attention function” in dynamic convolution from the perspective of reducing kernel dimension and increasing kernel number significantly. **Although these points are described in a relatively dense but not very detailed manner, they are presented at least in an understandable way, to the best of our understanding**; **(2)** Following your kind suggestion, in our revised manuscript we restructure the Introduction section into four major paragraphs to improve the presentation; **(3)** Furthermore, besides the Figure 1, we add an Additional figure (Figure 2) to better illustrate the key components of KernelWarehouse mentioned above, add an Algorithm table to show the implementation of KernelWarehouse, and give a better clarification of the proposed attention function, following constructive suggestions by Reviewer okim and Reviewer MCn2; **(4)** We believe that the aforementioned revisions could improve the paper writing to a large extent.

Responses to Official Review by Reviewer AvKF: Part 2

2.**To your comment regarding the second weakness** “Although the proposed method could save parameters, it actually requires more time to run (see Table 10), which makes the resulted model far from real deployment.” **Our responses** are: **(1)** According to Table 10, **(a)** for lightweight MobileNetV2 backbone, the runtime model speed of KernelWarehouse is slower than that of DY-Conv **but is faster than** that of existing best-performing ODConv on a single GPU, and their runtime model speeds are at a similar level on a single CPU; **(b)** for relatively large ResNet50 backbone, the runtime model speed of KernelWarehouse is slower than that of DY-Conv **but is still faster than** that of existing best-performing ODConv on a single GPU, and the runtime model speed of KernelWarehouse is obviously lower than that of ODConv|DY-Conv on a single CPU. **More precisely speaking**, the runtime model speed of KernelWarehouse is lower than the vanilla dynamic convolution DY-Conv on both GPU and CPU, **which is the main limitation of our method, as we faithfully clarified in the original manuscript**; **(2)** The speed bottleneck of KernelWarehouse is primarily due to the dense attentive mixture and assembling operations at the same-stage convolutional layers having a shared warehouse. Fortunately, thanks to the parallel property of these operations, the runtime model speed of KernelWarehouse on both GPU and CPU can be improved by the respective optimizations in implementation. Here, we conduct some experiments to explore two direct optimization strategies: **(a)** Firstly, we remove lots of judgement statements used to guarantee the convergence of the model training process which are not necessary at inference phase (but were included in our original benchmarking), by which the runtime model speed of KernelWarehouse on a single GPU can be improved. Detailed results are summarized in the first Table below, showing obvious speed improvements to KernelWarehouse in all cases. For instance, the runtime model speed of KW($4\times$) on a single GPU improves from 567.2|178.5 (frames per second) to 786.9|191.1 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 862.4|322.7; **(b)** Secondly, to further accelerate the runtime model speed of KernelWarehouse on both GPU and CPU, we propose an alternative implementation of KernelWarehouse in which we select multiple warehouses with large number of kernels from a ConvNet and split each warehouse shared at the same-stage convolutional layers into two (or more) disjoint same-sized warehouses, and use each warehouse to represent a half of kernels at each convolutional layer. In this way, the number of kernel cells in a warehouse is reduced to half, which can significantly improve the runtime model speed of KernelWarehouse on both GPU and CPU while retaining superior model accuracy compared to existing top-performing dynamic convolution methods DY-Conv and ODConv. Detailed results are summarized in the second Table below, showing obvious speed improvements to KernelWarehouse in all cases. For instance, the runtime model speed of KW($4\times$) on a single GPU improves from 786.9|191.1 (frames per second) to 864.4|282.7 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 862.4|322.7; the runtime model speed of KW($4\times$) on a single CPU improves from 8.5|0.6 (frames per second) to 9.7|2.1 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 11.8|4.1; **(3)** By these two direct strategies, the runtime model speed of KernelWarehouse on a single GPU|CPU can now better match to that of DY-Conv to a large degree. We will explore more effective strategies to further improve the runtime model speed of our method.

Responses to Official Review by Reviewer AvKF: Part 3

3.**To your question** “It is counter-intuitive to see KW (4x) performs worse than KW (2x) with additional parameters in Table 1. Could the authors elaborate on this?” **Our responses** are: **(1)** Here, the results are reported for the traditional training strategy which trains each model with 100 epochs and simple data augmentations. Under such a training strategy, it tends to incur overfitting on KW when using a larger convolutional parameter budget $b$ like KW($4\times$) against KW($2\times$). Actually, in Figure 2 of the original manuscript, we provided the training and validation accuracy curves of KW($2\times$), KW($4\times$), DY-Conv($4\times$), ODConv($4\times$) and baseline ResNet18 models to clearly illustrate this overfitting issue. It can be seen that KW($4\times$) brings much higher accuracy (2.79% absolute improvement) on training set yet relatively lower accuracy (0.49% absolute drop) on validation set, compared to KW($2\times$) which already attains better performance than DY-Conv($4\times$) and ODConv($4\times$) in terms of both model size and accuracy. That means ResNet18 with KW($4\times$) likely has better model capacity than ResNet18 with KW($2\times$) but may need strong regularization to get final model accuracy improvement on validation set. Such an overfitting issue is the reason why KW($4\times$) is lower than KW($2\times$) in Table 1, and we actually clarified it in our original manuscript (see the paragraph titled **“Results Comparison on ResNets18 with the Traditional Training Strategy”**); **(2)** This overfitting issue can be well resolved by the advanced training strategy adopted in ConvNeXt [2] which trains each model with a longer training schedule (300 epochs) and more aggressive augmentations, as can be clearly seen from the results shown in Table 2. To better explore the potentials of KW and get improved comparisons of KW with existing top-performing dynamic convolution methods DY-Conv($4\times$) and ODConv($4\times$), we adopt this advanced training strategy for our main experiments on relatively large backbones (including ResNets and ConvNeXt) and **get consistently improved model accuracy to all methods** (Table 2 vs. Table 1). Comparatively, our KW always achieves the best results on all backbones, and increasing the convolutional parameter budget $b$ consistently shows improved model accuracy. We also clarified these points in our original manuscript (see the paragraph titled **“Results Comparison on ResNets and ConvNeXt-Tiny with the Advanced Training Strategy”**). [2] Zhuang Liu, et al., “A convnet for the 2020s”. In CVPR, 2022. **Finally**, regarding more experiments and discussions that we have made during the rebuttal phase, you are referred to our top-level comments titled **“The Summary of Our Responses to All Official Reviews”**, our responses to the other reviewers, and the revised manuscript.

Responses to Official Review by Reviewer NHsB: Part 1

Thank you so much for the constructive comments, and the recognition of the motivation, the novelty and the effectiveness of our proposed method. Please see our below responses to your concerns and questions one by one. 1.**To your comment regarding the first weakness** “The work proposes quite incremental contributions over the dynamic convolution, however, the biggest concern I believe is related to the future use in practice of the proposed version for dynamic convolution. The proposed approach significantly impacts the computational costs in a negative way which can be an important bottleneck for future use in practice. Specifically, as presented in Table 10, the speed on GPU is reduced from 322.7 to 178.5 images/second (if we compare a similar number of parameters of the model between dynamic convolution and the proposed approach). This is a significant limitation of the approach, which adds even more computational costs (on top of already negative impact of the dynamic convolution over the standard convulsion). Also the memory requirements for training and inference can be a potential limitation, I think the work should also report the memory comparison.” **Our responses are in the following three aspects:** **Firstly, to your biggest concern about the negative impact on the runtime model speed**, **(1)** According to Table 10, the runtime model speed of our method KernelWarehouse is obviously lower than DY-Conv on both GPU and CPU (**but is faster than** existing best-performing ODConv on GPU) under the same convolutional parameter budget $b=4\times$, **which is the main limitation of our method, as we faithfully clarified in the original manuscript**; **(2)** The speed bottleneck of KernelWarehouse is primarily due to the dense attentive mixture and assembling operations at the same-stage convolutional layers having a shared warehouse. Fortunately, thanks to the parallel property of these operations, the runtime model speed of KernelWarehouse on both GPU and CPU can be improved by the respective optimizations in implementation. Here, we conduct some experiments to explore two direct optimization strategies: **(a)** Firstly, we remove lots of judgement statements used to guarantee the convergence of the model training process which are not necessary at inference phase (but were included in our original benchmarking), by which the runtime model speed of KernelWarehouse on a single GPU can be improved. Detailed results are summarized in the first Table below, showing obvious speed improvements to KernelWarehouse in all cases. For instance, the runtime model speed of KW($4\times$) on a single GPU improves from 567.2|178.5 (frames per second) to 786.9|191.1 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 862.4|322.7; **(b)** Secondly, to further accelerate the runtime model speed of KernelWarehouse on both GPU and CPU, we propose an alternative implementation of KernelWarehouse in which we select multiple warehouses with large number of kernels from a ConvNet and split each warehouse shared at the same-stage convolutional layers into two (or more) disjoint same-sized warehouses, and use each warehouse to represent a half of kernels at each convolutional layer. In this way, the number of kernel cells in a warehouse is reduced to half, which can significantly improve the runtime model speed of KernelWarehouse on both GPU and CPU while retaining superior model accuracy compared to existing top-performing dynamic convolution methods DY-Conv and ODConv. Detailed results are summarized in the second Table below, showing obvious speed improvements to KernelWarehouse in all cases. For instance, the runtime model speed of KW($4\times$) on a single GPU improves from 786.9|191.1 (frames per second) to 864.4|282.7 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 862.4|322.7; the runtime model speed of KW($4\times$) on a single CPU improves from 8.5|0.6 (frames per second) to 9.7|2.1 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 11.8|4.1; **(3)** By these two direct strategies, the runtime model speed of KernelWarehouse on a single GPU|CPU can now better match to that of DY-Conv to a large degree. We will explore more effective strategies to further improve the runtime model speed of our method.

Responses to Official Review by Reviewer NHsB: Part 2

**Secondly, to your request for the comparison of memory requirements**, we added the results for both training and inference in the below Table. We can observe that, for both training and inference, the memory requirements of our method are very similar to those of DY-Conv, and are much smaller than those for ODConv (that generates attention weights along all four dimensions of the kernel space including the input channel number, the output channel number, the spatial kernel size and the kernel number, rather than one single dimension as DY-Conv and KernelWarehouse), showing that **our method does not have a potential limitation on memory requirements** compared to existing top-performing dynamic convolution methods. The reason is: although KernelWarehouse introduces dense attentive mixturing and assembling operations at the same-stage convolutional layers having a shared warehouse, the memory requirement for these operations is significantly smaller than that for convolutional feature maps and the memory requirement for attention weights are also significantly smaller than that for convolutional weights, under the same convolutional parameter budget $b$. | Models | Params | Training Memory (batch size=128) | Inference Memory (batch size=100) | | --- |:---:|:---:|:---:| | ResNet50 | 25.56M | 11,084MB | 1,249MB | | + DY-Conv ($4\times$) | 100.88M | 24,552MB | 2,062MB | | + ODConv ($4\times$) | 90.67M | 31,892MB | 5,405MB | | + KW ($1/2\times$) | 17.64M | 23,323MB | 2,121MB | | + KW ($1\times$) | 28.05M | 23,231MB | 2,200MB | | + KW ($4\times$) | 102.02M | 24,905MB | 2,762MB | | Models | Params | Training Memory (batch size=32) | Inference Memory (batch size=100) | | --- |:---:|:---:|:---:| | MobileNetV2 ($1.0\times$)| 3.50M | 2,486MB | 1,083MB | | + DY-Conv ($4\times$) | 12.40M | 2,924MB | 1,151MB | | + ODConv ($4\times$) | 11.52M | 4,212MB | 1,323MB | | + KW ($1/2\times$) | 2.65M | 3,002MB | 1,076MB | | + KW ($1\times$) | 5.17M | 2,823MB | 1,096MB | | + KW ($4\times$) | 11.38M | 2,916MB | 1,144MB | **Thirdly, the contributions of our work are three-fold**: **(1)** Our main motivation, rethinking dynamic convolution to explore its performance boundary with a significantly large kernel number (e.g., $n > 100$ instead of $n<10$) while enjoying parameter efficiency, is not explored by existing works; **(2)** two critical insights, namely parameter dependencies within the same layer and parameter dependencies across successive layers, which create two technical pathways that serve as the basis to formulate our method KernelWarehouse, are also not explored by existing works; **(3)** with three key components, namely kernel partition, warehouse sharing to multiple convolutional layers and the proposed attention function, our KernelWarehouse redefines three basic concepts of “kernels”, “assembling kernels” and “attention function” in dynamic convolution from the perspective of reducing kernel dimension and increasing kernel number significantly, and sets performance records for the dynamic convolution on different benchmarks. For the first time, KernelWarehouse can even reduce the model size of a ConvNet while improving the accuracy, as illustrated with MobileNetV2, ResNets and ConvNeXt backbones in Table 2 and Table 3 of the original manuscript; **(4)** Considering the aforementioned points that have been well recognized by other reviewers, we argue that **our contributions are decent rather than quite incremental** over existing dynamic convolution works.

Responses to Official Review by Reviewer NHsB: Part 3

2.**To your comment regarding the second weakness** “It looks that a significant gain comes from the “new attention function” (without this, the work is underperforming the dynamic convolution), the results are not reported also when using the “new attention function” in the dynamic convolution framework.” **Our responses** are: **(1)** Yes, the proposed attention function is essential to our KernelWarehouse, which tightly couples with the other two core components (kernel partition and warehouse sharing) and makes our method can attain the best performance; **(2)** In the below table, we add experimental results for using the proposed attention function to existing top-performing dynamic convolution methods DY-Conv and ODConv, showing slight drop in model accuracy compared to the original Softmax function. This is because that the proposed method is specialized to fit three unique attentive mixture learning properties of KernelWarehouse: **(a)** the attentive mixture learning is applied to a dense local scale (kernel cells) instead of a holistic kernel scale via kernel partition and warehouse sharing; **(b)** the number of kernel cells in a warehouse is significantly large (e.g., $n>100$ instead of $n<10$); **(c)** a warehouse is shared to represent every kernel cell in multiple convolutional layers of a ConvNet. | Models | Parameters | Attention Function | Top-1 Acc (%) | Top-5 Acc (%)| | --- |:---:|:---:|:---:|:---:| | ResNet18 | 11.69M | - | 70.44 | 89.72 | | + DY-Conv ($4\times$) | 45.47M | Softmax | 73.82 | 91.48 | | + DY-Conv ($4\times$) | 45.47M | Ours| 73.74 | 91.45 | | + ODConv ($4\times$) | 44.90M | Softmax | 74.45 | 91.67 | | + ODConv ($4\times$) | 44.90M | Ours| 74.27 | 91.62 | | + KW ($1\times$) | 11.93M | Softmax | 72.67 | 90.82 | | + KW ($1\times$) | 11.93M | Ours| 74.77 | 92.13 | | + KW ($4\times$) | 45.86M | Softmax | 74.31 | 91.75 | | + KW ($4\times$) | 45.86M | Ours| 76.05 | 92.68 | 3.**To your comment regarding the third weakness** “Not clear why the parameters are decreasing when reducing the sharing levels (Table 6 and 7), I was expecting the opposite.” **Our responses** are: **(1)** In Table 6 and Table 7, **it is correct** that the model size is decreasing when reducing the sharing levels of warehouse; **(2)** The reason is: As shown in Table 6 and Table 7, all testing cases use the same convolutional parameter budget, $b=1\times$. When reducing the sharing level, the corresponding warehouse will have a reducing number of kernel cells while the size of a kernel cell does not change. As a result, the total attention parameters for linear mixtures are decreasing and the total convolutional parameters ($b$) keep the same, leading to a smaller model size when reducing the sharing level of warehouse.

Responses to Official Review by Reviewer NHsB: Part 4

4.**To your comment regarding the last weakness** “The improvements for ConvNeXt look quite marginal, it looks that the approach can have a scalability issue (basically not that useful for larger models)”. **Our responses** are: **(1)** Note that ConvNeXt [1] adopts an advanced training strategy (with the number of training epochs extended to from the original 100 to 300, lots of aggressive data augmentations such as randaugment, mixup, cutmix and lable smoothing, AdamW optimizer instead of SGD, etc., as we clarified in the original manuscript, see Section A.1.1 of the Appendix for details) that is close to the training strategy used for modern vision transformers. Under such a strong training setup, 0.48%|0.16% top-1 improvement by our KW($1\times$)|KW($3/4\times$) (with 15.7% parameter reduction to the backbone for KW($3/4\times$)) for ConvNeXt pre-trained on ImageNet-1K (Table 2) are obviously smaller compared to the improvements for other backbones, but are acceptable. Furthermore, when transferring the pre-trained ConvNeXt to more challenging downstream tasks (Table 4), the improvements by our KW($1\times$)|KW($3/4\times$) are significantly pronounced, e.g., attaining 1.4%|0.7% mAP gain on object detection; **(2)** Considering that ConvNeXt is a powerful convolutional network architecture designed to match the performance of modern vision transformers, another way to better explore the potentials of our method is to use a larger dataset for model pre-training. Following the paper of ConvNeXt, we conduct a set of new experiments in which ConvNeXt backbone is pre-trained on ImageNet-22K first, then fine-tuned on ImageNet-1K. Detailed results are summarized in below table. We can see that the top-1 improvement of KW($1\times$) is improved from 0.48% to 1.02%, demonstrating the good generalization ability of our method. | Models | Parameters | Pre-training | Top-1 Acc | Top-5 Acc | | --- |:---:|:---:|:---:|:---:| | ConvNeXt-Tiny | 28.59M | | 82.07 | 95.86 | | ConvNeXt-Tiny | 28.59M | $\checkmark$ | 82.90 | 96.62 | | + KW ($1\times$) | 32.99M | | 82.55 | 96.08 | | + KW ($1\times$) | 32.99M | $\checkmark$ | 83.92 | 97.03 | [1] Zhuang Liu, et al., “A convnet for the 2020s”. In CVPR, 2022. **Finally**, regarding more experiments and discussions that we have made during the rebuttal phase, you are referred to our top-level comments titled **“The Summary of Our Responses to All Official Reviews”**, our responses to the other reviewers, and the revised manuscript.

Responses to Official Review by Reviewer okim: Part 1

Thank you so much for the constructive comments and the recognition of our work. Following your constructive suggestions, we carefully reconstruct the Method section to improve the writing such that the proposed method KernelWarehouse could be presented in an easier understandable way. Specifically, in the updated manuscript, we have made the following improvements: **Improvement 1 to the Method section**: Besides Figure 1 showing the overall framework of KernelWarehouse, we add an Additional Figure 2 to better illustrate the different components of KernelWarehouse, showing at the same-stage convolutional layers of a ConvNet backbone how the size of the kernel cell is determined, how kernel partition is performed, how a shared warehouse is constructed, and how kernel assembling is done with the shared warehouse. **Improvement 2 to the Method section**: We also add an Algorithm table to show how KernelWarehouse is implemented, given a ConvNet backbone and the convolutional parameter budget $b$. **Improvement 3 to the Method section**: We also change the name of the proposed attention function from “New Attention Function (NAF)” to “Contrasting-driven Attention Function (CAF)”, better describing what our proposed function is doing. Our proposed attention function is specialized to fit three unique attentive mixture learning properties of KernelWarehouse conditioned on kernel partition and warehouse sharing: **(a)** the attentive mixture learning is applied to a dense local scale (kernel cells) instead of a holistic scale (whole kernels); **(b)** the number of kernel cells in a warehouse is significantly large (e.g., $n>100$ instead of $n<10$); **(c)** a warehouse is shared to represent every kernel cell in multiple successive convolutional layers of a ConvNet. Under these design properties, popular attention functions used in existing dynamic convolutional methods such as Sigmoid and Softmax work poorly for KernelWarehouse, but our proposed attention function works well, as tested by experimental results shown in Table 2 and Table 8. The success of the proposed attention function is due to its appealing property that can learn diverse attentions for all linear mixtures simultaneously and make the mixed kernel cells at multiple successive convolutional layers can learn informative and discriminative features hieratically. According to the definition of Equation 3, such a property comes from the binary attention initialization strategy (the second term) and the linear attention normalization function (the first term). Under different convolutional parameter budget $b$, the second term of the proposed attention function ensures that the initial valid kernel cells ($\beta_{ij}=1$) in a shared warehouse are uniformly allocated to represent different linear mixtures at multiple successive convolutional layers at the beginning of the model training, and the first term adopts a linear attention normalization function that **enables the existence of both negative and positive attention weights (popular attention functions do not generate negative attention weights) and encourages the training process to learn contrasting attention relationships** among all linear mixtures sharing the same warehouse. Considering the above motivation and the functioning mechanism, we rename the proposed attention function to “Contrasting-driven Attention Function (CAF)”.

Responses to Official Review by Reviewer okim: Part 2

**An important improvement to the experiments that you may be interested in**: After the paper submission, we further performed pilot experiments on DeiT [1] to validate the generalization ability of our method to vision transformer (ViT) backbones. Because of different feature extraction paradigms (a ConvNet learns local features by applying convolutional layers to image/feature pixels through sliding window strategy and stage-wise down-sampling, while a ViT learns global feature dependencies by adopting a patchify stem where self-attention blocks are applied to non-overlapping same-sized image/feature patches), there is no available research that can apply dynamic convolution to ViTs, to the best of our knowledge. Benefiting from kernel partition and warehouse sharing that enable us to apply the attentive mixture learning paradigm to a dense local kernel scale instead of a holistic kernel scale, we find that our method can be well generalized to improve the capacity of seminal ViT architectures by representing each of split cells of weight matrices for “value and MLP” layers as a linear mixture of kernel warehouse shared across multiple multi-head self-attention blocks and MLP blocks, except the “query” and “key” matrix which are already used to compute self-attention. Detailed results are summarized in the below Table. It can be seen that: **(i)** With a small convolutional parameter budget, e.g., $b=3/4\times$, KW can get improved model accuracy while reducing model size of DeiT-Small; **(ii)** With a larger convolutional parameter budget, e.g., $b=4\times$, KW can significantly improve model accuracy, bringing $4.38$%|$2.29$% absolute top-1 accuracy gain to DeiT-Tiny/DeiT-Small; **(iii)** These performance trends are similar to those reported on ConvNets in our original manuscript (see Table 2 and Table 3), demonstrating the appealing generalization ability of our method to different neural network architectures. [1] Hugo Touvron, et al., “Training data-efficient image transformers & distillation through attention”, In ICML,2021. | Models | Parameters | Top-1 Acc (%) | Top-5 Acc (%) | | --- |:---:|:---:|:---:| | DeiT-Tiny | 5.72M | 72.13 | 91.32 | | + KW (1$\times$) | 6.43M | 73.30 | 91.46 | | + KW (4$\times$) | 21.55M | 76.51 | 93.05 | | DeiT-Small | 22.06M | 79.78 | 94.99 | | + KW (3/4$\times$) | 19.23M | 79.94 | 95.05 | | + KW (1$\times$) | 24.36M | 80.63| 95.24 | | + KW (4$\times$) | 78.93M | 82.07 | 95.70 | **Finally**, regarding more experiments and discussions that we have made during the rebuttal phase, you are referred to our top-level comments titled **“The Summary of Our Responses to All Official Reviews”**, our responses to the other reviewers, and the revised manuscript.

Responses to Official Review by Reviewer T4nD: Part 1

Thank you so much for the constructive comments, and the recognition of the motivation, the design and the experimental comparison and the ablation of our work. Please see our below responses to your concerns and questions one by one. 1.**To your comment regarding the first weakness** “The improvement of KernelWarehouse is limited as shown in Table1 and Table4 considering two models with the same parameter (+ODConv (4×) vs +KW (4×)), which can not show the advantage of KernelWarehouse in terms of efficiency and performance.” **Our responses** are: **(1)** According to Table 1~Table 4, KW($4\times$) brings an absolute top-1 improvement of 1.60%|0.43%|0.50% to ODConv($4\times$) when pre-training ResNet18|ResNet50|MobileNetV2 backbone on ImageNet-1K dataset, and brings 0.30%-0.80% absolute mAP improvements when transferring the pre-trained ResNet50|MobileNetV2 backbone to downstream object detection and instance segmentation tasks with MS-COCO dataset, while maintaining the similar model size. Although the improvements of KernelWarehouse($4\times$) to ODConv($4\times$) are not as large as the improvements of KernelWarehouse($4\times$) to DY-Conv($4\times$), **these improvements are decent but not limited considering that** ODConv is the existing best-performing dynamic convolution method that generates attention weights along all four dimensions of the kernel space (including the input channel number, the output channel number, the spatial kernel size and the kernel number) rather than one single dimension as DY-Conv and KernelWarehouse; **(2)** **The improvement of KernelWarehouse to ODConv could be further boosted** by a simple combination of KernelWarehouse and ODConv to compute attention weights for KernelWarehouse along the aforementioned four dimensions instead of one single dimension. We add experiments to explore this potential, and the results are summarized in the below Table. We can see that, on ImageNet-1K dataset with MobileNetV2 backbone, KernelWarehouse+ODConv($4\times$) **brings 1.12% absolute top-1 improvement** to ODConv($4\times$) while retaining the similar model size. | Models | Params | Top-1 Acc (%) | Top-5 Acc (%) | | --- |:---:|:---:|:---:| | MobileNetV2 ($1.0\times$) | 3.50M | 72.02 | 90.43 | | + ODConv ($4\times$) | 11.52M | 75.42 | 92.18 | | + KW ($4\times$) | 11.38M | 75.92 | 92.22 | | + KW & ODConv ($4\times$) | 12.51M | 76.54 | 92.35 |

Responses to Official Review by Reviewer T4nD: Part 2

2.**To your comment regarding the second weakness** “The convolutional parameter budget $b$ is an important parameter, how to choose an appropriate parameter of the downstream task. The effect of $b$ on image classification, object detection and instance segmentation is different from the experiment.” **Our responses** are: **(1)** Yes, the convolutional parameter budget $b$ is an important hyper-parameter for our method (and also for existing dynamic convolution methods). When evaluating and comparing the generalization ability of our method and other top-performing dynamic convolution methods on downstream object detection and instance segmentation tasks, we follow the common strategy in the deep learning community to conduct experiments on MS-COCO dataset. Specifically, all dynamic convolution methods are merely used to each ConvNet backbone pre-trained on ImageNet-1K, while the neck and the head of Mask R-CNN still use normal convolution (ensuring that the neck and the head have the same parameters to all dynamic convolution methods for fair comparison). That is, for our method, **the convolutional parameter budget $b$ is defined in terms of a ConvNet backbone but not in terms of the whole Mask R-CNN framework** (consisting of the backbone, the neck and the head); **(2)** In the below Table, we add the results to compare the parameters of the pre-trained ConvNet backbones and the corresponding whole Mask R-CNN models; **(3)** In general, from Table 2, Table 3 and Table 4 in our original manuscript, we can observe that a ConvNet backbone trained by our KernelWarehouse with an increasing convolutional parameter budget $b$ gets consistently better performance on different tasks including image classification, object detection and instance segmentation. Therefore, for downstream tasks, a larger $b$ would be a better choice if there is no constraint on the model size, and otherwise it needs to choose a proper $b$ that can balance the desired model size and accuracy. | Backbone Models | Total Parameters for Mask R-CNN | Backbone Parameters | | --- |:---:|:---:| | ResNet50 | 46.45M | 23.51M | | + DY-Conv ($4\times$) | 121.77M | 98.83M | | + ODConv ($4\times$) | 111.56M | 88.62M | | + KW ($1\times$) | 48.94M | 26.00M | | + KW ($4\times$) | 122.91M | 99.97M | | MobileNetV2 ($1.0\times$)| 23.78M | 2.22M | | + DY-Conv ($4\times$) | 32.68M | 11.12M | | + ODConv ($4\times$) | 31.80M | 10.24M | | + KW ($1\times$) | 25.45M | 3.89M | | + KW ($4\times$) | 31.66M | 10.10M | | ConvNeXt-Tiny | 48.10M | 27.82M | | + KW ($1\times$) | 52.50M | 32.22M | | + KW ($3/4\times$) | 44.04M | 23.76M | **Finally**, regarding more experiments and discussions that we have made during the rebuttal phase, you are referred to our top-level comments titled **“The Summary of Our Responses to All Official Reviews”**, our responses to the other reviewers, and the revised manuscript.

Responses to Official Review by Reviewer MCn2: Part 1

Thank you so much for the constructive comments, and the recognition of the motivation, the novelty, the experiments and the ablation of our work. Please see our below responses to your concerns and questions one by one. 1.**To your comment regarding the first main concern on neural network architectures used for performance benchmarking** “1) Architectures have evolved a lot from ResNet18/ResNet50, both in the research as well as the industry areas. Specifically: Experiments with ResNet or even MobileNet feel a bit out of sync with the current literature in terms of architecture design. I appreciate the inclusion of convnext-tiny and, while for imagenet results show only moderate gains, there are some clear gains for object detection and segmentation. Besides convolutional architectures, does this kind of technique work for transformers at all?” **Our responses** are: **(1)** Dynamic convolution operators are designed to replace normal convolution operators and enhance the capacity of modern ConvNet backbones, and the main motivation of our work is to advance the dynamic convolution research from the perspective of making dynamic convolution operator capable of using a significantly large kernel number ($n>100$ instead of $n<10$) while maintaining parameter efficiency. Therefore, to validate our motivation and proposed method, **in experiments we follow popular benchmarking protocols used in dynamic convolution research** and compare our method with existing top-performing dynamic convolution methods on various vision benchmarks by applying them to train different types of ConvNet backbones including popular lightweight MobileNetV2 and relatively large ResNets, and the recently proposed ConvNeXt (a more powerful ConvNet designed to match the performance of modern vision transformers). Under this context, we believe our current experimental comparisons are decent and convincing; **(2)** Yes, neural network architectures have evolved a lot, and the SOTA models are now mainly vision transformer (ViT) models. However, because of different feature extraction paradigms (a ConvNet learns local features by applying convolutional layers to image/feature pixels through sliding window strategy and stage-wise down-sampling, while a ViT learns global feature dependencies by adopting a patchify stem where self-attention blocks are applied to non-overlapping same-sized image/feature patches), there is no available research that can apply dynamic convolution to ViTs, to the best of our knowledge. After the paper submission, we further performed pilot experiments on DeiT [1]. Benefiting from kernel partition and warehouse sharing that enable us to apply the attentive mixture learning paradigm to a dense local kernel scale instead of a holistic kernel scale, we find that our method can be well generalized to improve the capacity of seminal ViT architectures by representing each of split cells of weight matrices for “value and MLP” layers as a linear mixture of kernel warehouse shared across multiple multi-head self-attention blocks and MLP blocks, except the “query” and “key” matrix which are already used to compute self-attention. Detailed results are summarized in the below Table. It can be seen that: **(i)** With a small convolutional parameter budget, e.g., $b=3/4\times$, KW can get improved model accuracy while reducing model size of DeiT-Small; **(ii)** With a larger convolutional parameter budget, e.g., $b=4\times$, KW can significantly improve model accuracy, bringing $4.38$%|$2.29$% absolute top-1 accuracy gain to DeiT-Tiny/DeiT-Small; **(iii)** These performance trends are similar to those reported on ConvNets in our original manuscript (see Table 2 and Table 3), demonstrating the appealing generalization ability of our method to different neural network architectures. [1] Hugo Touvron, et al., “Training data-efficient image transformers & distillation through attention”, In ICML,2021. | Models | Parameters | Top-1 Acc (%) | Top-5 Acc (%) | | --- |:---:|:---:|:---:| | DeiT-Tiny | 5.72M | 72.13 | 91.32 | | + KW (1$\times$) | 6.43M | 73.30 | 91.46 | | + KW (4$\times$) | 21.55M | 76.51 | 93.05 | | DeiT-Small | 22.06M | 79.78 | 94.99 | | + KW (3/4$\times$) | 19.23M | 79.94 | 95.05 | | + KW (1$\times$) | 24.36M | 80.63| 95.24 | | + KW (4$\times$) | 78.93M | 82.07 | 95.70 |

Responses to Official Review by Reviewer MCn2: Part 2

2.**To your comment regarding the second main concern on the runtime model speed** “2) Latency is heavily affected. Specifically: Latency might be partially due to the lack of optimized CUDA kernels for certain operations, but not completely, as CPU also shows similar issues. I believe the sequential nature of the attention mechanism and the need to put together the kernel are important factors contributing to the latency degradation. These issues are very hard to solve for dynamic convolutions, definitely not just restricted to the current work.” **Our responses** are: **(1)** Existing dynamic convolution methods generally suffer from lower runtime model speed compared to normal convolution, just as you said. According to Table 10, the runtime model speed of our method KernelWarehouse is obviously lower than DY-Conv on both GPU and CPU (**but is faster than** existing best-performing ODConv on GPU) under the same convolutional parameter budget $b=4\times$, **which is the main limitation of our method, as we faithfully clarified in the original manuscript**; **(2)** The speed bottleneck of KernelWarehouse is primarily due to the dense attentive mixture and assembling operations at the same-stage convolutional layers having a shared warehouse. Fortunately, thanks to the parallel property of these operations, the runtime model speed of KernelWarehouse on both GPU and CPU can be improved by the respective optimizations in implementation. Here, we conduct some experiments to explore two direct optimization strategies: **(a)** Firstly, we remove lots of judgement statements used to guarantee the convergence of the model training process which are not necessary at inference phase (but were included in our original benchmarking), by which the runtime model speed of KernelWarehouse on a single GPU can be improved. Detailed results are summarized in the first Table below, showing obvious speed improvements to KernelWarehouse in all cases. For instance, the runtime model speed of KW($4\times$) on a single GPU improves from 567.2|178.5 (frames per second) to 786.9|191.1 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 862.4|322.7; **(b)** Secondly, to further accelerate the runtime model speed of KernelWarehouse on both GPU and CPU, we propose an alternative implementation of KernelWarehouse in which we select multiple warehouses with large number of kernels from a ConvNet and split each warehouse shared at the same-stage convolutional layers into two (or more) disjoint same-sized warehouses, and use each warehouse to represent a half of kernels at each convolutional layer. In this way, the number of kernel cells in a warehouse is reduced to half, which can significantly improve the runtime model speed of KernelWarehouse on both GPU and CPU while retaining superior model accuracy compared to existing top-performing dynamic convolution methods DY-Conv and ODConv. Detailed results are summarized in the second Table below, showing obvious speed improvements to KernelWarehouse in all cases. For instance, the runtime model speed of KW($4\times$) on a single GPU improves from 786.9|191.1 (frames per second) to 864.4|282.7 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 862.4|322.7; the runtime model speed of KW($4\times$) on a single CPU improves from 8.5|0.6 (frames per second) to 9.7|2.1 for MobileNetV2|ResNet50 backbone, while DY-Conv($4\times$) attains a runtime model speed of 11.8|4.1; **(3)** By these two direct strategies, the runtime model speed of KernelWarehouse on a single GPU|CPU can now better match to that of DY-Conv to a large degree. We will explore more effective strategies to further improve the runtime model speed of our method.

Responses to Official Review by Reviewer MCn2: Part 3

3.**To your comments** “Some of the design choices are key to making the proposed warehouse idea work, e.g. the initialization, temperature and attention design. Do these components work when are combined with other dynamic convolution approaches or is there something in their design that makes them specific to the kernel warehouse idea? Overall, it feels like the proposed method improves upon prior work on dynamic convolution, but still is not clear in which sense it offers some optimal trade-off.” **Our responses** are: **(1)** The proposed attention function, which is defined as Equation 3 consisting of the initialization, the temperature and the generation of the attention, makes our method KernelWarehouse work. Note existing dynamic convolution methods such as DY-Conv and ODConv already use the temperature in their attention functions; **(2)** Combining the proposed attention function with existing dynamic convolution methods leads to slight drops in model accuracy compared to their original attention functions, as can be seen from the experimental results summarized in the first Table below. This is because that the proposed attention function tightly couples with the other two core components (kernel partition and warehouse sharing) and makes our method can attain the best performance by fitting three unique attentive mixture learning properties of KernelWarehouse conditioned on kernel partition and warehouse sharing: **(a)** the attentive mixture learning is applied to a dense local scale (kernel cells) instead of a holistic kernel scale; **(b)** the number of kernel cells in a warehouse is significantly large (e.g., $n\gt 100$ instead of $n\lt 10$); **(c)** a warehouse is shared to represent every kernel cell in multiple convolutional layers of a ConvNet; **(3)** **The performance of KernelWarehouse can be further boosted** by a simple combination of KernelWarehouse and the existing best-performing dynamic convolution method ODConv (that generates attention weights along all four dimensions of the kernel space (including the input channel number, the output channel number, the spatial kernel size and the kernel number) rather than one single dimension as DY-Conv and KernelWarehouse) to compute attention weights for KernelWarehouse along the aforementioned four dimensions instead of one single dimension. We add experiments to explore this potential, and the results are summarized in the second Table below. We can see that, on ImageNet-1K dataset with MobileNetV2 backbone, KernelWarehouse+ODConv($4\times$) **brings 1.12% absolute top-1 improvement** to ODConv($4\times$) while retaining the similar model size; **(4)** **Our work improves prior dynamic convolution research in three-fold**: **(a)** our main motivation, rethinking dynamic convolution to explore its performance boundary with a significantly large kernel number (e.g., $n \gt 100$ instead of $n\lt 10$) while enjoying parameter efficiency, is not explored by existing works; **(b)** two critical insights, namely parameter dependencies within the same layer and parameter dependencies across successive layers, which create two technical pathways that serve as the basis to formulate our method KernelWarehouse, are also not explored by existing works; **(c)** with three interdependent components, namely kernel partition, warehouse sharing to multiple convolutional layers and the proposed attention function, our KernelWarehouse redefines the basic concepts of “kernels”, “assembling kernels” and “attention function” in dynamic convolution from the perspective of reducing kernel dimension and increasing kernel number significantly, and sets performance records for the dynamic convolution on different benchmarks. For the first time, KernelWarehouse can even reduce the model size of a ConvNet while improving the accuracy, as illustrated with MobileNetV2, ResNets and ConvNeXt backbones in Table 2 and Table 3 of the original manuscript. | Models | Parameters | Attention Function | Top-1 Acc (%) | Top-5 Acc (%) | | --- |:---:|:---:|:---:|:---:| | ResNet18 | 11.69M | - | 70.44 | 89.72 | | + DY-Conv ($4\times$) | 45.47M | Softmax | 73.82 | 91.48 | | + DY-Conv ($4\times$) | 45.47M | Ours| 73.74 | 91.45 | | + ODConv ($4\times$) | 44.90M | Softmax | 74.45 | 91.67 | | + ODConv ($4\times$) | 44.90M | Ours| 74.27 | 91.62 | | + KW ($1\times$) | 11.93M | Softmax | 72.67 | 90.82 | | + KW ($1\times$) | 11.93M | Ours| 74.77 | 92.13 | | + KW ($4\times$) | 45.86M | Softmax | 74.31 | 91.75 | | + KW ($4\times$) | 45.86M | Ours| 76.05 | 92.68 | | Models | Params | Top-1 Acc (%) | Top-5 Acc (%) | | --- |:---:|:---:|:---:| | MobileNetV2 ($1.0\times$) | 3.50M | 72.02 | 90.43 | | + ODConv ($4\times$) | 11.52M | 75.42 | 92.18 | | + KW ($4\times$) | 11.38M | 75.92 | 92.22 | | + KW & ODConv ($4\times$) | 12.51M | 76.54 | 92.35 |

Responses to Official Review by Reviewer MCn2: Part 4

4.**To your three constructive suggestions to improve the clarity of the paper** “Comments that might help with the clarity of the paper, no need to reply: "Kernel partition”…I believe…"Parameter Efficiency and Representation Power"…I believe...It would be nice to have some justification for Eq. 3…I'm sure there was a motivation or inspiration”. **Although you labeled “no need to reply” to these three constructive suggestions as**, we happily follow them to improve the clarity of our paper. Specifically, the updated manuscript includes three-fold presentation improvements: **Firstly**, “Kernel Partition” is performed across both spatial channel dimensions (illustrated by the “$W$” in Figure 1 and clarified in the paragraph titled Warehouse Sharing in the original manuscript). In the updated manuscript, we add an additional figure (Figure 2) to better illustrate “Kernel Partition” and “Warehouse Construction and Kernel Assembling” components of our method. **Secondly**, following your constructive suggestion, we use the backbone models pre-trained with KW($1/2\times$)|KW($1\times$)|KW($4\times$) in Table 2 and Table 3 of the original manuscript as exemplifications to better explain how our method can balance "Parameter Efficiency and Representation Power". **Thirdly**, we also add some justifications to better clarify the motivation and the insight of the proposed attention function (Equation 3). Our proposed attention function is specialized to fit three unique attentive mixture learning properties of KernelWarehouse: **(a)** the attentive mixture learning is applied to a dense local scale (kernel cells) instead of a holistic scale (whole kernels) via kernel partition and warehouse sharing; **(b)** the number of kernel cells in a warehouse is significantly large (e.g., $n>100$ instead of $n<10$); **(c)** a warehouse is shared to represent every kernel cell in multiple successive convolutional layers of a ConvNet. Under these design properties, popular attention functions used in existing dynamic convolutional works such as Sigmoid and Softmax work poorly for KernelWarehouse, but our proposed attention function works well, as tested by experimental results shown in Table 2 and Table 8. The success of the proposed attention function is due to its appealing property that can learn diverse attentions for all linear mixtures simultaneously and make the mixed kernel cells at multiple successive convolutional layers can learn informative and discriminative features hieratically. According to the definition of Equation 3, such a property comes from the binary attention initialization strategy (the second term) and the linear attention normalization function (the first term). Under different convolutional parameter budget $b$, the second term of the proposed attention function ensures that the initial valid kernel cells ($\beta_{ij}=1$) in a shared warehouse are uniformly allocated to represent different linear mixtures at multiple successive convolutional layers at the beginning of the model training, and the first term adopts a linear attention normalization function that **enables the existence of both negative and positive attention weights and encourages the training process to learn contrasting attention relationships** among all linear mixtures sharing the same warehouse facilitated by a temperature. Considering the above motivation and the functioning mechanism, we rename the proposed attention function to “Contrasting-driven Attention Function (CAF)” in the updated manuscript. **Finally**, regarding more experiments and discussions that we have made during the rebuttal phase, you are referred to our top-level comments titled **“The Summary of Our Responses to All Official Reviews”**, our responses to the other reviewers, and the revised manuscript.