VanillaNet: the Power of Minimalism in Deep Learning

Summary

This paper aims to minimize the complex design of modern deep models. The authors propose an architecture, VanillaNet, which eliminates high depth, residual connections, non-linear activations, self-attentions, and normalization layers, but during training, the model still use some components, such as activations and normalizations, to help model training. They also propose an activation function series to boost the capacity and performance of the models. Due to the simplicity, the model is ideal for resource-constrained environments and has lower latency than complex deep models. Their experiments show that the proposed model has similar performance than modern deep models.

Strengths

1. The proposed model does not have high depth, residual connection, and normalization layers, which is beneficial for latency. 2. Despite the lack of complex design, the performance of the proposed idea is comparable to modern deep models. 3. The proposed activation function series offers performance boost with slightly more FLOPs and parameters.

Weaknesses

1. It would be better to have a figure indicating where the merging operation describe in section 3.1 occurs in the model, so it is difficult to see where the merging opportunities are. 2. The idea of merging batch normalization with convolution describe between line 132 and line 137 is an already proposed inference acceleration called Batch Normalization Folding. Please make a clear distinction between what design are novel in this paper and what design are coming from existing work. 3. The content between lines 177-183 calculates the complexity (which directly tie to FLOPs) of the activation function series and claims the cost is very small, but in lines 90-91, the authors claims FLOPs is not important, so they are contradicting. Rather, it would make more sense to compare the latency of proposed activation and convolution. 4. The latency (profiled using a batch of size 1) in Table 4 could be misleading. In many inference cases, multiple instances can be batched together to improve throughput and make better usage of available hardwares, so it would be much better if throughput or latency (measured using reasonably large batch size) is also provided. 5. Code is not provided.

Questions

The main concerns are listed above. In addition, is the proposed activation function series used in inference? If so, then the model still has activations and batch normalization (although the later one is folded into convolution), and compared ResNet, only the residual connections are removed. If not, does this mean all convolutions in each stage can be merged into a single convolutions? This is an important question since the price of VanillaNet is that the parameter count and FLOPs need to be significantly higher to achieve similar performance compared to baselines.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

There is no discussion on limitations. You should include a small limitation section. For example, you could include a discussion about the need for higher parameter count and FLOPs.

Summary

The paper presents VanillaNet, a neural network architecture that emphasizes simplicity by avoiding high depth, shortcuts, and complex operations like self-attention. The proposed VanillaNet performs comparably to well-known deep neural networks and vision transformers.

Strengths

**Originality**: To the best of my knowledge, the proposed VanillaNet is novel. **Quality**: This paper is of high quality since it proposed a new model architecture and conducted extensive experiments on ImageNet and ablation studies to demonstrate its effectiveness. **Clarity**: The paper is well-structured and clear, providing detailed explanations of the VanillaNet architecture and the proposed techniques. **Significance**: The paper is significant as it paves the way for a new direction in neural network design.

Weaknesses

- Although the proposed VanillaNet reduced the latency a lot, the required Params and FLOPs increased a lot. e.g. FOR SWIN-S and VannillaNet that have comparable accuracy on ImageNet, the proposed vanillaNet needs 127.8 M and 48.9 FLOPs while Swin-S only needs 49.6 M and 8.7B FLOPs.

Questions

- Since pre-training significantly impacts the model's performance, it would be better to see how the proposed model architecture performs in pre-training scenarios.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

the authors adequately addressed the limitations.

Summary

The paper introduces VanillaNet, a neural network architecture that emphasizes simplicity in design while maintaining remarkable performance in computer vision tasks. VanillaNet achieves this by avoiding excessive depth, shortcuts, and intricate operations like self-attention. The authors propose a "deep training" strategy to train VanillaNets, which starts with several layers containing non-linear activation functions and progressively eliminates them as training proceeds. Additionally, an efficient, series-based activation function incorporating multiple learnable affine transformations is introduced to augment the networks' non-linearity. The experimental results show that VanillaNet achieves comparable or even better performance than state-of-the-art deep neural networks and vision transformers on various computer vision tasks while being more efficient and resource-friendly.

Strengths

1. This paper is well-written and organized, with clear explanations of the proposed VanillaNet architecture. The paper also provides detailed experimental setups and results, making it easy to understand and reproduce the experiments. 2. The proposed VanillaNet is novel. By avoiding excessive depth, shortcuts, and intricate operations like self-attention, VanillaNet offers a streamlined and efficient design. This simplicity not only makes VanillaNet well-suited for resource-limited environments but also challenges the prevailing notion that increasing network complexity leads to improved performance, which may have implications for future research in the field. 3. The experimental results are convincing. Despite its simplicity and low depth, VanillaNet delivers performance on par with renowned deep neural networks and vision transformers. For example, a 13 layers' VanillaNet achieves an 83% top-1 accuracy on ImageNet.

Weaknesses

1. While VanillaNet's simplicity and low depth make it an attractive option for efficient deployment in GPUs, one weakness of the proposed method is that it has significantly more parameters and FLOPs than some existing methods. It is noted that VanillaNet has a large number of parameters in each layer to capture useful information from the images with limited non-linearity. This can make it more challenging to train and deploy VanillaNet on certain devices with limited computational resources. 2. The proposed deep training strategy is used to enhance the performance of VanillaNet by introducing stronger capacity during training, which suggests that the proposed method requires more training FLOPs.

Questions

1. Table 3 shows that shortcuts are useless for the proposed VanillaNets. As the shortcuts are one of the most popular components in modern networks, it would be better to provide more explainations about this phenomenon. 2. As shown in the suppmentary material, the VanillaNet-13 is achieved by adding convolutions on the 3rd stage of VanillaNet-7, what is the designed logic behind?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

yes.

Summary

This paper introduces VanillaNet, a neural network architecture that emphasizes simplicity in design. By discarding commonly used ingredients, e.g., high depth, shortcuts, and self-attention mechanism, VanillaNet is refreshingly concise. VanillaNet series have only 5-13 layers. The authors devise the “deep training” strategy, in which nonlinear activation functions are pruned after training to restore the original architecture. They also leverage series-based activation function to incorporate multiple learnable affine transformations. The experiments on classification and detection demonstrate the effectiveness of the proposed VanillaNet.

Strengths

Motivation: This paper is different from the previous research direction of the backbone architecture, abandoning the exploration of the residual connection or deepen the network. It shows the potential of the shallow network. The proposed VanillaNet avoids complex operations and make the structure more compact and easy to understand. Quality: The paper is clear and easy to follow. Originality: The “deep training technique” proposed in this paper is well motivated and effective. The activation function is gradually reduced to an identity mapping during training. And finally two convolutions can be easily merged into one convolution. I like this idea. In addition, the “series activation function” which enables global information exchange also seems effective. Experiment: The authors conduct sufficient experiments and ablation studies to demonstrate the effectiveness of the proposed VanillaNet. VanillaNet can achieve comparable performance to current SOTA models on several computer vision tasks with faster inference speed Overall, the motivation challenges the “more is different” foundational model philosophy, opening up new possibilities for efficient deployment of deep learning models.

Weaknesses

About the application of “deep training technique”. It seems that this technique can only be apllied to BN. If the normalization layer is LN, it cannot be merged. About the number of parameters. Although the inference speed of VanillaNet is superior to other methods, the heavy number of paramters and “deep train” strategy may increase the training cost of the network. According to my understanding, the amount of network parameters in training stage should be much larger than the paratmers shown in Table 4 (in inferece stage).

Questions

Inference speed: The authors report the latency on GPU and pytorch. However, in more scenarios (or platform), such as tensorRT, will the gap between VanillaNet and other networks become smaller? Series Informed Activation Function: What’s the difference between SIAF and BNET? I think these two methods should be explained in more details.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The authors propose a minimalist neural network backbone architecture as an alternative to current ResNets and ViTs. The minimalism is focussed on faster inference, avoiding complex operations with higher time complexity when implemented on modern GPUs. In their proposed architecture, individual operations are limited to convolutions, activation functions and linear projections, avoiding more time-complex operations like skip connections or self-attention. The resulting 6-layer network surpass latest CNN and ViT backbones at a fraction of inference latency on ImageNet top-1 accuracy metrics. A novel deep training strategy (merge two consecutive layers post-training) and series activation functions (stacked activation functions for more complex non-linearity under lower time complexity) are introduced to facilitate better learning by these minimalist shallow networks. In addition to the base architecture, a series of scaled-up versions provide increasing performance, still at minimal inference latency.

Strengths

1) Most of the paper is extremely well written with good logical flow, strong motivation for each proposal, and clear explanations of methodology. 2) Proposed deep training and series activation functions appear novel, well-motivated, and interesting. 3) Thorough ablations and discussion clearly demonstrate motivation for each design choice 4) Strong results over SOTA backbones for classification, detections, and segmentation 5) Detailed training details (in supp.) supports reproducibility of results

Weaknesses

1) Missing details in series activation function: it appears scale and bias terms (Eq 5, 6) are learned parameters - please explicitly mention this or otherwise explain how these values are selected. Assuming former, how are these values initialized? Does the initialization affect final outcome (maybe conduct some ablation)? What are the final values for these parameters that are learned across runs (do they collapse in any occasion? do they convey any information about data distribution)? Can these parameters be fixed at start instead of learning? 2) Writing (minor): grammatical errors in numerous places across sections 3 & 4, captions of figures not descriptive enough or self-contained (e.g. Table 4 - explain terms like depth, real-acc vs acc in caption - reader needs to skim through main text to understand headers of table). 3) More evaluations could strengthen the paper (and convince about universality of new backbone): maybe include results on some long-tailed classification datasets, more than one detection and segmentation dataset. 4) Section 4.2 & Figure 2 (attention map) does not appear to convey anything useful in current form. Consider updating description / main text to discuss one of the examples in detail (or maybe move to supplementary).

Questions

See 1 in weaknesses

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

No clear discussion of limitations.

Post Rebuttal

Thanks for the detailed rebuttal. Assuming updates to final manuscript in accordance to rebuttal, I raise my vote further. I strongly support acceptance for this paper.