Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers

Summary

The paper generalizes ProxConnect with forward-backward quantizers and introduces ProxConnect++ that includes some binarization techniques as special cases. With the derived ProxConnect++, the paper proposes BNN++ to illustrate the effectiveness of ProxConnect++. Experiments show the advantages of BNN++ on image classification benchmarks with CNNs and vision transformers.

Strengths

The paper generalizes ProxConnect and presents the unified framework ProxConnect++, which includes some binarization techniques as special cases. With ProxConnect++, one may design new forward-backward quantizers. Extensive experiments on CNNs and vision transformers demonstrate the advantages of the proposed ProxConnect++.

Weaknesses

The paper claims that it develops a unified framework ProxConnect++, which allows to design various forward-backward quantizers. However, the forward and backward quantizers used in the paper are limited to the existing ones. To show the advantages of ProxConnect++, it should design new forward and backward quantizers, which outperform the existing ones. The paper only gives an example BNN++, which is also inspired the existing forward and backward quantizers.

Questions

The paper presents a unified framework ProxConnect++. It is claimed that ProxConnect++ can include many binarization techniques as special cases. I have two concerns for the unified framework. First, it is desirable to present a unified framework. So one can analyze existing algorithms in a unified way. As a unified framework, the most important thing is to propose new superior algorithms following the framework. However, the paper does not give more insights into developing new algorithms. Note that BNN++ is also inspired by the exiting forward and backward quantizers. Second, the paper compares with some variants of ProxConnect++ and shows that BNN++ performs the best. It is better to include other state-of-the-art binarization techniques in the experiments.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors addressed the limitations of their work. There is no negative societal impact.

Summary

Thanks to authors for submitting their work to NeuRips 2023. After nicelly flowing introduction the lines 47-58 lay goals of the paper and its contributions out in the context of recent advances, cf. Dockhorn et al. [13], PC (ProxConnect). In particular, the paper generalizes PC to forward-backward binarization, as opposed to PC using only forward binarization, labelling this more general approach PC++. As a practical result of presented theory the fully binarized (8-bit integers) BNN++ is shown to be competitive against existing approaches on most tasks, including experiments on several datasets and architectures (incl. CNN and ViT). It achieves 30x reduction in memory and storage with a modest 5-10% accuracy drop compared to full precision training. Binarization results on vision transformers, rare up until now, is especially timely and promising given the recent deployment of the transformers in deep learning. Architecture ablation experiments conducted demonstrate practicality of the hereby developed theory and increase potential impact of the paper. Besides Theorem 1, being rather a straightforward extension of previous work Dockhorn et al. [13], including convergence guarantees applied on PC++ in appendix, I value authors used this general result to present a unifying and pragmatic framework for binarization of the NN in response to increasing energy demands for training and fine-tuning the recent large transformer models. I find paper timely and relevant contribution to research community.

Strengths

+ $\textbf{[Accessibility, Approach]}$ The goal of the paper is achieved in very accessible and neat form. Proving Theorem 1 (the main result) introduces a compelling and general framework for designing binarization algorithms for NNs. + ProxConnect (3) ++ >>> nice idea of formally adding transformation T that “cancels out” (formally in the Theorem 1 sense) in backward step, so the original theory of PC, Dockhorn et al. [13], applies. + $\textbf{[Unifying framework]}$ anAdditional and very impactful merit of the paper is its unifying approach of currently quite scattered and uneasy to access research of neural networks binarization. Especially natural inclusion of the transformers is very timely, convenient and impactful addressed by only few previous works, e.g. Y. He et al. [19]. + $\textbf{[Demonstration of practical usefulness]}$ (Architecture) Ablation experiments present practical usefulness of hereby developed theory (Theorem 1) (line: 283-284) and increasing the impact of the paper on further research.

Weaknesses

- Is the extension of previous work of Dockhorn et al. [13] novel enough? As noted above in Summary and Strength sections, formally adding transformation $T$ that “cancels out” (formally in the Theorem 1 sense) in backward step, so the original theory of PC, Dockhorn et al. [13], applies is smart yet rather straightforward (also demonstrated in the proof in Appendix). Some may see this rather a direct consequence of the previous work, that should be fully contributed to the respective authors :-)... .Despite this caveat I believe the paper's focus on application of this theory outweighs the theoretical merits and yields a work with potential high impact on further advances in the field. I leave to authors without impacting my decision whether or not to recognise previous work of Dockhorn et al. [13] more and consider renaming the Theorem 1 to "Corollary" to refer directly to previous work. - Extend of experiments in terms of datasets and modes is well selected. Yet number of runs to avoid statistical error, especially in Tables $3$ and $4$, is insufficient and should be higher. See Questions section.

Questions

- Would it be possible to Increase an extend of randomly seeded pool of experiments for Table 3,4? - reported averages on 3 runs together with rather diminishing differences between columns may not be significant. Because the results are used to draw performance claims, as on line 250, 251, “In particular, for end-to-end training, the best performing ProxConnect++ algorithms achieve ≈ 5% accuracy drop”, the extended experiments are suggested to be reported in camera ready version. Especially as it concerns ViT, treated rarely before and thus of high impact and interest to community. - The Figure 2., p.8, presents well why more involved techniques that post-training binarization (PTB) are needed. Would it be worth to use it in the beginning of the paper as a motivation figure? Accompanied with similar figure of memory and storage requirements, perhaps. - What does Table 2 show? (accuracy?) While it is commented in the text it is not clear from Table captions. I recommend to factor it in for the sake of higher clarity. - Line 136: “proved” —> “proven”

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Suggestion: For a camera ready version notes on limitation of the method are suggested to be brought back to main body of the paper from the Appendix G.

Summary

This paper proposes a new framework for training neural networks with binary weights, which generalizes ProxConnect and takes it as a special case. In the new framework, forward and backward quantizers are defined. A consistency result of the two quanziters is derived (Theorem 1). Extensive experiments are conducted to evaluate the usefulness of the new framework.

Strengths

1. The paper is very well-written and easy to the follow. I love the clarity of the paper. 2. The theoretical framework is sensible and the analysis is rigorous. I did not find a technical flaw. 3. The empirical evaluation is sufficient. The code is provided for reproduction.

Weaknesses

1. The proposed framework seems a direct generalization of ProxConnect. Thus, it's novelty is somewhat limited. 2. It is not straightforward for practitioners how to use the framework to design new proximal quantizers. Please provide practical guidelines and tips.

Questions

See above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

See above.

Summary

This paper studied binary neural networks which extend the existing theory of ProxConnect(PC) to ProxConnect++ and explored the fully binarized scenario, where the dot-product accumulators are also quantized to 8-bit integers. The authors also proposed BNN++ with non-linear forward and backward approximation to the sign function. Authors did experiments on CIFAR10 and ImageNet datasets.

Strengths

Binarizing weights and activations in neural networks is a challenging problem worth studying. This paper proposed an in-depth study of forward and backward functions for binary neural networks and proposed a new method that performs outstanding accuracy on CIFAR10 and ImageNet datasets.

Weaknesses

The authors only compared their method with several BNN methods. But many classic and famous BNN works are not even mentioned, such as XNOR-Net. And some papers studying the backward approximation are missing, e.g., the second-order approximation to the sign function proposed in Bi-Real Net should be compared in Fig.1. I would suggest authors compare or at least mention the binary neural networks to make the paper more comprehensive. To name a few: [1] Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. (CVPR) [2] Zhijun Tu, Xinghao Chen, Pengju Ren, and Yunhe Wang. Adabin: Improving binary neural networks with adaptive binary sets. (ECCV) [3] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. (ECCV) [4] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. (ECCV) [5] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. (CVPR) [6] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. (ICLR) [7] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. (ECCV) [8] Chunlei Liu, Peng Chen, Bohan Zhuang, Chunhua Shen, Baochang Zhang, and Wenrui Ding. Sa-bnn: State-aware binary neural network. (AAAI) [9] Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia-Wen Lin. Rotated binary neural network. (NeurIPS) [10] Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng. How do adam and training strategies help bnns optimization. (ICML) [11] Hyungjun Kim, Jihoon Park, Changhun Lee, and Jae-Joon Kim. Improving accuracy of binary neural networks using unbalanced activation distribution. (CVPR) [12] Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang-Ting Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. (NeurIPS)

Questions

The authors use the non-linear approximation of the sign function in the forward pass, will that yield other real-valued outputs besides the binary values? Line 170 is confusing: “BNN++ is more desirable than BNN+ empirically.” The author means BNN++ is better than PC++?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Did our rebuttal address your concerns?

Dear reviewer YJmg, as the discussion deadline is approaching, we are wondering if our response has addressed your concerns. We would be happy to answer any of your further questions. Thank you!

Did our rebuttal address your concerns?

Dear reviewer WcKF, as the discussion deadline is approaching, we are wondering if our response has addressed your concerns. We would be happy to answer any of your further questions. Thank you!

Did our rebuttal address your concerns?

Dear reviewer fayS, as the discussion deadline is approaching, we are wondering if our response has addressed your concerns. We would be happy to answer any of your further questions. Thank you!