Deep learning is computationally intensive, with significant efforts focused on reducing arithmetic complexity, particularly regarding energy consumption dominated by data movement. While existing literature emphasizes inference, training is considerably more resource-intensive. This paper proposes a novel mathematical principle by introducing the notion of Boolean variation such that neurons made of Boolean weights and inputs can be trained -- for the first time -- efficiently in Boolean domain using Boolean logic instead of gradient descent and real arithmetic. We explore its convergence, conduct extensively experimental benchmarking, and provide consistent complexity evaluation by considering chip architecture, memory hierarchy, dataflow, and arithmetic precision. Our approach achieves baseline full-precision accuracy in ImageNet classification and surpasses state-of-the-art results in semantic segmentation, with notable performance in image super-resolution, and natural language understanding with transformer-based models. Moreover, it significantly reduces energy consumption during both training and inference.
Paper
Similar papers
Peer review
Summary
This paper proposes a novel mathematical concept, termed Boolean variation, and demonstrates that it can be used to define optimization algorithms for training deep neural networks (NNs) consisting of Boolean weights and activations. This is in stark contrast to contemporary binarized NNs, which achieve efficient inference but rely on floating-point latent weights during training, and hence are relatively computationally expensive. Specifically, the authors introduce Boolean variation under Boolean algebra and show how it can be related to integer and real arithmetic to define NN blocks that operate with mixed-signals (e.g., real and Boolean). The authors then develop an optimization algorithm that can be used to train NNs with Boolean weights and activations, without the need of floating-point latent weights, and present a convergence analysis of the algorithm. Utilizing the newly introduced Boolean variation and its optimization algorithm that integrates well into existing deep learning frameworks, the authors conduct a wide range of experiments within various domains (image classification, super-resolution, image segmentation and NLP). The experimental results show that the proposed method achieves performance almost at the level of full floating-point models while consuming significantly less energy during training and inference. In almost all benchmarks, the proposed method sets the SOTA compared to binarized NNs.
Strengths
- The work introduces a novel mathematical concept (that of Boolean variation) and demonstrates how it can be used to define optimization algorithms for training NNs with Boolean weights and activations. This offers a different (more energy efficient) path for arriving at optimized binary NNs for efficient inference vs. existing related work on binarized NNs (that rely on floating-point latent weights for training). - The related work is adequately cited. - Theoretical analysis is presented for the proposed optimization algorithm. - The authors perform experiments across various deep learning application domains (image classification, super-resolution, image segmentation and NLP). The experimental results support their claims. - The methodology of the paper is clear (detailed derivations of theoretical statements are provided, code samples in Python are provided, and significant discussion is presented on training regularization techniques, model architectures used for experiments, as well as how the energy estimation was conducted). - The proposed method advances the SOTA with respect to binarized NNs across various deep learning domains. Moreover, the proposed method offers a potential solution to reduce the energy demands of training large models. The ideas presented are therefore of both theoretical and practical interest to the NN and deep learning communities.
Weaknesses
- Small points on clarity and quality for experiments (see Questions for details)
Questions
- **Methods / Experimental setting:** What logic gate/s were used for $\text{L}$? Were different gate types tested, and were there significant performance differences? Is a mix of gate types better than a single gate type? Do different architectures/blocks perform best with different gate types? - **Experimental results:** On Line 279 it reads: “Particularly noteworthy are the superior results achieved on SET14 and BSD100”. In Table 3, all BOLD results have less PSNR than SMALL EDSR. The results are strong, although “superior” made me think these were instances in which BOLD outperformed a FP baseline.
Rating
9
Confidence
3
Soundness
4
Presentation
4
Contribution
4
Limitations
The authors have adequately addressed limitations.
Summary
The paper introduces an innovative approach to deep learning by flipping the binary weights and inputs. This method optimizes convergence in the Boolean domain, promising significant improvements in training efficiency and energy consumption. The extensive experiments conducted validate the effectiveness of this approach. Pros: - Extensive Experiments: The paper conducts comprehensive experiments to validate the proposed method. - First to Provide Convergence Bound for Binary Weight: This is the first work to present a convergence bound for binary weights, a significant advancement over previous works that only considered convergence bounds for latent weights. - Entire Binary Data Stream: The method operates on a completely binary data stream, ensuring reduced complexity and improved efficiency. Cons: - Assumption A6 (E[Q0(w) | w] = w): The paper assumes E[Q0(w)∣w]=wE[Q0(w) | w] = wE[Q0(w)∣w]=w. However, the justification for this assumption is unclear and needs further explanation. Suggestions: - Ablation Study on \beta_t: An ablation study on the parameter $\beta_t$ could provide more insights into its impact on the model's performance and convergence. - Stochastic Rounding: The current method of stochastic rounding by taking the sign might not ensure that the expectation equals zero. Investigating alternative rounding methods could enhance the performance and reliability of the proposed approach.
Strengths
- Extensive Experiments: The paper conducts comprehensive experiments to validate the proposed method. - First to Provide Convergence Bound for Binary Weight: This is the first work to present a convergence bound for binary weights, a significant advancement over previous works that only considered convergence bounds for latent weights. - Entire Binary Data Stream: The method operates on a completely binary data stream, ensuring reduced complexity and improved efficiency.
Weaknesses
- Assumption A6 (E[Q0(w) | w] = w): The paper assumes E[Q0(w)∣w]=wE[Q0(w) | w] = wE[Q0(w)∣w]=w. However, the justification for this assumption is unclear and needs further explanation.
Questions
Suggestions: - Ablation Study on \beta_t: An ablation study on the parameter $\beta_t$ could provide more insights into its impact on the model's performance and convergence. - Stochastic Rounding: The current method of stochastic rounding by taking the sign might not ensure that the expectation equals zero. Investigating alternative rounding methods could enhance the performance and reliability of the proposed approach.
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
N/A
Summary
Proposal of a training method for binary (or ternary) neural networks (inputs and weights are 2-valued, -1,+1, or 3-valued, -1,0,+1), which works without latent floating-point weights, but directly adapts the binary (or ternary) weights, thus improving training (especially in terms of memory consumption). A kind of "discretized" calculus for binary (or ternary) weights is derived for this purpose.
Strengths
The approach seems to hold considerable potential as it bypasses the need for latent floating-point weights for training binary neural networks. The theoretical underpinning based on a "discretized" calculus for binary (or ternary) numbers appears to be sound and general. A convergence result is presented, as well as various experimental results that support the claims.
Weaknesses
The presentation of the theory is extremely condensed, with a lot of specifics delegated to the appendix. Together with the appendix, the paper has more than 50 pages and hence would be long even for a journal publication (which might be more appropriate for this material, since it requires quite some preliminaries for which a conference publication does not offer appropriate space). As a consequence, the paper left me with a lot of questions (see below, maybe some are answered in the appendix, which I did not read in detail, as it is simply too long). This makes it a bit difficult to properly judge the merits of the paper. In principle, the idea is intriguing and I see a lot of potential in it, but I doubt that the very condensed presentation is appealing and conveys the intricacies well enough.
Questions
Forward activation: Wouldn't it be natural to define y = 0 if s = tau for 3-valued logic? As it is written, the threshold function appears to be the same for 2-valued and 3-valued logic. Is this intentional? Same with Def. 3.6: Shouldn't there be some order relations involving 0? (I understand that it is stated that a and b are in B, but why this restriction?) Def. 3.7 uses 0, so this seems to be 3-valued logic, at least for the result, while a and b are again in B. The mixed use of 2- and 3-valued logic is confusing. The notation with the caligraphic F does not seem to be defined anywhere, not even in the Appendix. What exactly does it mean? It's some function class, but what are the two arguments to F, given that functions f from F have only one argument? Domain and image space of f? If I understand correctly, pre-activations are *not* Boolean, but integer. There is counting involved, not just Boolean operations (even though one can, of course, construct a Boolean gate network that performs the counting, but that requires something like 7n logic gates (asymptotically) for counting the 1s in n bits). I do not really understand how this counting is modeled and whether it is properly taken into account for the cost/energy evaluation (should be in E.3.3, I guess, but I don't really see it there). More sophisticated optimizers, as described on page 6, seem to need floating point numbers for the aggregates q, not just Boolean values, right? If that is the case, does that not defeat the purpose? Because the weights may now be Boolean (2- or 3-valued), but with each weight at least one floating point aggregate has to be associated (depending on the optimizer). Doesn't this considerably reduces the memory gains during training? And how is it with batch normalization (Figure 1 shows BOLD with batch norm)? Floating point? Discretized to Boolean values before fed into next layer? How are the Boolean functions L chosen? Can they be any Boolean function, possibly even including those that are constant or ignore one input? Are they the same for all neurons? Or are they different and maybe randomly initialized? I understand that the use of xnor in Section 3.3 is only an example (this is how I interpret page 5, line 199) and not meant to say that L is always xnor.
Rating
6
Confidence
3
Soundness
3
Presentation
2
Contribution
3
Limitations
A short paragraph on page 9 addresses potential limitations.
Official Comment by Authors
Dear Reviewer VF3V, We are grateful for the time and effort you have dedicated to reviewing our work. We have carefully studied and responded to each of your concerns and suggestions. We sincerely hope that our response addresses your concerns and will be at your disposal for any further questions you may have. Best regards, Authors
Summary
The paper explores advancements in binary neural networks (BNNs), particularly focusing on Boolean-weighted methods. It highlights the limitations of current BNN approaches and proposes a new method, B⊕LD, that operates directly on Boolean weights, improving energy efficiency and accuracy without relying heavily on full-precision components.
Strengths
The proposed method significantly reduces energy consumption compared to existing BNN approaches by directly operating on Boolean weights. The method achieves higher accuracy than several state-of-the-art BNN methods, as demonstrated on the CIFAR10 dataset.
Weaknesses
The method's reliance on Boolean weights may limit its adaptability to new data and tasks that require more flexible representations.
Questions
What are the practical implications of deploying the B⊕LD method in real-world applications with respect to computational resource requirements and scalability?
Rating
7
Confidence
1
Soundness
3
Presentation
3
Contribution
3
Limitations
NA
Official Comment by Authors
Dear Reviewer faoj, We are grateful for the time and effort you have dedicated to reviewing our work. We have carefully studied and responded to each of your concerns and suggestions. We sincerely hope that our response addresses your concerns and will be at your disposal for any further questions you may have. Best regards, Authors
Summary
The paper proposes a strategy for 1-bit backpropagating through the logic expressions of binary neural networks, drastically improving training speeds of BNNs.
Strengths
It is great to see a paper tackling 1-bit precision training. This is a really important topic, and tackling it on the level of logic is very interesting. With the right hardware, the approach could lead to drastic training speedups.
Weaknesses
Not in any particular order. 1. The grammar in the sentence “It is clear that there can be only a unique family of binary activation functions that is the threshold one.” is unclear. 2. The notation $f’(x)$ appears to be overloaded (first defined as the variation of $f$ w.r.t. x, then defined as the derivative of $f$.) 3. Notationally, I find it a bit odd that a binary value or Boolean value is called “logic” and a logic operation is called “logic connective”. 4. It is hard to follow l. 169 ”its derivative has been defined in the literature as …” because there is no reference to any literature. In general it would have been helpful to have references in Section 3, as it is unclear whether these notations 3.1—3.10 are proposed by you or are common in some literature you mention but not cite. Also reviewing the supplementary material, I didn’t find a reference for the notation or the derivative that "has been defined in the literature”. 5. What does the notation of an equality with a superscript of “xnor” mean? 6. At first, it is not clear whether Figure 1 relates to training or inference cost or whether is estimated; a reference to Table 2 would also be helpful. Reviewing the supplementary, details of the estimation become clear, but it would be helpful to already more clearly understand how you come to the numbers and without significant effort, it is difficult to follow the 4.5 pages that state the assumptions for the energy estimation. After rereading the paper, I am still left wondering whether any of the V100 energy consumption reductions are measured or just “we estimate analytically energy consumption”. The sentences l. 243-245 imply it is estimated, but how could you estimate the energy for a device V100 to be efficient, if you cannot measure it because it is not energy efficient in reality? If you estimate it and it is in fact not efficient, it means that your estimate is incorrect and uses assumptions that do not fit the V100 device. 7. In Equation 7, it seems like the variation (gradient?) can be larger than 1. Am I misunderstanding this? Then, in Eq. 9, we apply xnor between an integer and a ternary weight? 8. Given that you respond to provide your code in the checklist, and provide a few code snippets in Apx. B, it would have been really helpful if you had actually provided executable code, rather than snippets that leave significant room for interpretation. Unfortunately, the code is not directly copyable, and reimplementing it is out of scope for reviewing. 9. What is the precision used for the computations and accumulator states within the optimizer? Are they included in the energy estimate? In the code, accumulator seems to be a full precision float (param_group[’accums’]). 10. On the boolean optimizer, does it make sense to replace “accumulator” with “momentum” or is that an inappropriate correspondance. It would be great if the notations are aligned with existing literature. 11. What is the “VGG-SMALL” architecture? Reviewing [86], I do not find a “small” architecture for CIFAR-10. 12. What is “base” in table 5? “‘Base’ is the mapping dimension of the first layer” doesn’t clarify it for me unambiguously. Does it mean the number of channels after the first layer? What is the precision of this first layer? Do you use binary inputs? Why do the larger models (e.g., B⊕LD (Base 256)) use such a base of 256, that would be 4x the original number of channels? If you are using full precision inputs, that would mean that you made the most expensive layer 4x as expensive here, giving you a potentially unfair advantage over the baselines (as your model is bigger and more expensive.) 13. Details on which layers exactly are binary and which are integer or full precision for each architecture are missing. 14. It would be helpful to include in each (2-5), whether you are using only logic or also floating point precision more expliclity. “FP shortcut” clearly indicates floats. It should be pointed out more expliclity that the only ImageNet model with drastic speedups achieves 51.8% accuracy. For the KD, knowledge distallation, it could be unfair to specify a training cost reduction, if training requires first training a larger full precision network. 15. No details on the backpropagation through softmax and attention in the BERT experiments is given beyond the statement "extreme binarization of transformer-based model is not trivial”. 16. The regularization details (Apx. C) seem to be nowhere mentioned in the main paper. 17. Number of epochs are very important in extremely low precision training, it seems like the paper uses substantially more epochs than typical for the data sets. This makes sense as such training usually requires more epochs than regular training, but it would be very important to clarify this in the main paper. This is an important limitation because if we need 2-5x as many epochs, the training time savings are reduced. 18. Regarding related work, it is already extensive. Recently, "Differentiable Weightless Neural Networks" (DWNN, https://openreview.net/pdf?id=GBxflz0qdX) was published. This paper covers backpropagation through differentiable logic gate networks via binary precision, see first equation of paragraph "Finite Difference (FD)". While DWNN is designed for logic gate networks (in particular differentiable logic gate networks (https://arxiv.org/pdf/2210.08277)) and not for binary neural networks, DWNN still somewhat applies to binary neural networks. (full-1bit BNNs are LGNs, but not all LGNs are BNNs.) As your work and DWNN both deal with binary training of a network that is considered from a logic perspective, a short discussion would be great, in particular when considering your all encompassing "for the first time" claims. (I am not arguing against the claim, as the mentioned works do LGNs directly and you do BNNs (from an LGN perspective), but there are methodological relevances.)
Questions
Listed in weaknesses.
Rating
5
Confidence
4
Soundness
2
Presentation
2
Contribution
4
Limitations
Listed as references to paragraphs in weaknesses: 6. 9. 12. 14. 17. No potential negative societal impact.
Additional Response to Reviewer xoY6
As mentioned in our rebuttal, due to many questions raised, we are unable to address all questions within the 6000-character limit. Therefore, we only address the main questions in the Limitations (Q 6, 9, 12, 13, 14) in the main rebuttal. In this and following comments, we aim to respond to the remaining minor questions. These additional comments have been submitted before the rebuttal deadline. These comments comply with the instructions from the PC, which state: *"The author rebuttal period ends on Aug 6, 11:59pm AoE ... Comments to paper and reviews will be fine. Comments can be seen in time."* We hope the reviewer will consider all our responses, as they comprehensively address the raised questions. --- **Q1. The grammar in the sentence “It is clear that there can be only a unique family of binary activation functions that is the threshold one.” is unclear.** Thank you for your feedback. We will revise that sentence as follows: "It is clear that there can only be one unique family of binary activation functions, which is the threshold function". --- **Q2. The notation f'(x) appears to be overloaded (first defined as the variation of f w.r.t. x, then defined as the derivative of .)** Thank you for your comment. Indeed, it is intentional. One of the advantages of our method is that it can support a combination of Boolean and full-precision components. We use a general notation for both Boolean variation and real derivative for convenience when we consider mixed Boolean and full-precision data types, see, for example, Definition 3.10 and Theorem 3.11. Nevertheless, to avoid any misunderstanding, we explicitly mentioned these definitions at lines 164 and 169. --- **Q3. Notationally, I find it a bit odd that a binary value or Boolean value is called “logic” and a logic operation is called “logic connective”.** A value of the Boolean logic or of the three-valued logic M is called a logic value to emphasize its difference to a numerical value such as real or integer. In the standard mathematical logic, logic connectives are similar to logic operators and are required when defining a new logic. --- **Q4. It is hard to follow l. 169 ”its derivative has been defined in the literature as …” because there is no reference to any literature. In general it would have been helpful to have references in Section 3, as it is unclear whether these notations 3.1—3.10 are proposed by you or are common in some literature you mention but not cite. Also reviewing the supplementary material, I didn’t find a reference for the notation or the derivative that "has been defined in the literature”.** We regret that this part created some confusion to the reviewer. Line 169 reminds the existing definition of the discrete derivative, i.e., of a real-valued function taking integer variable. Regarding that this existing definition is quite straightforward and common sense, we didn’t provide a specific reference for it. Line 169 is the only existing definition that is not ours. By using the proposed notion of the Boolean variation, Definition 3.10 extends the existing definition of the discrete derivative to a function that takes an integer variable and values in numeric or logic domain. --- **Q5. What does the notation of an equality with a superscript of “xnor” mean?** In equations (4) – (6), equality with a superscript of “xnor” means that when the logic operator L used for the neuron is a XNOR gate as we previously mentioned in line 198 – 199. We thank the reviewer for this feedback, we will make it more explicit by replacing it with “L = xnor”. --- **Q7. In Equation 7, it seems like the variation (gradient?) can be larger than 1. Am I misunderstanding this? Then, in Eq. 9, we apply xnor between an integer and a ternary weight?** Yes, the variation can be greater than 1 as it is the aggregated signal. For Eq 9 and others, mixed-type logic is as defined in Definition 3.5. --- **Q8. Given that you respond to provide your code in the checklist, and provide a few code snippets in Apx. B, it would have been really helpful if you had actually provided executable code, rather than snippets that leave significant room for interpretation...** To respect the industrial confidentiality aspect, we could not provide the source code. However, the example code that we provided is a true python code, not simply a pseudocode (which could have been also acceptable otherwise). This all not only complies with the conference submission policy but also shows our motivation to provide the best we could. Additionally, we included extensive appendices detailing our experiments for reproducibility and providing further insights for the community and practitioners.
Additional Response to Reviewer xoY6 (con't)
**Q10. Does it make sense to replace “accumulator” with “momentum” or is that an inappropriate correspondance.** In the current form of our Boolean optimizer, its accumulator places precisely the role of an accumulator. After that, it can be called “momentum” if preferred at the terminology consideration. --- **Q11. What is the “VGG-SMALL” architecture? Reviewing [86], I do not find a “small” architecture for CIFAR-10.** Indeed, although VGG Small is not explicitly mentioned in [86], it is actually widely used and considered as a standard benchmark in the binarized neural network community. Some prominent works use that architecture are: binary connect, XNOR-Net, IR-Net, etc. As mentioned in the paper of binary connect [24], that architecture is greatly inspired from original VGG architecture [86]. Nevertheless, in the camera-ready we will cite both [86, 24] for that architecture. --- **Q12. What is the precision of this first layer?** To complement our response to Q12 in the rebuttal, we indicate below the lines of code showing the first and last full precision layers among the competing methodologies that provide code in their official git repository. In this case, they simply use the default operation of the framework (as it already uses FP parameters): - XnorNet: script models/alexnetxnor.lua, lines 32 and 46 - BirealNet: script BiReal18_34/birealnet.py, lines 94 and 103. - ReactNet: script resnet/1_step1/birealnet.py, lines 112 and 121. - BNext: script src/bnext.py, lines 462 and 472. --- **Q15. No details on the backpropagation through softmax and attention in the BERT experiments is given beyond the statement "extreme binarization of transformer-based model is not trivial”.** Thanks for your comment. Indeed, as mentioned at line 311, "Our Boolean BERT model is inspired by BiT for binarizing activations and incorporating KD during training, where the full-precision teacher guides the student in a layer-wise manner". We follow the experimental setup of BiT, including using the same method for binarizing activations and backpropagation for Softmax and attention in the BERT model. Unlike BiT, which uses full-precision latent weights, our method operates directly in the Boolean domain. --- **Q16. The regularization details (Apx. C) seem to be nowhere mentioned in the main paper.** We respectfully disagree with this comment as in the last paragraph from the line 208 we explicitly mentioned that "We encourage the readers to check the detailed implementations, practical considerations, and example codes of our proposed method, available in Append. B and Append. C." --- **Q17. Number of epochs are very important in extremely low precision training, it seems like the paper uses substantially more epochs than typical for the data sets...** Thanks for your comment. As the reviewer mentioned, it is common for training low-precision models to require more epochs compared to full-precision ones. However, each epoch takes significantly less time, resulting in a substantial reduction in total training time. Additionally, we provided the number of training epochs in our experiments in Append. D. Compared to the BNN baselines, we use at least the same or fewer epochs. For example, on ImageNet, BNext-18 requires 512 epochs, whereas our models are trained with 300 epochs. Moreover, our models are still much more energy-efficient. --- **Q18. Regarding related work, it is already extensive. Recently, "Differentiable Weightless Neural Networks" (DWNN) was published...** Thanks for your comment and for acknowledging that our sections of related work are already extensive. Unfortunately, the suggested paper was actually published nearly two months after the NeurIPS submission deadline. However, we will add the following discussion in the revised paper. *"Another related line of research involves logic gate networks (LGNs) [1]. In LGNs, each neuron functions as a binary logic gate and, consequently, has only two inputs. Unlike traditional neural networks, LGNs do not utilize weights; instead, they are parameterized by selecting a specific logic gate for each neuron, which can be learned. Compared to the standard neural networks or our proposed method, LGNs are sparse because each neuron receives only 2 inputs, rather than m inputs, where m is the number of neurons per layer. Recently, [2] expanded on LGNs by incorporating flexible and differentiable lookup tables. While these advancements show promises, adapting them to modern neural network architectures such as CNNs or Transformers is challenging. Furthermore, these approaches have not been validated on large-scale datasets like ImageNet or on tasks that require high precision, such as image segmentation or super-resolution, as demonstrated in our work."* Refs: [1] Petersen et al. Deep Differentiable Logic Gate Networks. NeurIPS 2022. [2] Bacellar et al. Differentiable Weightless Neural Networks. ICML 2024.
I will respond later to the other responses due to the length of the rebuttal, but I already wanted to respond to the potentially most pressing concern: **Q6:** Regarding V100, and your response > limited by current software and hardware capabilities, which lack neither per-bit memory allocation and my concern > how could you estimate the energy for a device V100 to be efficient, if you cannot measure it because it is not energy efficient in reality My issue with the V100 is indeed that this device lacks hardware functionalities that could enable the efficiency. But you are estimating what a hypothetical chip with similarities to V100 (?) but without the limitations of V100 could do, but presenting it as "V100", which makes up a big difference. It means it could never be done on a V100, which in my opinion was misrepresented. It is not clear to me in what way anything in this chip would be similar to V100, i.e., which criterion is used for "equivalence of hardware".... e.g., is it number of bits processed? Please correct me if I am misunderstanding something.
Official Comment by Authors
Thank you for having further clarified your concern, we are happy to provide you with the following answer. > **Q.**: *``in what way anything in this chip would be similar to V100, i.e., which criterion is used for "equivalence of hardware"....''.* **A**: In terms of computational complexity, in particular energy consumption (EC), equivalent hardware includes factors that impact data movement and computation EC. The factors of data movement EC include the chip's memory hierarchy (i.e., capacity of each cache level and memory bandwidth), dataflow, and data bitwidth. Factors of computation EC includes the utilized arithmetic logic unit (ALU) such as ADD INT8, ADD INT16, ADD FLT16, MULT FLT16, MULT FLT32, native LOGIC, etc. and compute unit architecture (also called Processing Element Array). To evaluate EC with V100 (same for Ascend or any existing hardware), our analytical model includes the following factors: - Actual factors of the chip: Memory hierarchy, dataflow, computing unit architecture (for instance, how many processing elements, inter-connection between PEs, etc). - Hypothetical factors: capability of 1-bit memory allocation (i.e., not allocating 1 byte for 1-bit data point), and capability of the computing unit in supporting native logic operator such as logical AND, OR, XOR, XNOR. We insist that our model includes information that is publicly available from the V100, going beyond this is cumbersome, as there is proprietary information that is not in the public domain. Given that current V100’s do not support Boolean computing, we should incline to an analytical model including representative features of the V100. We hope that this can answer your question.
My issue with the V100 thing is that the paper reads as if it could be possible to run it at this efficiency on V100, at least that is how I interpret, e.g., the caption "Figure 1: Comparisons of our method against notable BNNs on CIFAR10 with VGG-SMALL, evaluated on Nvidia Tesla V100." "x axis: Energy Consum. w.r.t. FP (%)" This is misleading, as hypothetical factors (1-bit memory allocation + supporting native logic operators) that are not the case imply that you cannot specify it as "Nvidia Tesla V100". It is not a perfect solution, but I'm thinking about something like "V100 equivalent" or "hypothetical V100 equivalent with native 1-bit support".
Official Comment from Authors
Thank you for your comment. We acknowledge your point. Indeed, starting from line 242, we explicitly mentioned that *"Given the absence of native Boolean accelerators, we estimate energy consumption analytically by evaluating the arithmetic operations, data movements within storage/processing units, and the energy cost of each operation. This analysis is applied to the Nvidia GPU (Tesla V100) and Ascend [62] architectures. Further details can be found in Appendix E."* To prevent any potential confusion, we will revise the caption of Fig. 1 to clarify that *"Comparisons of our method against notable BNNs on CIFAR10 using VGG-SMALL. The energy complexity is analytically evaluated considering the Nvidia Tesla V100 specifications; see Section 4 for more details."*
Thank you for your response. In my opinion, even the updated text and line 242 do not convey that the analysis is applied to a *modification of V100*. --- Regarding 16: Will you add details to the fact that you are doing (I assume very critical) training regularizations in the camera-ready? The line referenced does not convey this. In general, it will be helpful if you put an emphasis on providing a very high-level intuition of such details in the paper itself. --- Regarding 18: If I wasn't clear, I do not consider it as anything against the paper. The reason I mentioned DWN in the context of your paper are the equations and discussions around the bottom right of the third page in https://openreview.net/pdf?id=GBxflz0qdX. There, in the "Finite Difference (FD)" section, binary back propagation rules are discussed, which may (or may not) be expressible through the Boolean domain using Boolean logic, and thus relate to the "for the first time—efficiently in Boolean domain using Boolean logic instead of gradient descent and real arithmetic" claim. I wanted to emphasize that this methodological relevance should be discussed. Regarding technical correctness, DWN *was* published on May 1 on OpenReview thus before the deadline, but, again, I am not holding it against you. --- Regarding 4: If you say "has been defined in the literature", one would expect that you provide a reference. --- I appreciate the authors' effort in addressing the concerns raised. However, I observed that the rebuttal tends to be quite defensive, often referencing lines of the paper or providing additional details that are not included in the current manuscript, mostly without committing to making changes to the camera-ready. After carefully revisiting the mentioned locations and explanations by the authors, even when explicitly looking at their pointers, I often do not think that it is adequately explained in the paper or in a few cases remain as misleading as they were before. To avoid prolonging the discussion and to ensure clarity moving forward, I would like to suggest that the authors provide a list of the QXX IDs for all the points where they are committed to making clarifications in the camera-ready version. I believe this will help streamline the process and prevent the need to reiterate many of the concerns. (It might also be the authors' style to not state to address something in the camera-ready, but intend to still perform the clarifications in the camera-ready in order to avoid confusion of future readers of the publication, thus listing the IDs where they do intend to clarify the camera-ready might be helpful.)
Official Comment by Authors
Dear Reviewer xoY6, Thank you for further extending the discussion with your observations. We thank you in particular for the precious extensive suggestions and comments that you have given us. We have appreciated the rebuttal process as a unique, precious opportunity for exchanging between the reviewers and the authors. We have thoroughly addressed the reviewers’ comments. In our understanding, referencing to some parts of the current paper that can directly clarify the raised questions is a natural way. We have also provided additional insights when necessary, which have been well-received by other reviewers, such as Reviewer **ojfq** and Reviewer **HRmB**. In particular, in respect of your wish, this discussion process is unfortunately not endowered to clarify your any remaining concerns. We confirm that all the revised texts and modifications that we have presented to all the reviewers will be included in the camera-ready version as they enhance the quality of the paper. Following your last suggestion, here-after we iterate the list of items that will be committed to the camera-ready version. If any item may be missing, we would be highly grateful for letting us know. - Q1. Rephrasing the sentence as follows: *"It is clear that there can only be one unique family of binary activation functions, which is the threshold function"*. - Q2. Adding a sentence to stress that the notation $f'(x)$ is intentionally used to denote both Boolean variation and real derivative for convenience when we consider mixed Boolean and full-precision data types. - Q4. Providing a reference for that definition. - Q5. Making it more explicit by replacing it with “L = xnor”. - Q6. Revising the caption of Fig1. "Comparisons of our method against notable BNNs on CIFAR10 using VGG-SMALL. The energy complexity is analytically evaluated considering a hypothetical V100 equivalence with native 1-bit support; see Section 4 for more details." - Q11. Citing both [86, 24] for the VGG-SMALL architecture. - Q15. Refining the sentence from the line 311 as *"Our Boolean BERT model is inspired by BiT for binarizing activations and incorporating KD during training, where the full-precision teacher guides the student in a layer-wise manner. We follow the experimental setup of BiT, including using the same method for binarizing activations and backpropagation for Softmax and attention in the BERT model."* - Q16. Moving some of the regularization details from the appendix, such as *"C.1 Summary of the Main Techniques"* to the main paper. - Q18. Adding a discussion regarding LGNs and DWNN as presented previously in the rebuttal. - Minor point: Based on the official timeline of ICML 2024 *"Papers will not be public during the review period. Only accepted papers will be made public through OpenReview, and only after the camera-ready deadline."* (which was May 29, 2024 AoE this year), DWNN paper was only publicly available from May 29. (https://icml.cc/Conferences/2024/Dates, https://icml.cc/Conferences/2024/AuthorInstructions) These textual updates can be straightforwardly integrated into the camera-ready version. Once again, we thank the reviewer for your time and feedback. Best wishes, The Authors
Thanks for your response.
The response answered some of the questions clearly. Given the potential impact of this paper and its soundness, I will increase my score by 1 point.
I thank the authors for their response and appreciate their clarifications to and insight on the questions raised.
Decision
Accept (poster)