Convolutional Differentiable Logic Gate Networks

Summary

This paper proposes a novel computational architecture for differentiable logic gate networks (LGNs), a machine learning methodology that aims to learn networks of logic gates for fast, gate-efficient inference on logic gate-based hardware. Specifically, the authors propose extensions to a prior work on differentiable LGNs inspired by the computer vision literature. They propose (i) logic gate tree convolutions which are layers that convolve trees of logic gates with the input, (ii) logical or pooling (inspired by max pooling) layers that compute the disjunction of the receptive field, and (iii) residual initializations that bias the initial distribution over logic gates to be an identity gate of one of the two inputs. The authors also detail various computational techniques to optimize the efficiency of the new architecture in training, simulation and hardware. Experimental results on computer vision tasks (CIFAR-10, MNIST, Fashion-MNIST) and extensive comparison to SOTA methods demonstrate that the proposed architecture achieves competitive (if not SOTA) accuracy while being either significantly smaller (in terms of gate count) or faster (in terms of inference speed on FPGAs or CPUs). The authors demonstrate through ablation studies that the proposed components and architectural choices are all integral to the achieved performance. Moreover, the authors provide experimental results for insightful studies on the proposed components, such as why logical or pooling doesn’t result in too much network activation, the induced stability of residual initializations, and the effects of gate distribution discretization.

Strengths

- The proposed methods build upon prior work on differentiable logic gate networks (LGN) and take inspiration from compute vision literature. The contributions are novel and advance the SOTA for fast and efficient inference of machine learning models. The results are of importance to embedded and real-time machine learning applications. - Related work is discussed in detail, and experimental results are compared to various prior works. - The submission is technically sound, with claims supported by experimental results (see weaknesses for point on statistical significance). The authors demonstrate through ablation studies the utility of each of the proposed architectural components, and discuss tradeoffs, strengths and weaknesses for their techniques. - Methods are detailed enough for reproducing the proposed architecture and results. - The authors provide substantial insight into methodology choices, making the presentation clear and informative.

Weaknesses

Lack of statistical significance measures for results. (The main prior work on differentiable LGNs (F. Petersen et al.) provides standard deviations in their appendix). However, the authors provide justification for this within the NeurIPS Paper Checklist.

Questions

- If we take the CIFAR-10 results as an example, another method achieved greater accuracy (91%, Hirtzlin et al.) but requires significantly more gates. What is the current limitation on scaling LogicTreeNets to larger gate counts (beyond 111M)? If, for example, a greater accuracy were desired. - Similar to the study conducted by F. Petersen et al., is there any insight to be gleaned from the learnt distribution over logic gates in logic gate tree convolution kernels? - Possible typos - Lines 344-345: is a forward slash missing?: “transistor count / chip area” - Line 351: is an M missing?: “55.8M gates”

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors have adequately addressed limitations.

Summary

In this work the authors propose a convolutional-like architecture along with two novel mechanisms oriented to differentiable logic gate neural networks, making the training and inference of such networks possible in more intense tasks in context of logic gate neural networks. More specifically, the authors augment the current state-of-the-art capabilities of Differentiable Logic Gate Networks by introducing a convolutional architecture and training approach that is based on logic gates and along with the proposed “logical or pooling” and “residual initialization” achieving higher accuracies in various dataset with lower number of gates, reducing significantly the inference time. The authors claim that the proposed method unlocks the capabilities of differentiable logic gate networks providing a comprehensive review on works that target efficient inference, discussing the benefits of adopting the proposed architecture on application that requires efficient and low cost inference.

Strengths

The paper is well organized providing a comprehensive review on methods that target efficient inference, discussing in depth the benefits of logic gate networks. The technical details, arguments and experimental results provided in the paper regarding the realization of the method in hardware are convincing. The authors provide some experimental results in actual hardware demonstrating the effectiveness of the proposed method in terms of efficiency. The authors provide interesting ablation studies to support experimentally some of the designing decisions. The motivation is solid and easy to understand. Additionally, the contribution is clear, achieving state-of-the-art performance in the context of larger logic neural networks.

Weaknesses

In many cases the work seems incremental in reference to [1], without, however, overcoming or justifying some theoretical lacks that are spotted in this previous work. More specifically, the random connections that applied in [1] are adopted in this work without well being theoretically justified or proposing an alternative way. In addition to that, I find myself referring occasionally to [1] in order to understand some technical details. For example, the differentiable logic gates are presented only schematically in the paper. From a technical point of view, I find it difficult to conceptualize the computational graph that is built during the training. Do the authors introduce a projection layer, parameterized by vectors z, for each channel of the kernel on the available gates? To this end, is the z vector optimized taking the partial derivative of z on the classification loss? Introducing some details on the optimization process will be useful. The random selection of inputs on the receptive fields raises concerns regarding the consistency of the training process, with the paper not providing error bars regarding different training runs. Although the authors discuss the reasons for attaching higher probability to the logic gate choice A (or B), they do not discuss how they conclude on $z_3 = 4.905$. Such empirical decisions potentially hinders the generalization ability of the proposed method. A proof reading is required. There are some minor typos in the paper and in the appendix. (e.x. Paper-L293 “LGNs differ from the LGNs”, missing reference in L6 of the appendix. [1] Petersen, Felix, et al. "Deep differentiable logic gate networks." Advances in Neural Information Processing Systems 35 (2022): 2006-2018.

Questions

How many additional trainable parameters are introduced during training in contrast to the traditional CNNs and which of them are discarded during the inference? How do the authors comment on the observation that the proposed method seems to not generalize equally well to smaller architectures? How do the authors conclude on $z_3 = 4.905$? I would like to stress the consistency of the proposed method in different training runs due to the fact that it is based on the random selection of inputs of the receptive field. Is the robustness of training preserved and what are their experimental observations?

Rating

8

Confidence

3

Soundness

4

Presentation

4

Contribution

4

Limitations

Some technical details in the training process are not clear. Some empirical decisions made on paper are not well justified neither experimentally nor theoretically. The proposed method leads to lower accuracies in smaller models (e.x. LogicTreeNet-S of the table 1) Taking into account that they promote logic gate choice A, it would be very interesting if the authors report the per layer distribution of logic gates after the training. This could be interesting also in contrast to Gaussian initialization.

Summary

The presented work is a significant extension to "Deep Differentiable Logic Gate Networks" previously presented at NeurIPS 2022 [7]. Additional contributions are the support for convolutions including logic gate trees / or-pooling and residual initializations. All additions together allow to train logic gate networks that are deeper, achieving SOTA accuracies and beyond while using remarkably fewer resources during inference and training as well. The authors promise to also make the code publicly available.

Strengths

Improving efficiency of (small scale) neural networks substantially. Lowest latency of all SOTA baseline results, the majority of them being much slower with even worse accuracy.

Weaknesses

There are a few issues with the clarity of the presentation. Upfront it should be mentioned that the Appendix is vital to understand many details (architectures, choice of parameters, memory usage, memory access, etc.) and should certainly be published as well. It was only supplementary material as part of the review. Figure 4 only shows an effect, but does not explain why pre or-pooling is superior to the other 2 and the text does neither. Lines 228-229 seem to contradict the statement made in line 115. Does the training time mentioned in line 115 mark a baseline? If yes, then please state that and put it in relation to the "substantially improved computational training efficiency" lateron. In figure 6 (and from the associated text) it is not clear which 10 of the 18 subnetworks are trained. Is it possible to mark those? Are the blue networks the connection index tensors? A few more detailing remarks would help to understand the figure better. Table 1 does not include a SOTA baseline using float weights. Even if that is a bit out-of-scope, it would help to put accuracies vs. number of bits (e.g. 32 (for FP32) * N parameters) in perspective. Table 2 lists results for a Xilinx XC7Z020. If not mistaken, the upper limit for the number of configurable gates is ~1.3M. How does the execution of models M/L/G that all exceed that number by far actually work? Same for the MNIST model L in Table 3. Are any additional resources of the FPGA device being used? A breakdown would be very useful. Table 5 contains a line for "No or pooling". It is unclear why it lists the number of total layers to be 10 and not 14. Please explain. Appendix, Section A.3.3 CPU Inference Times: the CPU in use is not being mentioned. Is this a desktop/workstation CPU or the ARM-based CPU of the Xilinx XC7C020? Typos: line 441: "This means the that the..." -> "This means that the..." Appendix, line 6: "... from Figure ??..." -> please cite the correct figure, my guess is #6.

Questions

Reflecting on lines 216-226: the reviewers personal view of residual connections is similar to the authors' and can be summarized as means to preserve (mutual) information throughout the network. Although residual initializations seem pivotal to training of LGNs, they could potentially also help float-based NN trainings without the need to add residual connections. If the authors share that view it would be great to add a discussion on this to the presentation. Why is the choice of gates being limited to 2-input variations? Especially in the context of convolutions and pooling, wouldn't it make sense to also allow for N-input OR-gates with N >> 2? It would also allow for reducing depth and signal propagation delays.

Rating

9

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

Despite improved training and inference efficiency, experimental results are limited to very small classification tasks. A single bigger task would prove scalability (or not). Lines 416-417. The chosen model sizes (S,M,L,G) do not prove saturation of accuracy (except maybe for MNIST), but you even mention that they improve with increasing model depth. If there's a reason why you stop early, please mention it in the presentation.

Summary

This paper introduced a convolutional logic gate network (LGN), which works effectively on high-dimensional spatial images. Inspired by LGN and convolutional neural nets (CNNs), the authors proposed (1) Logic Gate Tree as convolutional kernels (2) Logical OR as the pooling layer, and (3) residual initialization (instead of Gaussian random init). Besides, the authors also developed an engineering strategy to speed up the training, using low-level CUDA kernels, which is well admired. The authors have shown impressive results, in terms of performance and efficiency, on CIFAR10 and MINIST datasets.

Strengths

- Several novel ideas (technical contributions) exist in this paper. I especially admire the idea of primarily using a feedforward logic gate during initialization, which prevents both loss of information and vanishing gradients. The motivation and intuition are very clear in L208-215. - The authors have demonstrated their design using insightful experiments. For example, when introducing logic OR as pooling, the authors discussed that training can implicitly prevent saturation of activations using experiments, which is very interesting. - The experimental performance is impressive. - The engineering strategy and CUDA implementation (and open-source) would benefit the community and future research a lot. - The paper is very well-written. Though I'm not an expert in this domain, it is easy to understand the storyline, technical details, related works, and intuition.

Weaknesses

- Some suggestions on Figure presentation. - Figure 1. Also consider showing the speed advantage, as today's high-performance edge devices (Nvidia Xavier, Orin, etc) can accommodate large-weight networks and the weights of other works are already relatively small. Reporting that you can run inference the CIFAR-10 image in ~0.7 $\mu s$ on an FPGA chip would be very impressive even without looking into your paper. Besides, what is "Pareto-curve" (in the caption)? I didn't see it shown in the figure. - Figure 2. Maybe change the input into some "flattened inputs" (L112) to better show that vanilla LGNs are not designed to process images. - Figure 3. Maybe re-arrange the figure and get some space for the details of your structure. Better to also show how your network process "channels", as currently it is not clear from the figure. I also suggest adding some annotations, e.g., in the figure, adding the same notations like "depth d=2", the green squares are the selected Cm Ch Ck, in the caption also mention your NN can share weights like CNN in different spatial regions. Polishing this figure can help the reader understand the processing details quicker than understanding Eqn. (3). - Some technical questions need to be better explained. - L114-115, why are vanilla LGNs "very computationally expensive to train"? Is it because they didn't implement CUDA kernels? - I noticed that in the design of the network, the authors chose a relatively small depth but large channels (2 vs 40,400,2000+). Is there any intuitive reason to do so? How many layers (depth) does vanilla LGN have? - The authors have implemented CUDA kernel, but in the speed comparison (Table 2), the results are from Xilinx FPGA (I guess only has CPU). Why didn't the authors implement experiments on GPU? Is it for fair comparison w/ others? Maybe I missed something, but on CPU, what's the advantage of implementing CUDA kernel？

Questions

See weakenss.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors could provide a paragraph discussing their potential limitation to solving more complex CV tasks involving continuous decisions. E.g., regressing boundaries of bounding boxes (Object Detection/Tracking), localization and mapping (SLAM), generative CV, etc.