Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies

Summary

This paper introduces the parallel spiking neuron (PSN) and several variants. The primary benefit of PSN over the existing spiking neurons is the parallel implementation on digital hardware, which brings dozens of times of acceleration on GPU. The accuracy also demonstrates the effectiveness of this paper.

Strengths

1. The idea of parallelized neuron implementation is interesting and can be efficiently processed on GPU. 2. The experiments show good performances.

Weaknesses

The authors pinpoint a disadvantage of current SNN, that is, low running speed on GPUs. However, it is known that GPU is not the most ideal device for deploying SNN, rather, it should be the neuromorphic hardware. PSN focuses on the optimization of SNN on GPU, however, GPU cannot utilize the binary spikes to lower the energy. So, even if PSN can accelerate the inference on GPU, compared to ANN on GPU, there is no advantage of efficiency still compared to ANNs. I was wondering whether the optimization on GPU is really useful cause people can always use ANNs on GPU.

Questions

Can authors show the ablation study of PSN and vanilla SNN on static image datasets?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Listed above.

Summary

The paper presents an approach to improve the efficiency and accuracy of Spiking Neural Networks by using a dependency method to generate hidden states, resulting in parallelizable neuronal dynamics and a significant increase in simulation speed.

Strengths

The authors analyze the impact of removing the reset function from standard charge-fire-reset neuronal dynamics, proving that the parallelization of the spiking neuron can be achieved without it. They also present a general formula by rewriting the neuronal dynamics without a reset and introduce the PSN, a spiking neuron with entirely parallelizable neuronal dynamics. The paper also assesses the PS family's performance on sequential, static, and neuromorphic data classification tasks, showing that they attain higher accuracy than traditional spiking neurons.

Weaknesses

Please see Questions.

Questions

1. Since the proposed models use parallel input of X[t-1], X[t], X[t+1], I am wondering how memory-intensive the models will be for parallel processing, and especially how it changes for the different datasets. Basically, it would be interesting to see a comparison of FLOPS/MACs for the standard vanilla spiking neurons and the proposed parallel spiking neurons 2. The VGG shows better performance for ImageNet and CIFAR-DVS. The authors show the performance of the VGG model is currently state of the art using vanilla neurons for the ImageNet dataset. It would be great if the authors could report the results of the VGG model with parallel neurons for the ImageNet dataset. I understand the tight time constraint and hope this won't be very difficult, especially because the authors have already shown results for VGG on the CIFAR-10 DVS 3. I am also curious how stable are these parallel spiking neurons compared to the vanilla neurons since the reset is removed. In the presence of input/weight noise, will it still perform as well? 4. When the authors evaluate the long-term dependencies, it would be interesting to see results on non-image datasets in Supplementary. Mainly because there is a high degree of correlation among the pixels for such image datasets, and I was just wondering how well it would work for other non-image-based datasets.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

It would be great if the authors could include what are the key limitations of this work and what are the trade-offs with the current vanilla spiking neurons.

Summary

This paper proposes the Parallel Spiking Neuron (PSN), which generates hidden states that are independent of their predecessors, resulting in parallelizable neuronal dynamics and extremely high simulation speed. The weights of inputs in the PSN are fully connected, which maximizes the utilization of temporal information. The authors evaluate the PSN family on simulation speed and temporal/static data classification, and the results show the overwhelming advantage of the PSN family in efficiency and accuracy.

Strengths

1. The motivation considering the parallel spiking neurons for high simulation speed is interesting and important for the futural application of spiking neural networks. 2. The experiments on the large datasets, such as ImageNet, improves the techinical soundness of the spiking neural networks.

Weaknesses

1. Since the main goal of the proposed parallel spiking neuron model is to improve the simulation speed, the current experiments do not reflect that advantage over other methods, in which only focusing on the accuracy and the firing rates analysis. 2. The network architecture of the spiking networks in Table 2 is not clear.

Questions

1. Please explain the advantage of the parallel spiking neurons as mentioned "high simulation speed " in abstract. 2. How about the whole training and test time of the proposed SNN compared with other methods? 3. Is there any limitation once applied the parallel spiking neurons into the neuromorphic hardware？How it could map to the hardware? 4.What is the network architecture used in Table2, such as the "VGG"? Please describe the detailed network architecture in the Table 2.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

1. Since the authors claim that the main goal of the proposed parallel spiking neuron model is to improve the simulation speed, the current experiments do not reflect that advantage over other methods, in which only focusing on the accuracy and the firing rates analysis. 2. The network architecture of the spiking networks in Table 2 is not clear. The above limitations have been explained clearly in the following response.

Summary

This paper removes the reset mechanism from the dynamics of conventional LIF/IF neurons, and proposes to reformulate the neuronal dynamics using matrix multiplication instead of the iterative updating for the membrane potential. This matrix multiplication then can be simulated in parallel to accelerate the training of deep SNNs. This paper makes strong assumption that hidden states are independent of their predecessors, and proposes the Parallel Spiking Neuron (PSN) and its variants, masked PSN, sliding PSN. The whole framework is built on unfolding the computing graph over the latency $T$ and then uses a fully connected layer $H = W X$ to replace the neuron dynamics.

Strengths

1. Parallelization: Parallelizing in spiking neurons is an interesting topic in SNNs. By removing the reset mechanism, the neuronal dynamics can be reformulated in a non-iterative form. The proposed Parallel Spiking Neuron (PSN) framework allows for parallelized neuronal dynamics, enabling efficient computations across multiple processing units or threads. 2. Utilization of Temporal Information: The PSN utilizes fully connected weights for inputs, maximizing the utilization of temporal information and potentially enhancing the model's ability to capture temporal patterns. 3. High Simulation Speed: The PSN framework, with its independent hidden states and parallelizable dynamics, achieves extremely high simulation speed, which can be advantageous for real-time applications and large-scale simulations.

Weaknesses

1. Lack of Reset Mechanism: The removal of the reset mechanism in the PSN may limit its ability to handle certain types of dynamics or tasks that rely on precise timing and reset behavior. The authors did not provide reasonable explanation of why neuronal resetting can be ignored, what should be done to compensate for reset removal. 2. Large number of trainable parameters introduced by the new Weights: The use of matrix multiplication in the PSN introduces additional trainable parameters in the new weights, the performance improvement could benefit from using more trainable parameters, but not the new PSN model. With the same number of parameters for both PSN and LIF, will the performance still perform good using the new PSN compared to LIF? 3. Using future information: when calculating $H[t] = \sum_{i=1}^T W_{t,i} X[i]$, in this PSN the future information is used. For the sliding PSN and masked PSN, the authors tried to avoid using future information. So that in order avoiding using the future information, it's better to use masked PSN and sliding PSN, but in the experiments, it's not clear which PSN version is used. 4. This paper makes strong assumption that hidden states are independent of their predecessors, the Parallel Spiking Neuron (PSN) is proposed based on this. How to describe the neuron dynamics along the time-dimension if we remove this hidden state dependency?

Questions

As above in Weakness. The whole PSN framework is built on the condition $u(t) < V_{th}$, how to get spikes if this is the precondition for PSN? The lack of dependency between successive time-steps is a concern for me. The whole framework is built on unfolding the computing graph over the latency $T$ and then uses a fully connected layer $H = W X$ to replace the neuron dynamics. It doesn't make sense. Can you give more detailed explanation on this?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. The whole PSN framework is built on the condition $u(t) < V_{th}$, not clear how to get spikes if this is the precondition for PSN. 2. Using future information: when calculating $H[t] = \sum_{i=1}^T W_{t,i} X[i]$, in this PSN the future information is used. For the sliding PSN and masked PSN, the authors tried to avoid using future information. So that in order avoiding using the future information, it's better to use masked PSN and sliding PSN, but in the experiments, it's not clear which PSN version is used. 3. This paper makes strong assumption that hidden states are independent of their predecessors, the Parallel Spiking Neuron (PSN) is proposed based on this. How to describe the neuron dynamics along the time-dimension if we remove this hidden state dependency?

Fix math errors

An additional line break causes the math errors of $c[i][j]$. It should be corrected as $$ c[i][j] = \prod_{l=i+1}^{j}G[l] = \begin{cases} \frac{p[i]}{p[j]}, j \geq i \\\\ 0, \mathrm{otherwise} \end{cases},\\ $$

Response to authors

I would like to thank the authors for providing thorough responses and for their efforts to enhance the paper based on my feedback. The clear and accurate explanations you provided exceeded my expectations, and I appreciate your dedication in addressing my questions.

Reply to the Author's Response

I thank the authors for their detailed response and for clarifying my doubts.