Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons

Summary

This paper focus on addressing the speed-accuracy tradeoff in adaptive spiking neurons. They utilize the existence of the absolute refractory period (ARP), and the fact that a neuron can spike at most once within this period, which could reduce the complexity to O(1), and develop an algorithmic reformulation of the adaptive ALIF model. Their model is simulated in blocks of time, instead of simulating network dynamics step by step. Their model achieves ~40x speed up in inference and ~53x speed up in training without the sacrificing the accuracy. And the model is also tested on real electrophysiological recording.

Strengths

1. Motivation: The paper is well-motivated, and addressing the speed-accuracy tradeoff in the biological simulation is an important question. And using the fact that neuron can spike at most once within the absolute refractory period (ARP) to speed up the computation is novel. 2. Evaluation: Two levels of evaluations were properly designed, one is to train SNN on spiking classification tasks (N-MNIST, SHD datasets), and the other one is to fit real neural recordings. 3. Result: Their method is mathematically proved that significantly improve the speed without sacrificing the accuracy, and it's also evaluated by multiple experiments at the same time.

Weaknesses

1. Generalization: One major weakness is that the proposed algorithm using the sparse firing in ARP to speed up the computation is quite limited to one particular type pf SNN, and hard to be generalized to other variants of neural network simulation. 2. Method: The method's performance depends on the value of hyper parameters ARP and DT, and their values are manually tuned for different experiments, which is not practical for real applications, where the ARP is typically unknown and varied across neurons. The method will be more efficient if these hyperparameters could be automatically optimized and learned. 2. Evaluation: No additional baseline was evaluated and compared under the same experiments.

Questions

1. How does this method could be generalized to standard spiking neurons models (leaky integrate-and-fire model without adaptive firing threshold, Hodgkin–Huxley model, etc.)? 2. How does the speed compared to continuous neural dynamical models such as neural ODE? 3. What's the additional overhead time and memory cost to use this algorithm?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

1. No potential societal impact of the work. 2. The ARP hyperparameters is not addressed, an automatically tuning algorithm might be potentially helpful.

Summary

The submitted manuscript proposed a method to speedup simulations of adaptive leaky integrate-and-fire (ALIF) neurons on a GPU by using the fact that during the absolutely refractory period ARP, neurons don't exchange any information and thus "blocks" on dt/ARP can be parallelized on GPU. This approach is then used to both train ALIF neurons on typical machine learning tasks and to fit individual ALIF neuron parameters to experimentally measured membrane potentials/currents/spike trains from the Allen Institute, with demonstrate a speedup achieved by the algorithm. Overall, the submission is well-written and the numerical experiments seem to be executed with due diligence. Unfortunately, no code is provided, which makes the claims and simulations not reproducible. The core idea to parallelize the time of the ARP is not new and has been used by other simulators (e.g. the NEST simulator for MPI parallelization as described here: https://nest-simulator.readthedocs.io/en/v2.20.0/guides/running_simulations.html). However, I have not seen it explicitly used for GPU parallelization and training. While the novelty of the submission seems to be minor, I would still think that it can be a useful contribution to the field once the code is made available and the implementation is benchmarked against existing implementations. Once the code is made available and benchmarks are added, I am willing to improve my score and recommend the paper for publication at NeurIPS. UPDATE POSTREBUTTAL: The code is now made available. While the originality of the contribution is somewhat limited, the preliminary benchmarks suggest that it might be a useful contribtion to the spiking network community.

Strengths

* The manuscript is written clearly. * The manuscript is relevant for both the 'spiking machine learning' community (with the ML tasks) and the neuroscience community with the neuron-fitting examples. * The implementation might be helpful for the community, as this trick seems not to be widely used yet. * figure 4.2 is great

Weaknesses

* The core idea seems not to be new: As described above, the idea to parallelize the time of the ARP was used by other simulators (e.g. the NEST simulator for MPI parallelization as described here: https://nest-simulator.readthedocs.io/en/v2.20.0/guides/running_simulations.html). It would be crucial to cite this previous work and/or describe what is new/different here. * The NEST parallelization seems to scale close to linear, thus the presented manuscript might even be not state of the art. * For the fitting of neuron models, ALIF is not state of the art (see for example Pozzorini et al. for a more recent benchmark of fitting different neuron models 2015 https://doi.org/10.1371/journal.pcbi.1004275). Currently, the manuscript claims "ALIF neurons have been shown to accurately fit real neural recordings." in line 32, and cites a paper from 2008 that is not state-of-the-art. Please reference a more recent paper and mention the caveat that the ALIF is not state-of-the art for fitting neuronal dynamics. * The notation O(T) compared to (T/T_R) slightly confusing. It seems to compare quantities with different units (assuming T and T_R have units of time). Does the speedup not also depend on DT? After reading more carefully, I guess what you actually want to claim is that the computational costs scale proportional to T and that during blocks of size T_R/DT no communication between threads has to occur which can lead to a theoretical maximal speedup of min(N_threads,T_R/DT) where is the number of parallel threads of the GPU or CPU. Minor issues: fix latex of N^in

Questions

* How does the speedup depend on the number of parallel threads and on dt? * Put the value of ARF in the caption of Figure 4a. Was it 100 steps, so 10 ms?? How biologically realistic is an ART of 10ms? * * Figure 4.3 time axis labels missing! Can you put a zoomed-in version in the supplement with finer time-axis so the artifacts introduced by large ART are easier to see? * Figure 5c: does ETV stand for? Please write in the caption, it is only defined in the supplement. * figure 5d: What is ETV for Dt=0.05ms? Maybe you have even better performance (and better speedup according to the reasoning above) for smaller dt? ;) * Why in 5f is blocks better than standard? shouldn't they be the same? * 5e: What happens for shorter ARP?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

* No code is provided, therefore the numerical results are unfortunately not reproducible. Also, the code will make the submission substantially more useful for the community. If code is provided during the review, I will reconsider my score. * ALIF is not state-of-the-art (see above). * using ARP for parallelization speedup is not strictly new (see above)

Summary

The authors implement an adaptive leaky integrate-and-fire model in a parallelized way on GPUs by incorporating a fixed refractory period that enables a window of time (‘block’) to be processed in one go. This accelerates the inference and training of spiking neuron models, while retaining accuracy, as shown by the authors.

Strengths

The authors provide an impressive array of results for speedups for training and inference of spiking neural networks (SNNs) and real neural recordings. The accuracy does not seem to decrease although a refractory period is introduced. This may have great implications for future inference of SNNs. The paper on the whole is written clearly.

Weaknesses

It is unclear why the accuracy does not go down in the empirical results presented. If the input signal is not being integrated in the case of the refractory period, then that part of the signal is forever lost. This should naturally lead to a decrease in accuracy by the network. Please elaborate why this is not happening. Is this something specific to the tasks that you are performing, and how is the accuracy impacted in general? Simple simulations with a couple of neurons may help here, while varying the ARP. This may also give some intuition as to how to pick this number to minimize loss of accuracy while still obtaining speedups.

Questions

See ‘Weaknesses’.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The limitations are adequately discussed.

Summary

Spiking neural networks can be computationally expensive to implement because of their history dependence. However, spikes can't occur at any time whatsoever; instead, after spiking, there is a refractory period during which they cannot spike again. While this period is ~1-2ms, to attain precise dynamical inferences the typical time step is ~0.1ms. The authors leverage the former fact, creating a block design, each block length equaling 1 ARP, to substantially reduce the number of required sequential simulation steps while maintaining generally high performance on both ML and neural-fitting tasks.

Strengths

This appears to be an important and very simple pragmatic algorithm that can benefit the field by greatly reducing SNN simulation times. The general presentation is clear.

Weaknesses

Minor: Fig 5: b) A timescale bar is needed, as is the ARP used. Given that there is very little performance change with DT=0.1ms, ARP=8ms (e), it would be helpful to visualize this here as well d-f) Bar graphs that do not go to zero and have no broken axis can lead to confusion. One or the other should be done. I also would advocate for reversing the ordering: f, e, d. f and e are the most compelling graphs, and f (left) provides an excellent overview comparison to begin with. d, while relevant, is also not surprising and could be better left as a final, expected finding. Wording: Lines 266 and 270, "computational neuroscientists" and "biologists." I understand where the authors are coming from; however, both are of interest to computational neuroscientists (who might do theory, computational studies, and/or data analysis), while "biologists" is overly broad. I would suggest simply indicating that both are of interest to computational and experimental neuroscientists, or to change "biologists" to "computational and experimental neuroscientists". Line 290: I'm not sure this is a fair characterization of your impressive findings in Fig 5e. The larger ARPs had little performance effect in that particular figure, while larger DTs had a large effect.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Y

acknowledgement of rebuttal // final question

The detailed responses are appreciated. * Code Availability: The reviewer has not been able to access the provided code. It is recommended that the area chair be contacted to ensure the code is made available for review. * Parallelization of ARP: The differentiation from the NEST simulator is appreciated. It is essential that the NEST algorithm is cited appropriately in the manuscript. * State of the Art: Reservations remain due to the absence of benchmarking against other spiking network simulators and AD implementations. However, the benefit of the doubt is given. A more detailed implementation and benchmarking would enhance the manuscript's strength. * Neuron Model Fitting: The absence of ALIF in Pozzorini is noted. For points 5 to 11, the clarifications and additional experiments conducted have addressed the concerns. Computational Graph Differences: The rebuttal mentions potential differences in the computational graphs between methods, leading to different gradients. An elaboration on the specific differences in the computational graph would be beneficial. In conclusion, while there are a few remaining reservations, significant efforts have been made to address the majority of the concerns raised.

Response to rebuttal

Thanks for the clarification! I am happy to increase my rating based on the authors' new experiments and clarifications. Useful paper!

Reminder to AC to share code

Dear AC, reviewer 9z9h has not been able to access the provided code. Could we please request for this to be made available? Thank you.