SparseProp: Efficient Event-Based Simulation and Training of Sparse Recurrent Spiking Neural Networks
Spiking Neural Networks (SNNs) are biologically-inspired models that are capable of processing information in streams of action potentials. However, simulating and training SNNs is computationally expensive due to the need to solve large systems of coupled differential equations. In this paper, we introduce SparseProp, a novel event-based algorithm for simulating and training sparse SNNs. Our algorithm reduces the computational cost of both the forward and backward pass operations from O(N) to O(log(N)) per network spike, thereby enabling numerically exact simulations of large spiking networks and their efficient training using backpropagation through time. By leveraging the sparsity of the network, SparseProp eliminates the need to iterate through all neurons at each spike, employing efficient state updates instead. We demonstrate the efficacy of SparseProp across several classical integrate-and-fire neuron models, including a simulation of a sparse SNN with one million LIF neurons. This results in a speed-up exceeding four orders of magnitude relative to previous event-based implementations. Our work provides an efficient and exact solution for training large-scale spiking neural networks and opens up new possibilities for building more sophisticated brain-inspired models.
Paper
Similar papers
Peer review
Summary
This paper introduces an efficient Spike-based recurrent network learning simulation method based on an event-driven implementation. The method is based on two main contributions. The first consists in a change of temporal reference frame that uses the phase of a neuron rather than its absolute time, the second contribution consists in finding an efficient data representation corresponding to the parsimonious structure of neural connectivity. This method is applied to different neuron models, from a linear LIF model to more complex models such as the quadratic LIF neuron.
Strengths
This paper clearly presents an original method for simulating and training a spiking neural network. The motivation for the paper is very well set out, and the methods are clearly explained both in terms of the definition of the mathematical equations and the path that leads to its numerical implementation. The results look very promising, and the preliminary results set out in the appendix are very promising indeed. In particular, the experimental results show that this method can be used to implement very large networks on computational architectures of reasonable size, which is a game changer for the SNN community. Minor: caption of Fig 4 : repetition of « higher densities »
Weaknesses
Although Figure 3 already presents a comparison between the proposed method and a conventional method, it would have been desirable to have a comparison between these two methods in terms of the quality of the obtained results. It would have also been beneficial to isolate the improvements achieved by the two contributions independently (the first contribution being the change in temporal reference, and the second being the sparse representation of data). Some improvements in the presentation can be made... For example, the "peek" function is not defined.
Questions
I am curious to know if the change in representation from absolute time to phase affects the precise latency of neuron firing? Additionally, it appears that biological neural networks include transmission delays and spikes between neurons. Is your implementation suitable for this scenario?
Rating
9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Since the computational results seems so promising, a limit of this paper is that it is not applied to standard benchmarks and notably to datasets which include a rich dynamics.
Summary
Simulation of RSNN (recurrent spiking neural network) is a difficult task, whose limits prevent large scale (say 1e.9 neurons) simulations. Often, in practice, one neuron is connected to only a few others, the sparsity. The present paper leverages this sparsity to upscale the simulations limits of RSNN. Sparsity is often leveraged to increase performance and scalibity but SparseProp (algo. presented in this paper) seems to be a newcomer for event-based simulations of RSNN.
Strengths
-The authors present a new and efficient algorithm for event-based simulations of RSNN. In its structure, the algorithm is fairly similar to classical ones, while taking advantage of the sparsity structure, it presents a very good ratio (high) efficiency / complexity. Practically, it mainly forces the practitionner to have a better data structure. -The exposition and background are in general clear and the algorithm seems natural to derive in the history-flaw of the paper (great!). Algorithms for non-sparse and sparse simulations are cleary written and can be easily compared by the reader. -Simulations are given, for SparseProp and classical algo, giving a good sense on how performant their method is (performant!). -The flow of the paper Abstract / General -> Particular / Practical, is smooth. -Examples on classical RSNN are given, which are good toy examples for the reader to play with. -Simulations are available in Julia.
Weaknesses
-General comment on citations: the authors could give more references, for example: -line 59: several classical papers showing the origin of the differential equation, would be appreciated. -figure 1: a reference paper, explaining the role and subtilities of the phase representation, would be appreciated (especially since its of crucial importance for this paper). -The claim of efficient simulation is for general univariate RSNN but the paper nearly entirely focuses on homogeneous ones. I understand that this can be generalized, as explained in Section 6, but no numerical results (as in Figure 3) are given and the algorithm is only explained (Section 2.1) for homogeneous networks. For pedagological purpose, its great, but it leaves with a taste of unfinished. It feels like the most interesting and difficult part is "left for the reader". I would suggest claiming a weaker result or giving more material for the understanding of heterogeneous networks (eventually in the appendix). This is the main reason why Soundness grade is only at 2. -Since the pulse-coupled phase oscillators (and constant phase velocity) is an important and simplifying assumption, the paper lacks a discussion on it or a reference that discuss it (is it a strong assumptions: yes, no, why - how is it linked to Eq. 1...). -Section 7 is simulation-less so the reader can not know if the proposed algorithm is efficient. Moreover it is mentionned "training of RSNN" in the title, which seems a bit of an overstatement since there is only a 20 line, simulation-less, section about it. I know that this is because the authors of EventProp did not make their algo available but this also should not be a problem for the reader. I would suggest removing the "training" in the title and keeping this section either in an appendix or as a "bonus" section.
Questions
-What is the interrogation point, line 190 ? Probably a reference to correct -Which type of network is used in Figure3 ? Is the scaling the same for heterogenous networks ?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
-limitations raised by the the weaknesses / questions stated earlier. -limitations raised by the authors themselves in section 8
Summary
This paper presents an efficient event-based algorithm named *SparseProp* for both inference and learning of sparse SNNs. It gets rid of the discretized simulation of ODEs in neuronal dynamics and utilizes phase representation for those neuron models with closed-form solutions between spike times. A priority queue is employed for recording and evolving the states of related neurons when a spike is triggered, which reduces the amortized time complexity to O(K log N) per spike.
Strengths
This work manages to perform a lazy update of neuron states. Phase representation rather than membrane potential is a brilliant idea since all neurons now share an identical magnitude of update in internal states, which makes it feasible to update $\phi^{reset},\phi^{th}$ alone as an alternative to the O(N) polling all neurons' potentials. The priority queue ensures the incoming spike always happens in the neuron at the top of the heap. The experimental results also show significantly improved simulation efficiency. Although there exists earlier work [1] applying priority queue to the simulation of event-based SNNs, the algorithm presented here for sparse SNNs is definitely not trivial. [1] Eduardo Ros, et al. "Event-driven simulation scheme for spiking neural networks using lookup tables to characterize neuronal dynamics." Neural computation, 2006
Weaknesses
There are still some major concerns to be resolved 1. The whole study seems to assume a positive external input, i.e. $I^{ext}>0$ in Line 155/544 and a special setting $V_{th}=0, V_{reset}=-1$ in Line 134. In such a case, we expect a spontaneous spike dynamic for those integrate-and-fire models. We have a well-defined unperturbed interspike interval $T^{free}$ and can deduce the phase $\phi$ accordingly. However, a popular setting in many recent SNN studies, such as ref.6 and ref.17, is $V_{th}>0, V_{reset}=0$, and $I^{ext}=0$. Under this assumption, a neuron won't fire if the incoming excitatory spike is not strong enough. There will be no spontaneous spike and thus no unperturbed interspike interval $T^{free}$ or valid phase representation, and the whole thing seems to fall back to polling of neuron states. In fact, any $I^{ext}\in[V_{reset}, V_{th}]$ lead to an invalid $T^{free}$ for the LIF model as Eq.7 shows. All in all, I don't think we can assume those parameters without losing generality as Lines 133-134 reads. 2. For an excitatory-dominated network, it might be possible that two neurons become postsynaptic neurons of each other. *SparseProp* may find that any firing of one will trigger a spike of the other, and it implies an endless loop of updating the priority queue. 3. The training algorithm coupled with *SparseProp* is more like a draft since no experimental results are really attached here. There might be many details to be considered when designing a backpropagation algorithm for *SparseProp*.
Questions
1. *SparseProp* seems only works for some specific cases, such as the spontaneous firing neuron model. Is it possible to generalize *SparseProp* to a broader variety of SNNs? 2. Does every operation in *SparseProp* have a well-defined gradient for backpropagation? - The reset of phase for the firing neurons, the phase changes of postsynaptic neurons, and the reset of the global phase (Lines 100-101), to name a few.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
1 poor
Presentation
3 good
Contribution
3 good
Limitations
See the weaknesses part
Summary
This paper proposes SparseProp, an event-based algorithm for efficient simulation and training of sparse Spiking Neural Networks (SNNs). By leveraging the sparsity of the network, SparseProp reduces the computational costs of operations from O(N) to O(log(N)) per network spike. This method avoids iterating through all neurons with each spike and adopts efficient state updates. Demonstrations include a simulation of a sparse SNN with one million LIF neurons, achieving speeds surpassing prior models by over four orders of magnitude. This innovation offers the potential for more sophisticated brain-inspired models.
Strengths
1. The proposed method decreases the computational cost of conventional event-based SNN simulation from O(n) to O(log(n)) per network spike. The cost decrease mainly comes from using efficient data structures for spike searching and change of reference frame for state evolution. 2. The authors demonstrate the potential to use the event-based simulator for training SNN. This can be a handy tool for SNN training combined with the EventDrop algorithm. It could decrease the training cost of the SNN when number of events is low. However, the authors performed no experiments on this matter, making the contribution vague.
Weaknesses
The main concern of the reviewer is the usefulness of the proposed approach in real-world applications. Thus, the proposed method may better serve as a neuroscience tool than be used in real applications, as advertised by the authors in the introduction. Here is a list of weaknesses of the paper the reviewer summarized: 1. The paper lacks experiments with SNN algorithms for real-world tasks. The experiments done in the paper concentrate on analytical modeling and non-realistic neural network architecture. However, SOTA SNN algorithms have different architectures and may generate different results. The authors need to perform experiments on real tasks to make a convincing statement on the applicability of real applications. 2. Event-based computation typically suffers from parallelization problems and scalability problems. The problem becomes more apparent when the number of spikes increases in the SNN compared to sparse networks. On the other hand, GPUs and typical deep-learning accelerators use vector computation which is very easy to parallel and can generate much lower latency compared with the event-based approach. The authors must discuss and compare the two approaches using actual workload and show under what conditions the event-based approach may have an advantage. 3. It is hard to say that the proposed method will have an advantage compared with the significant amount of SOTA SNN algorithms that use the Euler method to simulate the activity of spiking neurons. The step-based simulation method demonstrates its effectiveness in many complex tasks and efficiency in many neuromorphic hardware. Therefore, the significance of the proposed approach can be limited.
Questions
1. Can the proposed simulator scale in a paralleled system? Moreover, how well is the scalability? 2. Have the authors considered using the simulator for layered network architecture? What will the difference be compared with the paper's recurrent architecture? 3. What will be the performance comparison regarding algorithm accuracy and computation cost when comparing an SNN using the event-based simulator and an SNN using a step-based approach running in parallel computing frameworks?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The authors have listed the limitations in the paper. The reviewer agrees with the author that the major limitation of the paper is the need for actual workload and comparison.
Thanks for the authors' further clarification. I want to figure out the complexity when the external input is nonpositive. For the QIF model, the authors mentioned that it is possible to build a priority queue for those neurons with $V(t)>\sqrt{I_{ext}}$. However, for those neurons with $V(t)\leq\sqrt{I_{ext}}$, how will you maintain their state when a spike is triggered? You may need to assign a time variable recording the elapsed time until the last update of potential rather than simply updating the global phase, which brings in extra memory consumption (O(N) in the worst case). Worse still, in some cases, no neuron has a higher membrane potential than $\sqrt{I_{ext}}$. How will SparseProp act in this situation? Another concern is that when $I_{ext}$ is negative or zero, it seems impossible to generate any phase representation for LIF neurons. Considering that the LIF model is still the most commonly used one in SNNs, I'm afraid SparseProp could not bring a broader impact to the SNN community.
response to comment: memory consumption // QIF and LIF in the excitable regime // SparseProp for LIF neurons without phase representation
We sincerely appreciate the thoughtful comments and queries from the reviewer, which have significantly aided in enhancing the clarity and robustness of this submission. Please find our clarifications below: 1. **Regarding Memory Consumption in the QIF Model with Nonpositive Input**: - It's accurate that additional memory is required for positive input currents. However, the increase is minimal, necessitating only one more array of size $N$. As the memory cost is anyway $O(N)$, this does not qualitatively change our results. We added a row to our computational cost table, to make the memory cost of SparseProp more transparent. Note that we assumed that the adjacency matrix is generated on the fly, otherwise the memory cost will anyway be dominated by the sparse adjacency matrix which requires memory of order $O(K N)$. It's worth noting that the computational complexity remains $O(K \log(N))$ per network spike. - To provide a practical perspective, we've included implementations for both QIF and LIF networks in an excitable regime (so each neuron receives inhibitory input) but is driven by excitatory Poisson input, in our anonymous code repository. 2. **On Neurons with Membrane Potential below $\(\sqrt{I_{ext}}\)$**: - Indeed, in scenarios where no QIF neuron surpasses the unstable fixed point, the network remains silent forever. We note that this is **not** a shortcoming of SparseProp, but expected behavior for any correcly implemented spiking network simulator. - One way to avoid that situation is to drive the network with excitatory input Poisson spikes as demonstrated in the code. 3. **Concerning Phase Representation in LIF Neurons**: - We'd like to clarify that LIF networks, despite the absence of a phase representation, remain perfectly compatible with SparseProp. If our previous explanations did not capture this clearly, we apologize for the oversight. In this case, there is simply a binary heap of the next spike times of the input spike trains. Whenever a recurrent LIF neuron is pushed beyond threshold, it's postsynaptic neurons are updated. - In the code repository, we now also provide a SparseProp implementation of a recurrent LIF network with external excitatory Poission input. We would thus politely like to disagree that SparseProp does not apply to to LIF networks. We pointed out several limitation of SparseProp in the limitations paragraph, however, negative external input is no such limitation. - We genuinely value the critique, which prompted a deeper reflection on our work. While we acknowledged various limitations of SparseProp in the limitation section, we believe that negative external input isn't one of them. Hence, we remain confident in the versatility of SparseProp across different neural network types, including LIF networks. In conclusion, we're grateful for this ongoing dialogue, which we believe is greatly enhancing the quality and clarity of our manuscript. We're optimistic that the improvements made, as a result of these discussions, will resonate well with the broader SNN community. ### Comparison of computational cost for event-based simulation of recurrent spiking networks for different algorithms | | Conventional Algorithm | SparseProp | |-------------------------------------------|--------------------------------------------|--------------------------------------------------| | Find next spiking neuron | $\min_i(\phi_{\textrm{th}}-\phi_i/\omega)$ | peek($\(\phi_i\)$) | | Evolve neurons | \$\phi_i \mathrel{+}= \omega \cdot dt $ | $\(\Delta \mathrel{+}= \Delta\phi\)$ | | Update postsynaptic neurons | $K$ operations | $K$ operations + $K$ key updates $\mathcal{O}( \log(N)$| | Reset spiking neuron | One array operation | $\mathcal{O}( \log(N)$ | | Memory cost | $\mathcal{O}(N)$ | $\mathcal{O}(N)$ | | **Total amortized costs per network spike** | $\mathcal{O}(N +K)$ | $\mathcal{O}(K\,\log(N)\)$ | | **Total amortized costs for fixed simulation time** | $\mathcal{O}(N^2)$ | $\mathcal{O}(K\, N\cdot \log(N)$ |
Thank the authors for the further clarification. I think there might be some misunderstanding between each other. I should apologize for not making my point clear earlier in the discussion. So I want to talk about my interpretation of SparseProp and another trivial but faster (only in some cases) simulation method. Correct me if I'm wrong. One major component of SparseProp is a priority queue storing the states of neurons. In phase representation, spontaneous firing setting ($V_{th}=0, V_{reset}<0, I_{ext}>0$) and inhibitory input weight (negative weight) will lead to an occasion: a neuron will fire because there's been a period of time since it receives the latest negative input (because weights are negative), the positive and **stationary** external input drive potential to above the threshold zero and therefore trigger spikes. In this case, the possibility of firing lies in the neurons without any input. It's expensive to update the state of these silent neurons every moment since spikes trigger at most O(K) neurons and $K\ll N$, which hints the number of silent neurons is still O(N). The priority queue will precisely find those silent neurons with the highest phase (most close to the firing state) and adjust the heap in O(log N) time complexity, and therefore reduce the time complexity from O(N) to O(log N) per spike. However, if we look back to the most common setting in SNN studies (ref.6 and ref.17), i.e., $V_{th}>0, V_{reset}=0, I_{ext}=0$, and LIF models, not completely inhibitory network (all/part of the synaptic weights are positive, since some studies don't assume Dale's law and synaptic weights from the same neuron might include both excitatory and inhibitory weights) and some intermittent input spikes (Poisson encoded data or whatever kinds of spike inputs), the possibility of firing is different here. A neuron will fire only because it receives some spikes from several excitatory synapses (positive input). A neuron with very low membrane potential could fire because it receives so many positive inputs. In this case, the maximum number of firing neurons that could be triggered by a single spike is naturally O(K) rather than O(N). In short, **only spikes trigger spikes** in this setting. Without any spike inputs, the neuron is bound to be silent since the exponential decay will gradually drag the potential to $V_{reset}=0$. In the meantime, the most possible firing neurons have little to do with their rank of current membrane potential $V(t)$ but depend on the sum of current potential and the overall input spikes. Since the rank of state (phase, membrane potential, or any other representation) is no longer critical here, we could simply get rid of the priority queue and compute at most O(K) times for each spike. In fact, I don't quite understand why a priority queue is still needed for this case since the possible firing neuron needs not have the highest potential before charging. > In this case, there is simply a binary heap of the next spike times of the input spike trains. Whenever a recurrent LIF neuron is pushed beyond threshold, it's postsynaptic neurons are updated. I don't see there is a predictable 'next spike times' based on neuron states alone in this case, since a large enough input from excitatory synapses could bring the potential of any neuron above the threshold. Without input spikes, the silent neurons will keep silent and always have an infinite next spike time. From the above analysis, for the most common setting in deep SNNs ('spikes trigger spikes' case), we already have an O(K) time complexity per spike simulation method, which is also for sure event-driven, and the SparseProp won't make the simulation faster (O(K log N) per spike) under this hyperparameter setting. Now a problem is raised: could authors name some other occasions that **the above-mentioned O(K) simulation method cannot be applied but SparseProp can**? From the main text, we know that the $V_{th}=0, V_{reset}<0, I_{ext}>0$ is an applicative condition since the firing neurons are those O(N) neurons receiving no spikes. This is also my main concern because I believe this O(K) simulation method is rather trivial to me, but still more efficient than SparseProp. I think this might help the authors to restrict the range for application of SparseProp, rather than claiming it as a universal efficient simulation technique.
Concerns addressed?
We would like to ensure that the concerns raised in the review have been adequately addressed in the rebuttal as the author-reviewer discussion period draws to a close. To summarize the key points from the rebuttal: 1. We clarified the potential for parallelization of the SparseProp method, emphasizing the distinction between parallel speedup and the reduction in computational cost. 2. Simulations for a layered recurrent network architecture have been added, supplementing the focus on spiking recurrent networks. 3. A benchmark against NEST, a well-regarded spiking network simulator, is underway and will be included in the camera-ready version of the manuscript. We acknowledge and are grateful for the reviewer's suggestion to evaluate SparseProp in real-world scenarios. The authors also appreciate the reviewer's recognition of the innovative approach to spiking neural network simulation and training. If there remain any unresolved issues or further clarifications are needed, the authors are keen to address them. The insights provided in the review have been instrumental in refining the work.
Decision
Accept (poster)