Exploring the Limitations of Layer Synchronization in Spiking Neural Networks

Answer to question(s) relevant to the difference between synchronous and asynchronous simulation

NB. references to sections and line numbers cited in the following answers refer to the original version of the manuscript. In order to provide more coherent explanations below we tried to dissect and group together the review comments which raise similar points and that could be addressed in the same answer. We kindly ask the reviewer to suggest if/which of these answers are essential to include in the paper or supplementary material for improving the clarity and quality of our work. > The difference between synchronous and asynchronous simulation methods is not explained clearly. It is better to illustrate the difference by a figure, i.e., an example to show how the synchronous/asynchronous method simulates an SNN. > … why the existing simulation methods are not so good for them such as the difference between synchronous simluation and asynchronous on-chip inference We agree that 1 photo == 1000 words In figure 2 in the updated supplementary material we provide snapshots from 2 videos (from the internet) that visualise the fundamental difference between synchronous processing with per-layer synchronisation and asynchronous processing. If you think it is necessary for improving the quality and understanding of the paper, we can add parts of the following discussion in the paper appendix. As one can see in the provided figures, in the case of layer-synchronous execution, spikes from one layer are not propagated to the next unless processing of all neurons in the presynaptic layer is completed. Timing or order information of spikes across input synapses in each neuron is lost since all currents are integrated altogether before the membrane threshold is evaluated (only once per integration interval). The dynamics of spikes are therefore nulled and execution follows “breadth first order” (all neurons from one layer get evaluated before neurons of the next layer start their evaluation). This creates all sorts of issues which we enumerate in the answer of the followup question. By contrast, in the case of asynchronous processing without layer-synchronisation spikes arriving from any synapse at any neuron in the network are integrated immediately triggering the membrane threshold evaluation independently of any other (section 3.1.1) . In absence of layer synchronisation (barriers) they propagate further downstream before other neurons in the same layer complete their evaluation. This makes the flow dynamics of activations to follow the relative timing/order of spikes, making any part of the network potentially active at any moment in time and allowing “depth first order” of execution if the flow dynamics require it. This is the type of in-network operation that our simulation environment supports particularly when parameter `F=1`, and what neuromorphic accelerators support (analog ones, and digital ones like μBrain and Speck as cited in the paper) When it comes to digital accelerators (because they discretise processing) they often may employ small degrees of vectorization (`F>1` in our simulation environment), to batch event processing and reduce memory I/O. This means that for a small number of spikes they may collapse timing/order and align them to process them in batch. This tampers/interferes with timing/ordering of spikes at a very small scale, but since it does it is not restricted only to spikes/currents delivered to (adjacent) neurons of the same layer, it does not have the adverse effects and extent of enforcing per-layer synchronisation, so as to hamper the activation dynamics. It can merely be perceived as (annealing) noise, and therefore asynchronous processing still works. Our simulation and training framework takes these aspects into account to armourplate the model against these effects and preserve/recover the good accuracy.

Answer to first comment

NB. references to sections and line numbers cited in the following answers refer to the original version of the manuscript. Would like to thank the reviewer for the remarks and the opportunity to defend our work, make it better understandable, and improve its quality. > Does s represent whether there is a spike or the timing of the spike? If it represents the timing, why it is an integer? If it represents whether there is a spike, how to determine the arrival order of input spikes, which is critical in asynchronous simulation? In Algorithm 1, `s` is an int vector whose dimension equals the number of neurons in the network `N`, and conveys the information of which neurons in the network have fired/spiked (activation state in the network at any time point). Likewise `s_in` is analogous to `s` but for the input features (so it only has the dimension of the input layer `N_in`). The state in `s` is transitory across forward steps, and so it varies continuously. Thus the occurrence/presence of the spikes in it follow the discrete timing/order dynamics between forward steps. The concept of forward step is explained in lines 210-221, and again in 238-242 (but it is also further clarified in the answer of the follow up question raised later). To further clarify the misconception/ambiguity, in a (pure) event-based dataflow system there is **no hard requirement for explicit timestamping for asynchronous operation** in absence of explicit synchronisation points that stall the event propagation (e.g. at layers boundaries). Thus, events are propagated as they occur (based on flow dynamics). In our simulation environment (and in many neuromorphic accelerators) the **preservation of flow dynamics** is captured in the fact that **each incoming spike (current) in a neuron is immediately integrated and triggers validation of the membrane threshold** and potential firing (what we refer to as depth-first execution through the network). In practical reality digital accelerators may choose to use some degree of vectorization for batch event processing when events occur close to each other (which forces some time-alignment of these events). This architectural choice is captured in the concept of **forward groups** `F` in our algorithm. While this optimization tampers a bit with the “purity” of being single-event-driven it is there because it saves memory IO (which is important for digital platforms). Asynchrony nevertheless still works because this batching is not restricted to adjacent neurons in a layer only, and the forced time alignment/collapsing that occasionally emerges from it can be seen as (annealing) noise which the asynchronous system can learn to tolerate (far less severe than whole-layer synchronisation that blocks the flow dynamics). Another further point to keep in mind is that according to neuroscience theories (see literature cited in the citation [1] of reviewer FREL), exact timing of spikes appears to serve primarily the purpose of ordering. That is, the more critical information is captured in the ranking of spikes rather than the exact times, which justifies the emergence of rank-order and N-of-M codings as temporal/latency codes, and the attention to only the first spike in TTFS codes. So, again in this respect keeping track of timestamps is not critical so long as the enabled flow dynamics on average respect the ordering.

Answer to second and third comment

> In SelectSpikes() in Algorithm 1, the two scheduling policies seem not to consider the difference in input spike arrival times. How the asynchornization is achieved then? > Due to the above reasons, I am not sure whether Algorithm 1 reflects the real asynchronous property of SNNs. If not so, the comparisons in the experiments are not fair since layered training with async RS inference is not a practical situation. At a high level this may appear a reasonable concern, but we suspect the ambiguity arises in overlooking some of the details that provide the necessary information incrementally in various parts of the paper. Therefore below we try to bring all this information together and disambiguate the concern. If you find it essential for the understanding of the paper to have the following information in one place summarised, we can provide a dedicated section in the paper appendix. As we remarked in lines 114-116, and 172-175 **asynchronous processing is mainly about neurons acting independently of other neurons (also across layer boundaries) based on locally emerging dynamics**, which is feasible with rate coding. **So long as these dynamics are unobstructed by synchronisation barriers there is no need for explicit time-tracking**. Temporal codings which imply detailed time tracking may need explicit timestamps but this is also because of the presence of synchronisation points. Also, order codings and TTFS schemes may use timestamps to enforce strict per/across-layer ordering but they can also do without by working with queues. The overarching assumption, which is also supported by neuroscience, is that **exact timing is relevant mainly to the extent that it facilitates the relative ordering of events** (see previous answer). From this viewpoint, in our framework (sec 3.1.1) **synaptic currents arriving at a neuron are integrated immediately and each of them triggers evaluation of the membrane threshold independently of other incoming currents** (i.e. there is no integration time interval at the neuron level). This eliminates the need of keeping an explicit global clock for the spike times. The instantaneous membrane voltage of neurons then reflects the temporal dynamics (lines 176-188) of the incoming activations at the neuron level. The **dynamic scheduling allows for the order dynamics between neurons to be also reflected in the exchange of spikes without synchronization delays at layer boundaries**. In other words these mechanisms reflect relative temporal (order) dynamics among neurons across layers in the entire network. All this is accounting for asynchrony inside the network biassed only **(a)** from stimulus, and **(b)** from the way an event-based accelerator schedules event execution/processing (lines 277-287). Coming to the two scheduling policies we presented, they aim to represent these biases (lines 266-269), and integrate them in the model training (section 3.2.1). **Random Scheduling, reflects the fact that the spike propagation and neuron integration process is or can be inherently noisy (either epistemically or aleatorically), thus affecting the temporal/rank-order of spike occurrence**. It can be seen as annealing noise or dropout noise, which beyond a certain level breaks the system down but in small enough quantities (relevant to the vector pipelining of the accelerator), makes the system more robust. Note that it is applied across the entire network not layer-after-layer! **Momentum Scheduling, emphasis on the relative order dynamics in neuron evaluation as reflected in their membranes** (i.e. which neuron is likely to fire first next – relevance to TTFS and rank-order codes for the importance of the first spike(s) even though we work with rate codes). Note again this takes place across all layers, and remains unbiased by artificial layer-synchronisation barriers. The two scheduling policies tackle two different aspect of order dynamics. All this is meaningful primarily for neuromorphic and dataflow accelerators (for now). While we showcase only two scheduling policies, this is not to say these are the only possible ones. It is a topic of open exploration for the future, and co-design for dataflow AI accelerators.

Answer to fourth comment

> How the classification result is determined by the network? What do 'on output', 'on spiking done', and 'forward steps after output' in Figure 2 and Figure 3 mean? The stop conditions/criteria for inference are explained in lines 259-261 and in Table 1. `On spiking done` means that the network is left to drain of spike activity after presentation of stimulus (for a timestep), while `On output` means that spike processing terminates as soon as the output layer neurons start firing (which in absence of layer synchronisation can be very early before even all spike activity in the network has unfolded. The notion of `forward steps` and `forward pass` are explained in lines 210-221, and again in 238-242, and thereafter used in various parts of the text. In the temporal dimension, a network unrolls/unfolds across timesteps of input stimulus presentation (as standard for RNNs/SNNs and sequence models). At each timestep the network “unrolls” or unfolds spatially completely, layer-after-layer with synchronisation in-between. **In asynchronous processing the spatial unfolding is only in-parts**, and the parts that do they may follow any order, dictated by activation dynamics. To describe this partial unfolding of computations we use the term forward pass and within it forward steps, which corresponds to the number of computation steps (across the entire network) executed until a decision is made. The number of forward steps is thus typically variable, and depends on activation dynamics (spike activity integration speed and how it is propagated – i.e. the scheduling policy) as well as some accelerator characteristics (e.g. batching or vectorization primitives – `F` param in our algorithm).

part 1 of the answer

NB. references to sections and line numbers cited in the following answers refer to the original version of the manuscript. > comparative empirical results in terms of accuracy and sparsity with some previous works? For example, the state of the art directly trained SNN models. Such as, [1,2,3], I believe they belong to the class of synchronous computation of spikes. We appreciate the positive feedback and score, and thank the reviewer for raising the opportunity to better position our work in relation to results reported in the literature. We have read the suggested papers but as they involve experiments with larger datasets and far larger or more complex models we are not in position at the moment to carry out sound comparative experiments. Both because of time (complexity involved) and computational limitations (our training framework as is, reaches fast the limits of memory resources on our currently two GPUs for simulation – this is why we were limited with the VGG11 experiments in the first place). The suggested papers (as well as [1] from the first reviewer) help us pin-down specific directions for follow-up research however when it comes to scaling up and tackling more complex models. Specifically, making asynchronous processing viable in presence of batch-norm, auto-regression with attention, and gating mechanisms are all very interesting followup project work that cannot be concluded in the short time of the rebuttal. In parallel, addressing the computational cost of unlayered backpropagation for accommodating much deeper models is another direction we have started working on. We will cite and remark to these aspects in the discussion of a revised version for the manuscript that we are preparing until the end of the rebuttal period. Meanwhile, what we tried to do, in response to the review request, was to identify and report other in-literature results on the same datasets where comparable sized models (num of parameters), without asynchronous processing were used. The goal being to provide a reference for evaluating how sensible our scores are. These results are summarised in the table below for accuracy. | | NMNIST | SHD | DVS gesture | |:---:|:---:|:---:|:---:| | Bouanane et al. 2023 | 0.976 | 0.772 | | | Liu et al. 2023 | | 0.793 | | | He et al. 2020 | 0.983 | | 0.868 | | Ours synchronous | 0.949 | 0.783 | 0.739 | | Ours asynchronous (MS) | 0.963 | 0.816 | 0.856 | We note that while the depth of the models represented may differ, the number of parameters (model capacity) is comparable. Overall, we think that our reported results are comparable and within the range of what is reported in the literature, for the choice of models and baselines we used to demonstrate the effects of asynchronous processing (we opted always for model than 2 hidden layer deep models, since asynchrony typically unfolds after 2 hidden layers of depth). Regarding spike sparsity Bouanane et al 2023, and Liu et al 2023 report sparsity metric but it is not clear how they have normalized them (per timestep, per inference, or per the number of data points in the testset). So a tabular comparison is not meaningful/possible but the papers give an indicative impression we hope for comparison across the datasets. M. S. Bouanane, D. Cherifi, E. Chicca, L. Khacef. Impact of spiking neurons leakages and network recurrences on event-based spatio-temporal pattern recognition. https://doi.org/10.3389/fnins.2023.1244675 S. Liu, V. C. H. Leung, P. L. Dragotti. First-spike coding promotes accurate and efficient spiking neural networks for discrete events with rich temporal structures. https://doi.org/10.3389/fnins.2023.1266003 W. He, Y.J. Wu, L. Deng, G. Li, H. Wang, Y. Tian, W. Ding, W. Wang, Y. Xie. Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences. https://doi.org/10.1016/j.neunet.2020.08.001

Thanks for your response

I want to thank the authors for their response, I will keep the score.

Answer to first comment

NB. references to sections and line numbers cited in this and the following answers refer to the original version of the manuscript. Would like to thank the reviewer for the remarks and the opportunity to defend our work, in order to make it better understandable. > the accuracy of the Unlayered method (Table 2), and it generally falls below that of the traditional Layered method. What is the network architecture of the Layered methods compared in Table 2? The models in these comparisons have exactly the same topology/structure and capacity (number or parameters) as the baseline, and were trained with the same loss (cross-entropy). This makes the comparison in our eyes as fair as it can be, and devoid of other factors influencing the results (e.g. different topology, different depth, more complex neuron model, different network block-structure) apart from the asynchronous training. Our goal is to isolate, and amass information about the phenomenon and effects of asynchronous processing, and how to accommodate it in training. The details of the topology and training configuration per each dataset is reported in A.6.1 (for Table 2) and A.10 for the deeper VGG in section 4.6. The choice of the topology, as we remark in section 4.1, is such that each model has more than 2 hidden layers, because with up to 2 hidden layers, asynchrony barely unfolds even without any layer synchronization. The adverse effects (of synchronous training with asynchronous inference) become more and more pronounced as the depth increases. If one compares the reported accuracy to the SoA accuracy for these datasets, we acknowledge that higher scores have been reported in the literature. For example for SHD, higher accuracies have been reported in literature by using models with complex multi-compartment neurons, synaptic delays, batch-norm, and other structures (ultra-high parameter capacity models), but we do not see what purpose it would serve to compare against such models. Apart from being complex to implement in our simulation environment, it would necessitate a thorough ablation study afterward to isolate the part of the accuracy that is merited to “each trick” versus the merits of asynchronous execution. It is important to emphasise that the primary message we aim to convey is not solely about achieving high accuracy, but rather demonstrating competitive or recovered accuracy alongside significant improvements in energy and latency efficiency. This aligns with the premises of neuroscience in spiking neural networks (SNNs), where asynchronous processing offers unique advantages in these aspects.