Evolving Connectivity for Recurrent Spiking Neural Networks

Recurrent spiking neural networks (RSNNs) hold great potential for advancing artificial general intelligence, as they draw inspiration from the biological nervous system and show promise in modeling complex dynamics. However, the widely-used surrogate gradient-based training methods for RSNNs are inherently inaccurate and unfriendly to neuromorphic hardware. To address these limitations, we propose the evolving connectivity (EC) framework, an inference-only method for training RSNNs. The EC framework reformulates weight-tuning as a search into parameterized connection probability distributions, and employs Natural Evolution Strategies (NES) for optimizing these distributions. Our EC framework circumvents the need for gradients and features hardware-friendly characteristics, including sparse boolean connections and high scalability. We evaluate EC on a series of standard robotic locomotion tasks, where it achieves comparable performance with deep neural networks and outperforms gradient-trained RSNNs, even solving the complex 17-DoF humanoid task. Additionally, the EC framework demonstrates a two to three fold speedup in efficiency compared to directly evolving parameters. By providing a performant and hardware-friendly alternative, the EC framework lays the groundwork for further energy-efficient applications of RSNNs and advances the development of neuromorphic devices.

Paper

References (51)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer mDuJ6/10 · confidence 4/52023-06-30

Summary

This study presents an application of a previously developed approach, NES, to train RSNNs formulated based on connection probabilities. However, there are concerns regarding the ethical aspect of this work, specifically the reproducibility and proper citation of related works (see Weaknesses). Due to these reasons, it is challenging to recommend this manuscript for acceptance in its current form.

Strengths

n/a

Weaknesses

o The lack of code availability at the time of review poses a significant barrier to verifying the claims made by the authors. o Additionally, it appears that an existing work by Stockl et al., 2021 from Maass group, which applies NES to learn RSNNs formulated in terms of probability skeleton inspired from Billeh et al., is not cited in this manuscript. This omission warrants clarification as it may influence the perceived contribution of this work. o Beyond the ethical issues outlined above, this paper has several limitations that need to be addressed. Firstly, the results presented are purely empirical and there is a lack of theoretical contributions. Secondly, the manuscript lacks an analysis of how varying the sample size impacts the results, which leaves unanswered questions regarding the robustness and efficiency of the proposed methodology. Lastly, it is unclear whether the authors have performed a thorough tuning of the hyperparameters, which could significantly impact the performance of the benchmarks.

Questions

n/a

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

See weaknesses

Reviewer SW366/10 · confidence 3/52023-07-05

Summary

Facing on the inaccurate and unfriendly limitation of current surrogate gradient-based learning methods for recurrent spiking neural networks (RSNN), this study develops the evolving connectivity (EC) framework for inference-only training. The EC framework reformulates weight-tuning as a search into parameterized connection probability distributions, and employs Natural Evolution Strategies (NES) for optimizing these distributions. The performance of the proposed EC is evaluated on a series of standard robotic locomotion tasks, where it achieves comparable performance with deep neural networks and outperforms gradient-trained RSNNs.

Strengths

1. The motivation is reasonable and the application scene is interesting. 2. The framework of EC considering the weight reparameterization and connection evolution is reasonable.

Weaknesses

1. It seems that the weight-based parameterization method and the NES method used in EC is not novel, it seems to use the method proposed in other papers. 2. The energy consumption should be computed to show the efficiency of the proposed framework. 3. The experiments seems not enough to verify the effectiveness of the proposed model.

Questions

1. The novelty of the weight-based parameterization method and the NES methods in EC framework should be explained further. 2. The energy consumption should be computed to show the efficiency of the proposed framework. 3. How about the performance comparison with Transformer models?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

See the weakness.

Reviewer ho2W8/10 · confidence 5/52023-07-05

Summary

The paper describes an evolutionary training algorithm to optimize the binary weight matrix of a recurrent spiking neural network. The optimization algorithm is derived using natural evolutionary strategies assuming that the weights follow a Bernoulli distribution with parameter $\rho$. The resulting weight update is remarkably simple, and despite that, it seems to work very well. The algorithm is tested on the classical locomotion tasks of simulated robotics, and the performance is reported in terms of task return and time to solution in wall clock time. On all the task RSNN model trained in this way performs better or similar to the LSTM or GRU baselines trained with ES, and it is always better than the spiking networks trained with surrogate gradient. By leveraging the discrete values of the weight matrix, the implementation is also faster than a floating point evolutionary strategy.

Strengths

As said in the paper: inference-only training solutions can be very useful for hardware where the training algorithm cannot be implemented easily. This approach is likely to have important for training or fine-tuning architectures when deployed on unconventional devices. I would also highlight an important achievement that the authors did not mention enough in my opinion: it appears that this training algorithm is better than all existing training algorithms for spiking networks which is remarkable for something that is practicable and competitive with deep learning solutions to such problems. It could in fact provide opportunities for quantized and spiking architectures which are otherwise bounded in their performance by the approximate gradients that are usually computed with surrogate gradient and straight through. This is solved here since the gradients are unbiased (not mentioned by the authors as far as I see). This was probably why surrogate gradient baselines do not quite reach the performance of LSTM and GRU baselines as seen here as in previous papers, but the algorithm provides a solution to this problem.

Weaknesses

1) There are a few data points that would be useful to fully evaluate the quality of the training algorithm and separate the benefits of the spiking architecture from the benefits of the algorithm. a. Is it possible to train an LSTM or a GRU with EC? What is its return after training, is that also better than ES for the spiking neurons? b. What is the return obtained with LSTM / GRU and PPO? This is both important to verify that PPO is well-implemented and claim or not that the EC-RSNN is competitive with the regular deep learning approach. (If the performance is low, then maybe there is a problem with the surrogate implementation too). 2) I think it would be valuable to clarify that most surrogate gradient techniques were typically tested mainly without recurrent connections (Slayer, super-spike for instance). I find it admirable that the authors tested so many baselines already, but also unfortunate that they might have missed important details that were reported the be crucial in the presence of recurrent connections: depending on the weight initialization it was shown that the pseudo derivate should be multiplied by a dampening factor < 1 to avoid the accumulation of approximation errors, with Glorot initialization you might be fine without this (see Celotti and Rouat 2022). But what initialization did you use and did you test this? 3) I realized only later that most of the concrete hyperparameters are detailed in the appendix: How many layers, units per layer, batch size, epsilon. Please state clearly in the main text that all this important is available in the appendix when it's relevant. I find it in general even better to put directly these numbers in the main text when possible. (For instance, write $\epsilon = 0.001$ instead of $\epsilon \rightarrow 0$). Same remark when talking about the hardware. Instead of writing "GPGPU" or "identitical hardware" which I find vague since neuromorphic hardware is often mentioned in the main text. When possible, I would encourage writing just NVIDIA GPU and pointing out the appendix where the hardware reference is provided. On a related note, I would also encourage the authors to prefer a specific reference to the Figure number and panel at line 277 when commenting on their results. 4) In the context of this conference which is broadly addressed to the machine learning community, I think it is important that the surrogate gradient is nothing more than a variant of the straight-through gradient estimator from Bengio 2013 which is much better known outside of the neuromorphic community and was discovered before. 5) I find the first sentence line 20 a bit strange, and I find it weird to reference Pei et al. at this point. AGI is also never used elsewhere in the text (for good reasons since you have more concrete and good results to discuss), so why defining this acronym here ?

Questions

I realized that points 1a.b and 2 above would be best addressed with extra simulations. Given the time and effort that it requires, I do not consider those simulations necessary, but I would encourage the authors to address those questions verbally during the rebuttal and consider the feasibility and the cost/benefits of these potential simulations at some point. For points 3, 4 and 5. I would encourage the authors to do minor edits. More general question: Do the authors believe that this approach can replace gradient descent learning in some other context ? Or is it that ES and EC and bounded to make only sense in this type of toy reinforcement leanring task where the network size is small and the batch size can be super large?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

I see no potential negative impact. The limitations are clearly stated in the paper.

Reviewer Apb47/10 · confidence 3/52023-07-09

Summary

The authors present a new evolutionary algorithm, evolving connectivity (EC) to train recurrent spiking neural networks (RSNN). The algorithm alleviates the gradient estimation problem of surrogate gradient-based (SG) training methods which are often hard to implement in neuromorphic hardware. Further, by focusing the algorithm only on evolving network connectivity instead of the weight magnitude, EC can reduce all network weights down to 1bit {0,1} alternatives making it further hardware friendly. The paper also highlights the superior accuracy and performance of their training algorithm for a few robotics tasks involving sequential decision making against state-of-the-art Surrogate Gradient and Evolution Strategies based algorithms.

Strengths

The authors present a strong argument for developing RSNNs which focus more on the connectivity probability between layers instead of selecting specific weight magnitudes. This enables the network to implicitly fit many samples without overfitting to any specific subset of data. The paper explains the EC algorithm clearly and highlights justifications for its superiority in performance and accuracy compared to other algorithms (ES/SG) for targeted robotics benchmarks. Removing the gradient estimation requirement and reducing the weight matrix to 1-bit are significantly useful optimizations that make the algorithm hardware friendly.

Weaknesses

The paper doesn’t clearly describe the RSNN architecture that is adopted for the experiments discussed and how it compares in terms of number of neurons/connections to the Deep RNNs that are used as baseline for performance/accuracy. This makes the comparisons to the ES-RNN results difficult to understand for me. Further, comparing the RSNN based experiments only (ES/EC/SG), it appears that only EC algorithm is 1-bit owing to the focus on connectivity only. But this makes me wonder if 1-bit networks applying ES/SG algorithms could compete against EC approach in terms of performance. I understand that by the authors’ assertion that connectivity is more critical than assigning wide ranging weights to a fixed set of connections. However, for experiment’s sake I am curious to understand that if ES/SG algorithms also had the HW friendly assumption of only creating a binary SNN then would their accuracy/performance improve or deteriorate?

Questions

1. Describe the RSNN architecture and how it compares to the RNNs used in Experiments in terms of number of layers/params/model size/precision? 2. Experiment with 1-bit RSNN utilizing ES/SG algorithms for training to understand the impact on accuracy and performance? 3. The combination of ES and RSNN performs notably well on all the benchmarks tested, but it would be good to understand what part of it comes from the network architecture (try another architecture), what part comes from bit precision and lastly what is contributed by the training algorithm? This kind of study will help strengthen the credibility of EC algorithm for other benchmarks and network architectures.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors provide a useful discussion on the memory footprint of the different training algorithms discussed. Please address the concerns related to network architecture and precision impacting accuracy and performance as discussed in weakness section.

Reviewer SW362023-08-11

Thanks for the explaination and the additional experiments. It looks like the energy consumption of the EC-RSNN over SG-RSNN is not so advantagous, it is strange since EC is claimed to utilize a 1-bit discrete search space, please explain the reason further. The transformer on online reinforcement learning have been explored further in recent years, such as Zheng, Q., Zhang, A., & Grover, A. (2022, June). Online decision transformer. In international conference on machine learning (pp. 27042-27059). PMLR.

Authorsrebuttal2023-08-12

**[Comment 1]** *It looks like the energy consumption of the EC-RSNN over SG-RSNN is not so advantageous, it is strange since EC is claimed to utilize a 1-bit discrete search space, please explain the reason further.* **[Response]** Thank you for your insightful comment. We appreciate the opportunity to further clarify the energy consumption comparison between EC-RSNN and SG-RSNN. There are two primary reasons for it: 1. **Comparison Standard.** Since epoch in SG and generation in EC are different concepts, our initial comparison computed the final energy consumption under the similar total computation time, which did not account for the difference in the final return. In fact, EC-RSNN achieved a higher final return (i.e., 13,808) compared to SG-RSNN (i.e., 11,505). To provide a more comprehensive comparison of energy consumption, we have included an additional evaluation based on the energy consumption required to 'solve' the Humanoid task. We have set the 'solve' return thresholds at 11,300, which corresponds to the PPO return reported by the Brax maintainer. As illustrated in Table 1, when comparing the energy consumption at the same return thresholds, EC-RSNN demonstrates a more advantageous energy efficiency than SG-RSNN. This comparison highlights the benefits of utilizing a 1-bit discrete search space in EC-RSNN and provides a clearer understanding of the energy consumption differences between the two models. | Model | Return | Run time (h) | Power Consumption (KWh) | |:---------:|:------:|:------------:|:-----------------------:| | EC-RSNN | 11,300 | 6 | 1.7 | | SG-RSNN | 11,300 | 20 | 5.6 | 2. **GPU Implementation.** It is noteworthy that GPUs have significantly expedited floating-point operations, a prevalent data type within the deep learning community. However, 1-bit is not a built-in data type in JAX and GPU, like most computational platforms. In our current JAX code, we could only utilize the boolean data type as a substitution, whose efficiency is limited compared to a 1-bit implementation. Despite this implementation difficulty, we achieved 3x efficiency with the same return. It demonstrated that our EC-RSNN can have a potentially more significant advantage compared to SG-RSNN. In the future, we will further try to implement our EC-RSNN on specially designed neuromorphic devices supporting 1-bit implementation. --- **[Comment 2]** *The transformer on online reinforcement learning has been explored further in recent years, such as Zheng, Q., Zhang, A., & Grover, A. (2022, June). Online decision transformer. In the international conference on machine learning (pp. 27042-27059). PMLR.* **[Response]** Thanks for your insightful suggestion. The paper you referenced blends offline pretraining and online finetuning; it does not entirely adopt the framework of online learning. Actually, offline reinforcement learning has become a prevailing approach within the domain of transformer models (Lili Chen et al., 2021; Michael Janner et al., 2021). In contrast, online learning remains an area of ongoing research, primarily due to the persisting challenges related to stability (Emilio Parisotto et al., 2019). It is worth highlighting that all of our conducted experiments are grounded in the paradigm of online learning, thus rendering direct comparisons with pre-trained transformers incongruous. Despite this distinction, we intend to explore it and run additional experiments in the future.

Reviewer ho2W2023-08-11

Thank you

Thank you for clarifying things and running additional simulations. I find indeed that the comparison between PPO-RNN and EC-RNN are compelling. These are great additions for the paper, the choice of dampening factor 0.8 is not necessarily standard (I had seen 0.3 for initialization $1/\sqrt{n}$) but the results in the non spiking RNN suggest that the stability of the gradient estimator would not explain the failure of PPO, so I am now convinced. I had forgotten one question, do you subtract the mean or any baseline to the return R in equation (14). I believe this should yield a gradient estimator with lower variance? If not, do you know why is it not necessary?

Authorsrebuttal2023-08-12

Thanks for your professional question. Alternatively, we adopted a similar fitness shaping trick, center rank transform, as discussed in ES (Tim Salimans et al., 2017) and NES (Daan Wierstra et al., 2014). The transform sort the return of a population, then rescale the sorting indexes into a fixed interval $[-0.5, 0.5]$. Evidently, it yields an outcome with zero mean and fixed variance. Moreover, it reduces the influence of the outlier individual and helps with stability. We will include the details in our revised paper. Additionally, during the research process, we tried several other fitness shaping methods on population return, including no rescaling, subtraction of mean, and rescaling to fixed interval $[-0.5, 0.5]$. We found out that these methods can train EC, but resulting in a lower final return and often come with stability issues.

Reviewer ho2W2023-08-13

Thank you for the clarification, yes I think adding these details would be useful. Altogether, I see no flaws in the paper and the new simulations and clarifications are improving the paper quality. I therefore increase my rating by one point.

Reviewer SW362023-08-13

Thanks for your responses. I appreciate the addidtional experiments, however, the power consumption is not compared with ANN models, it makes it not convinced to show the energy efficiency advantage of SNNs. Is there any other ways to compare the energy consumption for the comparison?

Authorsrebuttal2023-08-15

Thank you for your question. To further demonstrate our energy efficiency, we have conducted computations pertaining to the estimated energy consumption of EC-RSNN when implemented on the Loihi chip [1]. The data we used are presented in the table below. *Table: Energy data from Loihi [1] & Network data from EC-RSNN* | Parameter | Value | |---------------------------------------|-----------| | Energy per synaptic spike op $P_s$ | 23.6 (pJ) | | Within-tile spike energy $P_w$ | 1.7 (pJ) | | Energy per neuron update $P_u$ | 81 (pJ) | | # Generations $G$ | 1000 | | # Population $P$ | 10240 | | # Time steps $S$ | 33200 | | # Neurons $N$ | 256 | | # Spikes per neuron per step $R$ | 0.025 | | # Connection per neuron $C$ | 128 | | # Update operations per neuron $I$ | 4 | Firstly, we calculated the estimated energy consumption of one network. $$ E_{one} = P_u * N * I * S + (P_s + C * P_w) * N * R * S = 2.8mJ $$ Then, we calculated the total energy consumption during training. $$ E_{tot} = E_{one} * G * P = 28 kJ $$ Our ANN baselines (ES/PPO-LSTM/GRU) demand several hours of training time when executed on GPUs, consequently leading to energy consumption on the order of megajoules (MJ). Our calculations, however, indicate a noticeable reduction in power consumption by an order of magnitude of approximately $1\sim2\times$. Please note that these computations are estimates and serve as a preliminary evaluation. In the future, we intend to conduct experiments using neuromorphic chips to empirically substantiate the disparity in energy consumption between EC-RSNN and RNNs. [1] Loihi: A Neuromorphic Manycore Processor with On-Chip Learning, 2018.

Reviewer mDuJ2023-08-13

Response acknowledged

I appreciate the additional experiments and details that the authors provided to enhance the robustness, addressing many of my primary concerns. I also value the discussion contrasting with Stockl et al. While including code during the review process would have improved transparency, I trust the authors to provide a copy to the area chair, alleviating this concern. With these issues addressed, I will raise my score above the acceptance threshold.

Authorsrebuttal2023-08-14

Thanks for your positive response to our rebuttal. We will try our best to make our paper more clear in the next version.

Reviewer Apb42023-08-16

Response to Author Rebuttal

Thanks to the authors for providing detailed feedback on the questions discussed in the review. A few comments on the response: 1. The breakdown of network parameters and memory footprint is very helpful. 2. I appreciate the effort to try out the suggested experiment, I do see a significant degradation in the performance of SG baseline on moving to the 1-bit precision. While the degradation for ES baseline is smaller, its performance is still lagging behind the proposed EC. Also this seems to highlight the difference in the optimization approach (continuous variable vs connectivity) and their performance for the same task. 3. Very useful data on Vanilla RNN training as well, thanks for generating it. In conclusion, I am satisfied with the authors responses and will increase my score by one point.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC