Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation

Summary

Recent research indicates that rate-coding is crucial for information representation in deep Spiking Neural Networks (SNNs) trained via Backpropagation Through Time (BPTT). Building on this insight, a new training strategy called rate-based backpropagation has been developed to leverage rate-based representations, reducing the complexity of BPTT. This approach focuses on averaged dynamics to simplify the computational graph, thereby lowering memory and computational requirements. Theoretical and empirical analyses demonstrate that this method closely approximates BPTT's gradient optimization, maintaining comparable performance while surpassing other efficient training techniques. This advancement is poised to enable more scalable and resource-efficient SNN training, particularly in environments with limited resources.

Strengths

1. The paper is very well written and documented. 2. The contributions have been discussed comprehensively. 3. The experiments have been conducted on multiple benchmarks.

Weaknesses

Some important details (such as the top-level algorithm of the proposed rate-based backpropagation method and details of the experimental setup) are reported in the appendix, while, due to their importance, they should be moved to the main manuscript.

Questions

1. Can the proposed rate-based backpropagation be implemented on existing neuromorphic chips with learning capabilities? 2. Looking at the results in Fig.4, the impact of the number of timesteps on the time and memory looks constant. How have specific numbers of timesteps been selected for each dataset?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations and societal impacts have been discussed in Appendix D.

Summary

This paper presents a novel rate-based backpropagation method for spiking neural network (SNNS) training, which effectively separates the time-dependent backpropagation (BPTT) process and thus reduces computational and memory costs. The method employs a rate-encoded approximation to capture the basic information and is validated by empirical experiments on various datasets, demonstrating that it is superior in terms of training efficiency and accuracy when compared to the traditional BPTT.

Strengths

1. Empirical results on multiple datasets (CIFAR-10, CIFAR-100, ImageNet, CIFAR10-DVS) support the theoretical claims and ensure accuracy while reducing memory and time costs. 2. The paper is well-written, clearly explaining the proposed method, theoretical underpinnings, and experimental validation.

Weaknesses

1. In lines 53-55, this paper mentions that the proposed method reduces training time, but there is no relevant experimental proof in the experiments section.

Questions

1. In lines 223-234, the reference to 'when cosine similarity close to 1 is interpreted as a high degree of consistency in the direction of the variable', does it take into account the effects of data distribution and noise, which may also occur in the case of uneven data distribution. Can additional experiments or theories be added to rule out the effect of data distribution and noise on the hypothesis presented in lines 223-234? 2. The approach proposed in the paper seems to be very similar to the one described in reference [1]. Although the general direction of the two is different, the core idea seems to be the same. Could you please explain the difference between your approach and the one outlined in reference [1]? 3. In Section 5.3, in the experiments evaluating the effect of time step on accuracy and training, only one dataset, CIFAR-10, was used. Could the experiment be supplemented with experiments using other datasets to demonstrate the scalability of the proposed method for larger values of T? 4. In the caption of Fig. 3, is the placeholder '#' missing from T{timesteps}? Reference: [1] Bu, Tong, et al. "Rate gradient approximation attack threats deep spiking neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors fully explain the limitations and potential social implications of their work.

Summary

This work falls into the category of efficient SNN training methods. This paper proposes a reduced computational graph to reduce the memory and computational demands of SNNs training. This work has the potential to train SNNs on resource-limited devices. The paper evaluates the methods on CIFAR-10, CIFAR-100, ImageNet, and other datasets.

Strengths

This paper addresses the issue of high time and memory costs in training spiking neural networks. This paper provides solid theoretical insights into the error bound and its relation to SNN BPTT training. The results of this work are comparable to the performance of the BPTT counterpart.

Weaknesses

Not a clear comparison of the differences with existing e-prop methods in terms of methodology. No generalization results on hyperparameters (e.g., $\lambda$) are presented in this work. I raise this question because most work on SNNs uses large values of $\lambda$, but this work used 0.2 as $\lambda$.

Questions

Why did the authors approximate the spiking rate directly with the mean over timesteps, instead of using a running mean with a decay parameter $\lambda$, which would more closely approximate the rate in the leaky integration mode? In Line 151, page 5, what does 𝑑 represent in $\frac{\partial I}{partial c} = Id$? Please elaborate further on the differences between rateM and rateS. The authors state that 'rateM represents the multi-step training mode where T loops are embedded within layers, while rateS refers to the single-step training mode with T loops outside the layer.' Regarding robustness to $\lambda$ In the paper, the neuronal parameter $\lambda$ is set to 0.2. Can you provide experiments with other values of $\lambda$, such as 0.4, 0.6, 0.8, and 1.0? I believe that the training cost in Fig. 4 should encompass not only the backward process but also the forward iteration process (which also contributes to the cost).

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

See weakness and questions.

Summary

This paper proposes a rate-based SNN training method, which can effectively reduce memory and time cost during training. They proved the efficiency of the rate-based back-propagation training and demonstrate that the rate-based training outperforms other back-propagation methods.

Strengths

The rate-based method achieves better performance and uses less computing resource compared with BPTT, which is impressive. This paper is well-written and well-organized.

Weaknesses

The novelty is weak. There are two previous works that share similar idea with this paper, since they all use rate-based backpropagation [1,2]. The author needs to briefly explain the differences between these papers. The rate-based backpropagation is not suitable for sequential tasks. [1] Li, Yuhang, et al. "Differentiable spike: Rethinking gradient-descent for training spiking neural networks." Advances in Neural Information Processing Systems (2021). [2] Bu, Tong, et al. "Rate gradient approximation attack threats deep spiking neural networks." Computer Vision and Pattern Recognition (2023).

Questions

The authors introduce the rate-coding approximation forward propagation in Section 4.1. Is this forward propagation method also used during inference? What is the performance of rate$_s$ on ImageNet dataset?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

See in weaknesses.

Response for the Rebuttal

In the rebuttal, the authors distinguished between this rate-based BP and the previous works[1,2], which I found convincing. Since they addressed my major concerns, I will raise my score.

Response to rebuttal

In light of the other reviews and the author's rebuttal, my score is confirmed.