Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators

Summary

This paper proposes a new position encoding technique CPG-PE for SNNs, which is inspired by the central pattern generator in the human brain and improves the ability of SNNs to process sequence data. Experimental results show that CPG-PE outperforms traditional SNNs in multiple fields such as time series prediction, text classification and image classification.

Strengths

1. The bio-inspired CPG-PE technique is proposed to enhance the sequence processing capability of SNNs. 2. The effectiveness of CPG-PE is verified in multiple fields, including time series, text classification, and image classification. 3. The performance of SNNs in various tasks is significantly improved, outperforming traditional models. 4. CPG-PE is designed with compatibility with neuromorphic hardware in mind, facilitating deployment in practical applications.

Weaknesses

1. No experiments were conducted on the large-scale image classification Imagenet dataset. 2. Other encoding methods in SNN such as rate coding [1] and temporal coding [2;3], were not discussed. [1] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75. [2] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404. [3] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.

Questions

1. I suppose the traditional positional encoding in Spikformer can be seen as direct encoding in SNNs. In contrast, the CPG-PE proposed in this paper is more like a temporal encoding with dynamic expressions related to time series. Could the authors discuss the differences between other encoding schemes [1;2;3] and positional encoding in SNNs within the related work? 2. Can the authors provide experimental results of CPG-PE on the ImageNet image classification dataset? 3. Some tyops: Formulas 4, 5, 6, 7, 8, and 10 are missing commas. There should be a comma after each formula. I would be pleased to raise the score if the authors address my concerns. [1] Kim Y, Park H, Moitra A, et al. Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?[C]//ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022: 71-75. [2] Han B, Roy K. Deep spiking neural network: Energy efficiency through time based coding[C]//European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 388-404. [3] Comsa I M, Potempa K, Versari L, et al. Temporal coding in spiking neural networks with alpha synaptic function[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 8529-8533.

Rating

7

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

The author has discussed the limitations

Summary

This paper introduces a novel positional encoding method for SNNs called CPG-PE, inspired by central pattern generators (CPGs) in biological neural systems. The authors demonstrate both theoretically and empirically that CPG-PE can effectively capture positional information in sequential data while maintaining the spike-based nature of SNNs. The approach is evaluated on a range of tasks including time-series forecasting, text classification, and image classification.

Strengths

1. Strong theoretical foundation: The authors mathematically demonstrate how CPG-PE relates to conventional sinusoidal positional encoding used in transformers. 2. Comprehensive empirical evaluation across time-series forecasting, text classification, and image classification tasks shows consistent performance improvements when incorporating CPG-PE. 3. The method is biologically plausible and potentially compatible with neuromorphic hardware, as it can be implemented using leaky integrate-and-fire neurons. 4. Well-structured paper with clear motivation, methodology, and insightful analysis of CPG-PE properties and their relationship to biological CPGs.

Weaknesses

1. Typographical error on line 158 where X is incorrectly specified as belonging to the real number domain instead of the binary domain {0,1} for spike data. 2. The authors should clarify the similarities and differences between the decay mechanism in CPG-PE and that of LIF neurons.

Questions

See my weakness part.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed limitations and future work in Appendix E, which is commendable. They discuss the challenges of applying CPG-PE to non-sequential data like images and propose potential solutions.

Summary

The lack of an effective and hardware-efficient spike-frm position encoding strategy in SNNs has been a consistent motivation for this study. Drawing inspiration from the central pattern generators (CPGs) in the human brain, which produce rhythmic patterned outputs without requiring rhythmic inputs, this work proposes a novel PE technique for SNNs, termed CPG-PE. Extensive experiments across various domains show the superior performance with CPG-PE.

Strengths

1.To the best of my knowledge, this is the first work on position encoding in SNNs, laying the foundation for efficient sequence modeling in SNNs. 2.The approach utilizes the coupling of multi-neuron pulse signals as position encoding, which is innovative. 3.The authors demonstrated the effectiveness of this method across various tasks. 4.The spike-position encoding generating method through mutual inhibition between two groups of spiking neurons is brain-inspired and hardware-friendly.

Weaknesses

1.This type of positional encoding, through aggregation, introduces a small number of additional parameters and computational overhead. Please provide ablation experiments demonstrating that the performance improvement is not solely due to these factors. 2.I suggest the authors include results on ImageNet to demonstrate the effectiveness on large-scale datasets.

Questions

1.In ANNs, positional encoding is typically added to features. Please analyze the similarities and differences of this category aggregation-based positional encoding compared to ANNs.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

I believe the authors' discussion on limitations is comprehensive.

Summary

This paper introduces central pattern generators (CPGs) from neuroscience into the SNN framework as a novel method for position encoding. Through mathematical derivation, it is proven that the existing abstract PE methods in transformers are actually a particular solution for a specific type of CPG. The effectiveness of CPG is validated through experiments across several domain benchmarks.

Strengths

1. This article connects existing abstract Positional Encoding (PE) methods in transformers with Central Pattern Generators (CPGs) in the human brain through mathematical derivation, showing that the former can be viewed as a specific mathematical solution to the membrane potential dynamics of the latter. This presents an interesting viewpoint. 2. The paper is well-organized and clearly written, offering high readability. Readers can effortlessly grasp the authors' intentions, supported by both the textual explanations and accompanying illustrations.

Weaknesses

1. The authors have not convinced me why the problem addressed in this paper is very important, i.e., why the existing PE methods in SNNs are such a big problem that they need to be improved by CPG. 2. The method proposed in the paper is simple and does not provide enough inspiring insights; the contribution is relatively limited. 3. The implementation of CPG-PE on hardware involves coupled nonlinear oscillators that require frequent updates of neuron membrane potentials, entailing floating-point computations and memory read-write operations, which result in additional energy expenditures. The paper should scrutinize and analyze whether the performance enhancements afforded by this encoding method justify the additional energy costs. This trade-off demands a detailed examination to assess its viability in practical applications. 4. In Sec3.1, why is "F(x)=b<=0, H(y)=d<=0" followed after “...gain membrane voltage with constant speed” instead of “F(x)=b>0, H(y)=d>0”? 5. In Sec4.2, does the CPG-Full method replace all linear layers in the model with CPG-Linear layers? Why is its performance not as good as that of CPG-PE? Can you provide further analysis and explanation?

Questions

Please refer to weakness 3, 4, 5

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

NA

Response to the rebuttal

Thank you for the response to my questions. The idea of building positional encoding in SNN is interesting and I am willing to increase my score to 6.

Thank you

The reviewer thanks the authors for the discussion. It addressed all of my concerns. I decide to keep my score.