Spiking Neural Network as Adaptive Event Stream Slicer

Event-based cameras are attracting significant interest as they provide rich edge information, high dynamic range, and high temporal resolution. Many state-of-the-art event-based algorithms rely on splitting the events into fixed groups, resulting in the omission of crucial temporal information, particularly when dealing with diverse motion scenarios (\eg, high/low speed).In this work, we propose SpikeSlicer, a novel-designed plug-and-play event processing method capable of splitting events stream adaptively.SpikeSlicer utilizes a low-energy spiking neural network (SNN) to trigger event slicing. To guide the SNN to fire spikes at optimal time steps, we propose the Spiking Position-aware Loss (SPA-Loss) to modulate the neuron's state. Additionally, we develop a Feedback-Update training strategy that refines the slicing decisions using feedback from the downstream artificial neural network (ANN). Extensive experiments demonstrate that our method yields significant performance improvements in event-based object tracking and recognition. Notably, SpikeSlicer provides a brand-new SNN-ANN cooperation paradigm, where the SNN acts as an efficient, low-energy data processor to assist the ANN in improving downstream performance, injecting new perspectives and potential avenues of exploration. Our code is available at https://github.com/AndyCao1125/SpikeSlicer.

Paper

References (50)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer cS276/10 · confidence 5/52024-06-18

Summary

This paper proposes to use spiking neural networks (SNNs) to slice the event stream in an adaptive manner before passing the voxelized events to the downstream inference model. The first step of the proposed method divides the input event stream into voxelized event cells with the same temporal interval. An SNN, constructed to have a scalar output, then takes the event cells recurrently. The timestamps when the SNN generates spikes are considered to be the slicing positions. The slicing SNN and the downstream inference model are trained together using the membrane potential-driven loss and the linear-assuming loss. The feedback-update strategy allows the two networks to be trained end-to-end. Extensive experiments on a toy example, object tracking, and recognition demonstrate that the proposed method can be easily integrated into existing models, bringing a noticeable performance improvement.

Strengths

1. The paper focuses on an interesting yet underexplored problem, which is to use a data-driven model to adaptively construct the event voxels. The proposed method is intuitive, and the key idea is convincing. It is clear that the authors have put in great effort in preparing this submission. 2. The key technical contribution involves two parts. First, the feedback-update strategy allows supervision signals from the downstream ANN to back-propagate to the SNN. Additionally, the membrane potential-driven loss and the linear-assuming loss control the spiking time through the supervision of the membrane potential value. The two parts complement each other, leading to an end-to-end trainable model. 3. Additionally, the paper also discusses how the hyperparameter $\alpha$ can be tuned together with the SNN weights and analyzes the implication of different $\alpha$ values to the spiking behavior. 4. The experiments are very extensive. The SpikeSlicer has been validated on several event-based applications, demonstrating its prediction quality, efficiency, and the fact that it can be easily incorporated into existing models.

Weaknesses

1. Despite the strengths above, the key design appears to be a bit simple. As a potential NeurIPS paper, this work is relatively weak on the technical sophistication and theoretical insights. However, this is complemented by extensive experimental evaluation and empirical analysis. 2. While SNNs are efficient and consume less energy than ANNs, SNNs are also less capable than ANNs. Since the speed of the entire SNN+ANN prediction pipeline is going to be slow anyway, it may be worthwhile to investigate whether using an ANN as an event slicer can lead to better prediction quality. 3. While the proposed losses are justified by proposition 1 and empirical analysis, it is unclear if the proposed feedback-update strategy is the best way to identify the desired trigger time $n^*$. In particular, it is unclear if the argmin operator can return any meaningful signal during the initial training stages.

Questions

I like the key idea and hope the paper can be accepted. I encourage the authors to respond to the third bullet point in the "Weaknesses" section. Additional experiments addressing the second bullet point are **not** expected as part of the rebuttal.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.

Reviewer HdTR6/10 · confidence 3/52024-07-13

Summary

This work proposes a novel method for adaptively sample event data and subsequently preprocess it, utilizing a spiking neural networks (SNNs) as module. The sampling method involves a feedback mechanism that triggers the activation of the SNN.

Strengths

Tests are done on dataset with different lighting conditions. Being robust to different event rates.

Weaknesses

The experiments conducted do not contain tasks such as optical flow, object detection, or image reconstruction. The type of tasks tested is limited.

Questions

no questions

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The implementation of the code appears to be challenging, which may affect its reproducibility. The application of this algorithm in embedded systems seems to be constrained due to the use of Spiking Neural Networks (SNNs). Furthermore, it is unclear whether the code will be made publicly available.

Reviewer ij7X5/10 · confidence 4/52024-07-13

Summary

The authors designed a plug-and-play event processing method, SpikeSlicer, to split event streams with an adaptive amount. The proposed method is a lightweight SNN, constrained by a custom Spiking Position-aware Loss (SPA-Loss) to regulate neuron states. Additionally, a downstream ANN refines the slicing decisions using a feedback-update training strategy.

Strengths

* The proposed plug-and-play event processing method, SpikeSlicer, based on SNN representation, which can be used for various vision tasks with event cameras. * The downstream ANN refines the slicing decisions using a feedback-update training strategy, and the performance of the downstream ANN provides feedback to adjust the representation. * Experimental results demonstrate that SpikeSlicer can effectively enhance the performance of object tracking and recognition with event cameras, while also leveraging the advantages of neural computation in processing speed and power consumption.

Weaknesses

* The comparison algorithm for event-based object tracking, DiMP, is from 2019. Why not try the latest methods? In recent years, many studies have focused on improving the effectiveness of event stream representation to enhance the performance of event vision tasks. * Many methods for object detection and tracking with event cameras have not been compared.

Questions

The input to SNNs also involves a time constant definition, which implicitly includes the concept of a time window. How did the authors determine this time constant?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

There is not much discussion on this aspect.

Authorsrebuttal2024-08-07

Reference part of the Response to Reviewer HdTR

Due to word limitations, the reference part of the response to reviewer HdTR is moved to the official comment, thanks! ***Reference:*** [1] Tan G, Wang Y, Han H, et al. Multi-grained spatio-temporal features perceived network for event-based lip-reading. CVPR, 2022. [2] Wang Y, Du B, Shen Y, et al. EV-gait: Event-based robust gait recognition using dynamic vision sensors. CVPR, 2019. [3] Mueggler E, Rebecq H, Gallego G, et al. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM. IJRR, 2017. [4] Nguyen A, Do T T, Caldwell D G, et al. Real-time pose estimation for event cameras with stacked spatial lstm networks. CVPRW, 2017. [5] Yu F, Wu Y, Ma S, et al. Brain-inspired multimodal hybrid neural network for robot place recognition. Science Robotics, 2023. [6] Deng L, Wang G, Li G, et al. Tianjic: A unified and scalable chip bridging spike-based and continuous neural computation. IEEE Journal of Solid-State Circuits, 2020. [7] Roy A, Nagaraj M, Liyanagedera C M, et al. Live demonstration: Real-time event-based speed detection using spiking neural networks. CVPR, 2023. [8] Kumar N, Tang G, Yoo R, et al. Decoding eeg with spiking neural networks on neuromorphic hardware. TMLR, 2022. [9] Viale A, Marchisio A, Martina M, et al. Lanesnns: Spiking neural networks for lane detection on the loihi neuromorphic processor. IROS, 2022.

Reviewer cS272024-08-09

Response to the Rebuttal

Dear Authors, Thank you for submitting the rebuttal! Given the unanimous support, this paper is likely going to be accepted. This is a solid work that deserves acceptance. However, I encourage the authors to investigate using ANNs to predict the slicing positions in the future. Sincerely, Reviewer cS27

Authorsrebuttal2024-08-09

Response to Reviewer cS27

Dear reviewer cS27, Thank you very much for supporting and recognizing our work!! We will explore the topic of the ANN-based event slicer in the future. Thanks! Best, authors

Reviewer HdTR2024-08-14

Thanks for the clarifications and effort. I'm going to keep my rating. I don't have any more questions.

Authorsrebuttal2024-08-14

Response to Reviewer HdTR

Dear reviewer HdTR, Thank you very much for your support!! We will incorporate the supplemented results into the future manuscript. Thanks! Best, authors

Reviewer ij7X2024-08-14

Thank you for your response and for providing a comparison with the WACV2024 method. However, I think the authors still lack comparisons with recent algorithms focusing on enhancing event representation, as this is a common challenge in event camera applications. Additionally, I suggest referring to the event-based vision resources on GitHub (https://github.com/uzh-rpg/event-based_vision_resources?tab=readme-ov-file#feature-detection-and-tracking), which continuously compile works on detection and tracking algorithms. Regarding point A3, if the authors use a fixed number of event cells 𝑁 as input to the model, does that mean the time interval 𝛿𝑡 becomes a variable? Based on the operating principle of DVS, the faster the motion, the more events are generated in a short period, and conversely, fewer events are generated during slower motion. Therefore, I find the authors' claim of "adaptive" somewhat ambiguous. How does this differ from the method illustrated in Fig. 1(b)? Considering these concerns, I may need to reconsider the reproducibility of this work as a plug-and-play module.

Authorsrebuttal2024-08-14

Response to Reviewer ij7X

**Q4:** *I think the authors still lack comparisons with recent algorithms focusing on enhancing event representation...* **A4:** Thank you for your question! **Our approach focuses on slicing the event stream rather than proposing a new event representation method** (refer to Line 27-32 of the original manuscript). Our method is designed to be compatible with **any** event representation. In terms of comparing different event representation methods, we have already provided comparisons in Table 4 of the manuscript, which includes EST, Timesurface, and voxel grid methods. These comparisons demonstrate the effectiveness of our approach across various event representations. As you mentioned, DVS cameras can produce uneven events due to variations in motion speed. **Our motivation is precisely to address the challenge of performance degradation caused by traditional methods that apply fixed slice to the event stream, which is independent of the age of the model.** We have validated the effectiveness of our method through extensive experiments across a range of models and settings, including tracking, action recognition, and object recognition tasks. Additionally, following your suggestion, we have included a 2024 model within the short rebuttal period. Moreover, **in the rebuttal period**, we have supplemented with **four** new downstream tasks: lip reading, gait recognition, pose relocalization, and object detection. The summary of these experiments (**more than 100**) are summarized in the table below: | | **Main Experiments** | | | **Ablation**| **Rebuttal Supplementary** | | | | |:-------:|:-------:|:-------:|:------:|:------:| :------:|:------:|:------:|:------:| | Name | Object Tracking | Action Recognition | Object Recognition| Event Representation| Lips Reading| Gait Recognition| Pose Estimation | Object Detection| | Experiment settings | 4 backbones, 4 scenes, 3 slicing methods | 3 backbones, 2 datasets, 3 slicing methods |3 backbones, 2 datasets, 3 slicing methods | 2 tasks, 3 representation methods, 3 slicing methods | 2 backbones, 1 datasets, 2 slicing methods | 2 backbones, 1 dataset, 2 slicing methods|1 backbone, 2 datasets, 2 slicing methods | 1 self-collected dataset, 3 backbones, 2 slicing methods | |Experiment Number | 48 | 18 | 18 |18 | 4 | 4 | 4| 6| Our extensive experiments can support our method's effectiveness. Thank you for sharing the information on the latest models. As the deadline is approaching, we will try our best to include the latest models in future versions. We sincerely appreciate your suggestion! --- **Q5:** *...How does this differ from the method illustrated in Fig. 1(b)?* **A5:** I think there might be some misunderstandings here. First, the "number of event cells" and the "event count" in Figure 1(b) represent two **different** concepts. Allow me to clarify the dynamic slicing process of the SNN as follows: Suppose we have an event stream from the FE108 airplane dataset with a total duration of 24994 $\mu s$, and the SNN's inputs are 10 event cells. Each event cell (as defined in Definition 1) corresponds to an event voxel with a duration of $\delta t = \frac{24994}{10} \mu s \approx 2.5$ ms . These 10 event cells are then fed into the SNN (based on an Integrate-and-Fire neuron without decay). If a spike occurs at the 3-rd position, we extract the sub-event stream within the time interval of [0, 3*2.5] = [0, 7.5] ms. At this stage, any event representation method can be used to convert the sub-event stream into an event representation, which can then be fed into downstream tasks. Since the SNN's spike output is dynamic, the event slicing process is dynamic as well. We hope our response helps to clarify your concerns. Thank you for your questions!

Reviewer ij7X2024-08-14

Thank the authors for their swift reply. I have no further questions and hope the authors will consider providing open access to the data and code.

Reviewer HdTR2024-08-14

I have no further questions and will keep my rating. Thank you to the authors for the clarifications.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC