Autonomous driving demands an integrated approach that encompasses perception, prediction, and planning, all while operating under strict energy constraints to enhance scalability and environmental sustainability. We present Spiking Autonomous Driving (SAD), the first unified Spiking Neural Network (SNN) to address the energy challenges faced by autonomous driving systems through its event-driven and energy-efficient nature. SAD is trained end-to-end and consists of three main modules: perception, which processes inputs from multi-view cameras to construct a spatiotemporal bird's eye view; prediction, which utilizes a novel dual-pathway with spiking neurons to forecast future states; and planning, which generates safe trajectories considering predicted occupancy, traffic rules, and ride comfort. Evaluated on the nuScenes dataset, SAD achieves competitive performance in perception, prediction, and planning tasks, while drawing upon the energy efficiency of SNNs. This work highlights the potential of neuromorphic computing to be applied to energy-efficient autonomous driving, a critical step toward sustainable and safety-critical automotive technology. Our code is available at \url{https://github.com/ridgerchu/SAD}.
Paper
Similar papers
Peer review
Summary
Although SNNs have potential of neuromorphic computing in sustainable and safety-critical autonomous technology, they still lack evidence in complex real-world computer vision applications. In this work, the authors propose a unified end-to-end SNN called SAD that consists of three models to generate safe trajectories, which presents improved energy efficiency. The researchers complete the validation on the nuScenes dataset, which not only verifies the good performance, but also proves the effectiveness to a certain extent.
Strengths
The proposed idea is novel and relevant to the NeurIPS community. 1. SNNs have attracted a lot of attention in recent years due to their better performance and low energy consumption, but the application potential remains to be explored. In this work, the authors investigate an end-to-end SNN-based approach applying in autonomous driving, which shows impressive performance similar to traditional neural network approaches. This increases the impact of the work. 2. The proposed method can help reduce energy consumption and plan a safe path, which shows that SNNs have great potential to be applied in the field of autonomous driving systems. 3. In experiments, the authors compare the proposed method with traditional deep learning works in recent five years, and obtain competitive results.
Weaknesses
Comparisons with state-of-art works in the last three years are lacking in the results of perception, prediction, and planning in Section 4.1.
Questions
1. In related works, the authors should add a relevant description of the connection between this research and End-to-end Autonomous Driving, as in the previous paragraph “We push...in this paper ”. 2. In Section 4.1, the authors mention that "The results, as summarized in Tab. 1, show that our SAD method, ..., competes favorably against state-of-the-art, non-spiking artificial neural networks (ANNs)”. But I only see a specific figure of 7.43%. Please add descriptions to support it. 3. This work lacks a discussion of the robustness of the proposed method, I would like to recommend adding experiments on more datasets. 4. In Table 4 and Table 5, the authors do not show complete ablation experiments, e.g., SEW+SP in Table 4, SA+SR in Table 5, etc. Please provide the necessary details.
Rating
6
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
Some details in the manuscript are not clearly stated and algorithmic performance needs more convincing.
Dear Reviewer XRtX, With the discussion period drawing to a close, we wanted to extend our heartfelt appreciation for your insightful comments and the positive feedback on our work, which has been incredibly helpful to us. We would be immensely grateful if you could kindly inform us whether our responses have resolved your concerns or if you have any additional questions that we can assist with.
Dear Reviewer XRtX, We're writing to kindly remind you that today is the final day of our author-reviewer discussion period. If you have any remaining concerns, we would be most grateful if you could share them with us as soon as possible. Thank you once again for your positive review. Your insights have been invaluable, and we sincerely appreciate your time and expertise.
Summary
This paper introduces Spiking Autonomous Driving (SAD), an end-to-end spiking neural network (SNN) designed for autonomous driving. SAD integrates perception, prediction, and planning into a unified neuromorphic framework. It achieves competitive performance on the nuScenes dataset and significantly outperforms traditional artificial neural network (ANN) methods in terms of energy efficiency. The authors anticipate that this work will spur further research into neuromorphic computing for sustainable and robust autonomous driving solutions.
Strengths
1. Autonomous driving can be seen as an energy-limited scene, so I think it is meaningful to introduce SNNs to this task. To the best of my knowledge, this paper is among the first to utilize SNNs to AD. 2. This paper provides a unified framework on how to apply SNNs toAD. The authors employ a dual-pathway architecture and I think this is an innovative design in the use of spiking neurons. 3. SAD achieves competitive results on the BEV segmentation IoU and semantic segmentation IoU, indicating the practical viability of the proposed method. The results in Table 3 have really surprised me, for its great energy-efficiency.
Weaknesses
1. Notation typos. It seems that the authors make mistakes in equations in Line 148, 155, 157, and 169. For example, I think $X \in$ \{0,1\} $^{C\times T\times L\times H\times W}$ instead of $X \in \mathbb{R}^{C\times T\times L\times H\times W}$ because $X$ should be a spike train rather than a floating-point matrix. 2. I think energy efficiency is the key to this paper. However, the authors talk little about this issue in Section 4.1. Can you provide me more about how you calculate the energy consumption and your detailed energy experiments of all 3 steps (i.e., perception, prediction, and planning)?
Questions
1. I wonder if the numbers in your proposed SNNs are spikes or not. $\mathbb{R}$ stands for real number field. 2. Can you show the detailed energy results of all steps in Table 3? I think it is incomplete to show the planning step only. 3. In Line 238, I wonder why you chose GRU to refine the trajectory. Does GRU have unique advantages on this issue?
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors mentioned limitations in the Conclusion Section, focusing on further validation through real-world on-vehicle testing. However, it would be better for authors to discuss limitations in more detail.
Summary
The authors adapted ST-P3 into a version with a binary spiking neuron and then stated that autonomous driving based on a spiking neural network can address the energy challenges. The authors stated that this neuromorphic technology can be a step toward sustainable and safety-critical automotive technology.
Strengths
This work has completed the end-to-end SNN-based autonomous driving pipeline, which is a heavy job. The work has a clear architecture. This work is easy to follow. The idea of using SNN in driving technology is interesting.
Weaknesses
The evaluation is not as effective as its reference work, ST-P3, which has evaluations of open loop validation and close loop validation. The implementation plan on neuromorphic hardware is not clear in this paper, which I think it’s the most important aspect for energy-efficient applications. No comparison with recent state-of-the-art ANN models (later than ST-P3), as the necessity of autonomous driving using SNN is an open question. It is better to include more comparison and further discussion.
Questions
What is the advantage of the SNN solution from the perspective of a safety-critical application? As this work is oriented towards this specific application, it is better to clearly state the advantages. Do you have specific safety considerations in the design of this model? Do you include any related work on SNN robustness in this paper? Dual pathways and other architecture designs manifest a problem of mapping on neuromorphic hardware. I suspect that SGRU is as energy efficient as simple LIF. Eqs. 6, 7, 8, and 9 imply heavy matrix computation. Can you provide an explanation for this? ANN seems more robust than SNN (Figs. 5, 7). In the figures of ANN outputs, are the vehicle markers clearer and more accurate? What do you think SNN will do to lead to this result? Do you have any solutions to this? Can you include experiments on this to address this problem? (This question is important.)
Rating
4
Confidence
4
Soundness
2
Presentation
3
Contribution
3
Limitations
See questions and weaknesses.
Extended Rebuttal
> ANN seems more robust than SNN (Figs. 5, 7). In the figures of ANN outputs, are the vehicle markers clearer and more accurate? What do you think SNN will do to lead to this result? Do you have any solutions to this? Can you include experiments on this to address this problem? (This question is important.) Response: Thank you for this important observation. You're correct that the ANN outputs appear more robust than the SNN outputs in Figures 5 and 7, with clearer and potentially more accurate vehicle markers. This difference highlights a crucial challenge in the field of neuromorphic computing. The primary reason for this disparity lies in the fundamental nature of SNNs. While SNNs offer significant advantages in terms of energy efficiency due to their discrete, spike-based processing, this same characteristic can lead to information loss during training and inference. This trade-off between efficiency and performance is a well-known challenge in the SNN domain. The discretization of information into spikes, while beneficial for efficiency, can result in a reduction of fine-grained details, potentially leading to less clear outputs compared to traditional ANNs. This performance gap is something the entire field is working to overcome. However, it's important to note that our work represents a significant step forward in applying SNNs to complex, real-world tasks like autonomous driving. We've made substantial efforts to bridge this performance gap: 1. We've developed novel architectural approaches that are specifically tailored to maintain spike-driven processing while improving performance in complex tasks. 2. Our training methods have been carefully designed to maximize the information carried by spikes, helping to mitigate some of the inherent limitations of spike-based computation. 3. We've introduced innovative components, such as our unique Spiking Token Mixer, which have allowed us to achieve competitive performance without relying on traditional ANN elements. These efforts have made it possible to apply SNNs to autonomous driving tasks, which was previously considered extremely challenging. While there's still room for improvement, our work demonstrates that SNNs can be viable for such complex applications. Regarding robustness, you've highlighted an important point that, while not the primary focus of this paper, is crucial for the practical application of SNNs in safety-critical systems like autonomous driving. Improving SNN robustness is indeed a vital direction for future research. There are several promising approaches to enhance SNN robustness, such as developing more advanced neuron models, exploring different surrogate gradient methods, or applying adversarial training techniques. These methods could potentially be integrated with our current work to further improve performance and reliability. In conclusion, while the current performance gap between SNNs and ANNs is evident in our results, our work represents a significant advancement in applying SNNs to complex, real-world tasks. We've laid a foundation that future research can build upon, not only to further improve performance but also to address critical aspects like robustness. This opens up exciting possibilities for the future of energy-efficient, neuromorphic computing in autonomous driving and other demanding applications. --- [R1] Ding J, Yu Z, Huang T, et al. Enhancing the robustness of spiking neural networks with stochastic gating mechanisms [AAAI]
Dear Reviewer j4mM, As we near the conclusion of the discussion period, we wanted to extend our sincere gratitude for your valuable feedback. We would be most appreciative if you could let us know whether our responses have resolved your concerns or if there are any areas where you feel further elaboration would be beneficial.
Dear Reviewer j4mM, This is a kind reminder that today is the last day of the author-reviewer discussion period. If you have any concerns, please let us know as soon as possible so that we can address them.
Summary
This paper presents an end-to-end SNN model for the autonomous driving to address the energy challenges. This model consists of three main modules: perception, prediction, and planning. The model is evaluated in the nuScenes dataset.
Strengths
1. This paper introduce the first SNN designed for end-to-end autonomous driving, integrating perception, prediction, and planning into a single model. 2. well-written.
Weaknesses
1. The novelty is limited. The paper just uses the SNN to autonomous driving and does not provide any special designs. 2. Only one dataset is used. 3. The energy conputation does not consider the data moving.
Questions
see weekness.
Rating
4
Confidence
5
Soundness
3
Presentation
3
Contribution
2
Limitations
None
Thank you for considering our rebuttal and raising the score. **Regarding energy consumption, we have provided a detailed explanation in Appendix F**. We'd like to further elaborate on two main aspects of energy consumption: ### **Memory Access** Memory access is critical for model inference and training latency. Techniques like Flash Attention [R1] use fused kernels to reduce memory access frequency, significantly improving the efficiency of attention kernels. However, memory access is not directly proportional to energy consumption; it's also related to the number of synaptic operations (SOPs) [R2] when inferencing on CPUs and GPUs. SOPs refer to the number of operations performed in neural networks, As we mentioned in the rebuttal, in neuromorphic hardware, a new paradigm called in-memory computing is emerging. Designs like IBM's NorthPole [R3] and Intel's Loihi [R4] partially adopt this concept. In these systems, synaptic weights between neurons are determined by synapse strength, greatly reducing latency and energy consumption while fully utilizing sparsity. ### **Compute Energy** Unlike memory access, compute energy is highly correlated with the SOPs in neural networks, even though it's not the primary bottleneck for training/inference latency. As shown in [R5], the number of neural network SOPs is almost directly proportional to energy consumption. SNNs gain energy efficiency mainly through two aspects: **a) Binary activations** In ANNs, the SOPs primarily involve MAC (Multiply-Accumulate) operations; but in SNNs, all MAC operations can be performed without multiplication, using only addition. This can be represented mathematically as: $\text{ANN: } y = \sum_{i} w_i x_i,$ where $x_i \in \mathbb{R}$ $\text{SNN: } y = \sum_{i} w_i x_i,$ where $ x_i \in \\{0,1\\}$ **b) Sparsity** The binary nature of activations results in many zeros, which neuromorphic chips can exploit to create event-driven sparsity. These two factors—addition-only operations and sparsity—form the foundation of SNN's superior energy efficiency. In modern neuromorphic chips like Loihi 2 [R4] and Speck [R6], sparsity is the primary factor in reducing computational energy. They leverage the graph-like nature of neural networks, constructing each neuron as a router rather than representing synapses as matrices, as in traditional GPUs. ### **Energy Consumption Calculation** To quantify the energy efficiency of our SNN architecture, we calculate the theoretical energy consumption using the following methodology: 1. Calculate Synaptic Operations (SOPs) for each block: $$\operatorname{SOPs}(l) = fr \times T \times \operatorname{FLOPs}(l)$$ where: - $l$ is the block number - $fr$ is the input spike train firing rate - $T$ is the neuron time step - $\operatorname{FLOPs}(l)$ are the floating-point operations in the block 2. Compute SNN energy consumption: $$E_{SNN} = E_{MAC} \times \mathrm{FLOP}{\mathrm{SNN}\mathrm{Conv}}^1 + E_{AC} \times \left(\sum_{n=2}^N \mathrm{SOP}{\mathrm{SNN}\mathrm{Conv}}^n + \sum_{m=1}^M \mathrm{SOP}{\mathrm{SNN}\mathrm{FC}}^m\right)$$ where: - $E_{MAC} = 4.6 \text{pJ}$ (MAC operation energy cost) - $E_{AC} = 0.9 \text{pJ}$ (AC operation energy cost) - $N$ and $M$ are the number of Conv and FC layers, respectively - The first Conv layer uses direct encoding, employing MAC operations, so FLOPs are used for its energy calculation. 3. For comparison, ANN energy consumption: $$E_{ANN} = E_{MAC} \times \mathrm{FLOP}{\mathrm{ANN}}$$ This approach allows us to directly compare the energy efficiency of our SNN with ANN models. We hope this explanation provides a more comprehensive justification for our energy consumption calculations. Thank you once again for your constructive comments and for giving us the opportunity to improve our paper. --- [R1]:Dao, Tri, et al. "Flashattention: Fast and memory-efficient exact attention with io-awareness." Advances in Neural Information Processing Systems 35 (2022). [R2]:Tripp, Charles Edison, et al. "Measuring the Energy Consumption and Efficiency of Deep Neural Networks: An Empirical Analysis and Design Recommendations." arXiv:2403.08151. [R3]: Modha, Dharmendra S., et al. "Neural inference at the frontier of energy, space, and time." Science 2023. [R4]: Orchard, Garrick, et al. "Efficient neuromorphic signal processing with loihi 2." 2021 IEEE Workshop on SIPS [R5]:Lahmer, Seyyidahmed, et al. "Energy consumption of neural networks on nvidia edge boards: an empirical model." IEEE, 2022. [R6]:Richter, Ole, et al. "Speck: A smart event-based vision sensor with a low latency 327k neuron convolutional neuronal network processing pipeline." (2024).
Summary
This paper presents Spiking Neural Network to reduce the energy consumption by normal neural network in autonomous driving. The network includes end-to-end architecture including perception, prediction and planning. It evaluates the model using nuScenes dataset and achieves comparable performance in three modules yet drawing much lower energy consumption.
Strengths
The author proposes a good architecture and describe in detail all of the modules: perception, prediction and planning. I enjoy the reading and found it very good to list in details of architecture, especially the results, visualization and ablation study. In general, I think this stands a good paper. The reason I rate it at 5 is due to not familiar with the work in energy consumption using spike neural networks.
Weaknesses
I do not write this due to not being familiar with the work.
Questions
- Do the author know if any self-driving company use spiking neural network in their real operation? This can justify the impact of this work. - Energy is saved but how about memory, inference time?
Rating
5
Confidence
2
Soundness
3
Presentation
3
Contribution
3
Limitations
I think the work addresses the proposed question about reducing energy consumption. Besides, I do not have expert knowledge in spiking neural networks, thus I shall refrain from talking about limitations.
Summary
This paper introduces a unified Spiking Autonomous Driving (SAD) system based on Spiking Neural Networks (SNNs). The system is trained end-to-end and comprises three main modules: a perception module that processes inputs from multi-view cameras and constructs spatiotemporal bird's-eye views; a prediction module that forecasts future states using a novel dual-pathway approach; and a planning module that generates safe trajectories based on the predictions. The authors evaluated SAD on the nuScenes dataset, where it demonstrated competitive performance in perception, prediction, and planning tasks. By leveraging its event-driven and energy-efficient nature, SAD effectively addresses the energy challenges faced by autonomous driving systems.
Strengths
1. This paper constructs the first end-to-end spiking neural network designed specifically for autonomous driving. By integrating perception, prediction, and planning into a single network, it enhances efficiency and reduces the complexity associated with managing these components separately. 2. The experimental results are comprehensive and persuasive, with a thorough comparison with other works, demonstrating the significant potential of SNNs in complex real-world applications.
Weaknesses
1. Energy Consumption Analysis: The main motivation for implementing autonomous driving tasks with SNNs in this paper is to enhance energy efficiency, with a corresponding energy efficiency analysis provided in Appendix F and Table 3. Nonetheless, I have reservations about the energy consumption calculations presented for the following reasons: - Since both ANNs and SNNs process the same input data (with SNNs employing direct encoding at the first layer), the difference in energy efficiency can only stem from differences in memory access or the energy consumption of MAC/AC operations. This paper only offers a quantitative comparison of energy consumption during computation (L748-L768). - It is well-known that energy consumption is primarily determined by memory access rather than FLOPS or IPS [1r], yet this aspect is not discussed in the paper, nor is it considered in the energy consumption calculations. This should at least be acknowledged, as it is a common shortcoming in publications claiming reduced energy consumption. Could the authors comment on how memory access impacts the energy consumption of SNNs? - ANN networks typically exhibit similarly sparse ReLU activations. Previous research has shown that even on general-purpose CPU hardware, the sparsity of ReLU can be utilized to accelerate (and reduce energy consumption of) inference [2r]. How would the comparison results between SNNs and ANNs change if ANN sparsity were considered? [1r]: An Analytical Estimation of Spiking Neural Networks Energy Efficiency. ICONIP 2022 [2r]: Inducing and Exploiting Activation Sparsity for Fast Neural Network Inference. ICML 2020 2. Inference Speed: In comparisons with six ANN methods, SNN performance only surpassed half of the existing methods. Assuming SNNs have lower energy consumption, we would consider using SNNs for autonomous driving tasks. What about their inference speed in practical deployments? Autonomous driving tasks rely on high dynamic resolution and timely responsiveness, such as detecting and avoiding obstacles on highways.
Questions
Referencing the Weakness section, I recognize that some evaluation methods are commonly used within the SNN community. However, due to differences in task formulations, explanations and clarifications are necessary.
Rating
5
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
N/A
Thanks or your response
Thanks for your response. My second concern have be well addressed. However, I still feel the novelty is limited and the energy computation is biased. I have raised my score. hope the author could further give a more detailed analysis for energy computation.
I have thoroughly reviewed the authors' response, and they have addressed my concerns. I also read their replies to other reviewers and noticed that some reviewers also had doubts about the energy computation of this paper as I did, but I solved the doubts about how to calculate energy consumption in the authors' second reply to reviewer EVdw.
Thank you
Thank you for your thorough review and for taking the time to read our responses to all reviewers. We are grateful for your increased rating of our paper and value your contribution to improving our work, and we appreciate your careful consideration of our explanations, especially regarding the energy computation.
Response
Thank you authors for providing a detailed response. I am satisfied with your answers. This is an important piece of work to reduce the energy consumption for autonomous driving and I believe it is quite novel as in the first time, I read energy-efficient architecture using SNN. I keep my score as 5 due to low confidence in the work. I appreciate the author's response to other reviewers and hope can improve the overall quality of paper regardless of decisions.
Dear Reviewer x6bz, As the discussion period is drawing to a close, we wanted to reach out and kindly inquire if our rebuttal has adequately addressed the concerns you raised. We would be grateful for any feedback on whether our responses have clarified the points in question or if you have any additional queries we can assist with.
Thank you for the author's response. I carefully reviewed the relevant content, and the author mentioned the sparsity of SNNs compared to ANNs and the low latency during inference, which convinces me that this research is indeed meaningful. Nevertheless, there are still many aspects in the author's response that are more of a commitment, such as the deployment of the algorithm on a chip. While this requires further validation, it is not a sufficient reason for rejection. Therefore, I have decided to raise my score from 4 to 5.
Thank you
Thank you for your recognition and increased score, and your decision to raise the rating from 4 to 5 is a great encouragement to us. We truly appreciate your careful review of our work and your understanding of its significance. Indeed, implementing our algorithm on a chip is a challenging process, but we hope our work can at least demonstrate that SNNs are capable of performing autonomous driving tasks, rather than just simple operations like voice wake-up on vehicles. The latter is currently the main function of the neuromorphic chip [installed in the Mercedes conceptual car EQXX](https://www.eetimes.com/mercedes-applies-neuromorphic-computing-in-ev-concept-car/). We wish our research can push the boundaries of what SNNs can achieve in automotive applications. In the future, we will also attempt to realize hardware implementation of our approach.
Epilogue by Authors
We would like to express our sincere gratitude to all six reviewers for their time and effort in evaluating our work. We are particularly thankful to the four reviewers who actively participated in the discussion period: Reviewers x6bz, GVJq, EVdw, and S2mt,. Their engagement and insightful feedback have been invaluable in refining and strengthening our research. **We are pleased to note that three of the four participating reviewers increased their scores by one point following our responses, indicating that our rebuttals successfully addressed many of their initial concerns.** Reviewer GVJq, while maintaining their original score, still found our responses satisfactory. All responding reviewers indicated that our rebuttals either fully or partially resolved their concerns: - Reviewer x6bz acknowledged the significance of our research in applying SNNs to autonomous driving tasks. - Reviewer GVJq recognized the potential impact of our work in advancing SNN-based approaches for autonomous driving. - Reviewer EVdw noted that our second concern regarding the dataset has been well addressed. - Reviewer S2mt found their doubts about energy calculations resolved after reading our responses. We are also grateful for the positive feedback from the other reviewers: - Reviewer j4mM recognized that "this work has completed the end-to-end SNN-based autonomous driving pipeline, which is a heavy job." - Reviewer XRtX mentioned that "The proposed idea is novel and relevant to the NeurIPS community." Once again, we extend our heartfelt thanks to all six reviewers and the Area Chair for their expertise and dedication in evaluating our submission. Your collective insights have been crucial in enhancing the quality and impact of our work.
Decision
Accept (poster)