**[Comment 1]** *It looks like the energy consumption of the EC-RSNN over SG-RSNN is not so advantageous, it is strange since EC is claimed to utilize a 1-bit discrete search space, please explain the reason further.*
**[Response]**
Thank you for your insightful comment. We appreciate the opportunity to further clarify the energy consumption comparison between EC-RSNN and SG-RSNN. There are two primary reasons for it:
1. **Comparison Standard.** Since epoch in SG and generation in EC are different concepts, our initial comparison computed the final energy consumption under the similar total computation time, which did not account for the difference in the final return. In fact, EC-RSNN achieved a higher final return (i.e., 13,808) compared to SG-RSNN (i.e., 11,505).
To provide a more comprehensive comparison of energy consumption, we have included an additional evaluation based on the energy consumption required to 'solve' the Humanoid task. We have set the 'solve' return thresholds at 11,300, which corresponds to the PPO return reported by the Brax maintainer. As illustrated in Table 1, when comparing the energy consumption at the same return thresholds, EC-RSNN demonstrates a more advantageous energy efficiency than SG-RSNN. This comparison highlights the benefits of utilizing a 1-bit discrete search space in EC-RSNN and provides a clearer understanding of the energy consumption differences between the two models.
| Model | Return | Run time (h) | Power Consumption (KWh) |
|:---------:|:------:|:------------:|:-----------------------:|
| EC-RSNN | 11,300 | 6 | 1.7 |
| SG-RSNN | 11,300 | 20 | 5.6 |
2. **GPU Implementation.** It is noteworthy that GPUs have significantly expedited floating-point operations, a prevalent data type within the deep learning community. However, 1-bit is not a built-in data type in JAX and GPU, like most computational platforms.
In our current JAX code, we could only utilize the boolean data type as a substitution, whose efficiency is limited compared to a 1-bit implementation. Despite this implementation difficulty, we achieved 3x efficiency with the same return. It demonstrated that our EC-RSNN can have a potentially more significant advantage compared to SG-RSNN. In the future, we will further try to implement our EC-RSNN on specially designed neuromorphic devices supporting 1-bit implementation.
---
**[Comment 2]** *The transformer on online reinforcement learning has been explored further in recent years, such as Zheng, Q., Zhang, A., & Grover, A. (2022, June). Online decision transformer. In the international conference on machine learning (pp. 27042-27059). PMLR.*
**[Response]**
Thanks for your insightful suggestion. The paper you referenced blends offline pretraining and online finetuning; it does not entirely adopt the framework of online learning. Actually, offline reinforcement learning has become a prevailing approach within the domain of transformer models (Lili Chen et al., 2021; Michael Janner et al., 2021). In contrast, online learning remains an area of ongoing research, primarily due to the persisting challenges related to stability (Emilio Parisotto et al., 2019).
It is worth highlighting that all of our conducted experiments are grounded in the paradigm of online learning, thus rendering direct comparisons with pre-trained transformers incongruous. Despite this distinction, we intend to explore it and run additional experiments in the future.