We respectfully disagree with the reviewer. There have been extensive community efforts in building foundation models for continuous control [11, 12, 13, 14], given their significant potential to create versatile robotic agents that can be utilized in various real-world applications. Since generalization capability inherently stems from the scale of the model and training data, most (if not all) approaches are built upon large-scale Transformer backbones. Although these models may not be advantageous for real-time operation compared to lightweight models, we believe that the value of these efforts should not be underestimated simply due to computational cost, especially when considering their capabilities and significance for general-purpose robots.
Additionally, there have always been parallel efforts in the machine learning community to develop more capable models and to accelerate their computation. As discussed in our rebuttal, there is an extensive body of work aimed at reducing the computational demands of Transformers, including techniques such as quantization[15], pruning[16], distillation[17], and linear attention[18], as well as methods to enhance hardware utilization, such as FlashAttention[19, 20, 21]. These approaches are orthogonal to our method and generally applicable to any Transformer backbone. Moreover, considering ongoing developments in hardware that rapidly reduce the cost of computation and enhance the applicability of large-scale models, we believe it is important to refrain from judging the real-time applicability based solely on current computational limitations.
[11] Janner et al., “Offline Reinforcement Learning as One Big Sequence Modeling Problem”, NeurIPS, 2021
[12] Sun et al., “SMART: SELF-SUPERVISED MULTI-TASK PRETRAINING WITH CONTROL TRANSFORMERS”, ICLR, 2023
[13] Chen et al., “Decision Transformer: Reinforcement Learning via Sequence Modeling”, NeurIPS, 2021
[14] Liu et al., “Masked Autoencoding for Scalable and Generalizable Decision Making”, NeurIPS, 2022
[15] Liu et al., “Post-Training Quantization for Vision Transformer”, NeurIPS, 2021
[16] Kim, Sehoon, et al. "Learned token pruning for transformers." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.
[17] Sanh, V. "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." Proceedings of Thirty-third Conference on Neural Information Processing Systems (NIPS2019). 2019.
[18] Choromanski, Krzysztof Marcin, et al. "Rethinking Attention with Performers." International Conference on Learning Representations. 2020.
[19] Dao, Tri, et al. "Flashattention: Fast and memory-efficient exact attention with io-awareness." Advances in Neural Information Processing Systems 35 (2022): 16344-16359.
[20] Dao, Tri. "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning." The Twelfth International Conference on Learning Representations.
[21] Shah, Jay, et al. "Flashattention-3: Fast and accurate attention with asynchrony and low-precision." arXiv preprint arXiv:2407.08608 (2024).