Thank for your review
We thank the reviewer for their comments. We request the reviewer to also read the General Response provided by us to all reviewers for more context on comparisons and real-world applicability of our method. In the following we respond to other questions raised by the reviewer.
- Inference Efficiency and Comparison with Approximate Methods:
While comparisons to token pruning, cache compression, and sparse attention mechanisms are valuable, we believe they are outside the scope of this paper. These are methods to approximate the full attention model with performance vs. speed tradeoffs, while we focus on an exact inference method for convolutional sequence models without quality loss. In our view, comparing with these methods requires understanding and defining the acceptable quality loss inherent in approximate attention models. Many of these methods involve trade-offs between speed and accuracy, often with parameters adjusted on a case-by-case basis. Given this, a fair comparison is difficult to make without a clear understanding of the quality loss in these approximations. As such, we consider it premature to make these comparisons in the current context.
As an example, consider the evaluation style in the Laughing Hyena paper (NeurIPS 2023), which contrasts the approximate Hyena model with the base Hyena model in terms of inference speed and approximation loss. Like us, they do not compare their method against sparse attention models. It is crucial to note that our method does not need to rely on providing trade-offs—our approach provides exact computations but at a significantly faster inference speed, with no need to introduce any approximation or quality loss.
- Applicability to GPUs Without Optimized FFT Support:
Regarding the use of GPUs without optimized FFT support, we acknowledge the ongoing efforts to improve FFT performance on GPUs, including recent works such as Flash FFT Convolution (https://arxiv.org/abs/2311.05908) and earlier studies (https://arxiv.org/abs/2302.06646). These works demonstrate that FFT-based convolutional methods now outperform traditional attention mechanisms on GPUs due to their faster computational complexity. We would like to draw the reviewers’ attention to Table 6 in Flash FFT Convolution paper ((https://arxiv.org/abs/2311.05908)) and Figure 1 in the Laughing Hyena paper (https://arxiv.org/abs/2310.18780), which show that FFT-based convolution methods significantly outperform attention in GPU settings. Moreover, in token generation tasks starting from a prompt, our method can directly leverage these GPU-optimized algorithms, as a single FFT call produces all the necessary prefil cache information for our approach, further boosting performance.
- Our apologies regarding the margin issue. We will fix and correct this.