Summary
This article addresses two issues:
1. The low efficiency of the LonvConv-based model during inference.
2. Whether it is advantageous to perform independent long convolutions on each channel or reduce the total number of filters without loss in quality.
To tackle problem (1), the authors propose distilling the LonvConv into a Diagonal State space model and train it using the $\ell_2$ loss function.
For problem (2), the authors suggest sharing long convolution coefficients across multiple channels, resulting in the MultiHyena structure.
The effectiveness of the proposed methods is validated on multiple datasets.
Strengths
Distilling LonvConv into a Diagonal State space model is indeed a novel and meaningful approach. The conclusion of sharing long convolution coefficients across multiple channels is also innovative.
Weaknesses
I think the main issue with this article lies in the focus of the writing. From Equation 3.4, it is clear that the ultimate goal is to represent the LonvConv coefficients using a Diagonal State Space Model. However, a significant portion of the article is spent describing unrelated aspects. The most crucial part, determining the hidden dimension of the State Space Model, is merely brushed over, even though it is the key factor that affects the quality of the distillation and the efficiency of the final inference. On the other hand, the motivation behind the design of MultiHyena should be addressed in the main text since it is of utmost importance.
Questions
1. The method for determining the hidden dimension of the State Space Model needs to be explained in more detail. I referred to Appendix E.3.2, and I'm wondering if the core idea is to perform an SVD decomposition and then select the dimension for dimensionality reduction based on the eigenvalues?
2. The solution to Equation 3.4 requires a more detailed algorithm description or pseudocode to help readers follow along. I attempted to replicate it following Appendix B.1, but the results were not quite good. Could you provide the training configurations and an estimate of the training time?
3. The motivation behind the design of MultiHyena should be included in the main text, and ablation studies (sharing coefficients vs not sharing coefficients) should be conducted to validate the design's rationale. The experiments should compare the effects and speeds.
4. Does MultiHyena utilize the Local conv + Global conv structure of Hyena? If so, how many layers are used? This should be mentioned in the experiments.
5. Is the Algorithm 1 on page 28 is the implementation of Figure 4.1?
6. Regarding the implementation of MultiHyena, in Algorithm 1 on page 28, for $z^m_t \in \mathbb R^{L\times N\times N}$, what does the subscript $t$ represent? On the other hand, is the shape of $T_h$ $L$ (all features share one set of convolution coefficients) or $L\times N\times N$ (each feature has independent convolution coefficients)?
7. Continuing with the implementation of MultiHyena, in Algorithm 1 on page 28, is the shape of $T_hz^m_t$ ${L\times N\times N}$? If so, does $T_h(z^m_t) q_t^m$ mean performing matrix multiplication between $[T_h(z^m_t)]_i \in \mathbb R^{N\times N}$ (for $i=0,\ldots,L-1$) and $q_t^m\in \mathbb R^{N}$, resulting in an output of shape $\mathbb R^{N}$? If not, what is the computation like?
8. The algorithm's output is $\bar{y} \leftarrow\left(\sum_m\right) y^m / M\in \mathbb R^{L\times N}$, while the input has a shape of $L\times D$. Is this inconsistency problematic, or did the algorithm omit something?
9. Algorithm 1 has several ambiguities. It is suggested to reorganize it for better clarity.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.