Summary
The paper presents S7, a simplified state-space model (SSM) designed for long-range sequence tasks. Building on the S5 model, it introduces input-dependence to allow dynamic adjustments to state transitions based on input content. The paper claims S7 achieves stable reparameterization and efficient performance across diverse tasks, including neuromorphic event-based datasets and Long Range Arena (LRA) benchmarks, without requiring the complexity of models like S4 or S6 (Mamba). The proposed model is argued to balance computational simplicity with adaptive filtering and content-based reasoning.
Strengths
*Input-Dependent Dynamics:* S7’s adaptation of the S5 model to be input-dependent is a promising approach. This could enhance the model’s responsiveness to input variability, a significant issue in long-range sequence tasks.
*Stable Reparameterization:* The model claims to maintain gradient stability over long sequences, addressing gradient explosion/vanishing issues commonly faced in deep learning. This feature has potential benefits for training efficiency and stability.
*Broad Applicability:* S7’s successful application across various domains, from physical time series to neuromorphic vision, suggests it may generalize well to different task types.
Weaknesses
**Limited Novelty:** The paper is introducing an input-dependent update mechanism (already introduced by S6) stabilized through the reparameterization framework and key equations (Eqs. 6, 7, and 8) borrowed directly from StableSSM [1], raising concerns about the originality of the contributions. The paper only states that it was “inspired” by stable reparameterization, yet much of the core methodology relies on prior work.
**Inconsistent Notation:** The notation for $𝐴_k$ is unclear, with dependency on input appearing inconsistently (e.g., it appears in Eq. 5 and line 267 -- e.g. $𝐴_k(u_k, \theta_m)$ but is omitted elsewhere -- e.g. $𝐴_k(\theta_m)$. This lack of uniformity in notation undermines the model’s theoretical presentation.
**Weak Justification for S5 Model Selection:** S5 is mentioned as the basis for S7, but no rationale is provided for not using S6 (Mamba) and the reparameterization technique from StableSSM. Moreover, no connection or description of the S5 model is given (MIMO approach etc.)
**Assumptions Clarity:** Assumptions (3.1, 3.2, and 3.3) are not well justified or examined for feasibility, and the text lacks guidance on implementing or verifying these assumptions. This leaves important theoretical aspects of the model unaddressed.
**Unclear Contribution of Neuromorphic Design:** The neuromorphic-specific design choices in Section 3.4 seem disconnected from the rest of the model’s development (no other mention on the first part of the paper). It’s unclear whether these additions (Eq. 11, 12) apply exclusively to neuromorphic tasks or extend to other benchmark tasks.
**Lack of Benchmark Justification:** The paper does not clarify why specific datasets were chosen. For instance, given the input-dependent nature of S7, benchmarks used by similar models like Mamba (e.g., Selective Copy or Induction Heads or other similar benchmarks -- see Section 7/Table 4 of the thematic survey [2]) might have been more appropriate for comparison.
**Poor Performance on LRA Benchmarks:** S7’s subpar performance on LRA benchmarks raises concerns about its applicability to heterogeneous data. The authors provide only a brief discussion, without substantial insight or proposed solutions for improving performance on these challenging tasks.
[1] Wang, Shida, and Qianxiao Li. "Stablessm: Alleviating the curse of memory in state-space models through stable reparameterization." arXiv preprint arXiv:2311.14495 (2023).
[2] Tiezzi, Matteo, et al. "State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era." arXiv preprint arXiv:2406.09062 (2024).
Questions
1. **Could the authors clarify the novelty of the reparameterization?** How does it differ from StableSSM’s reparameterization framework?
2. **Why was the S5 model chosen as the basis for S7?** Given that S6 with the StableSSM reparameterization might provide similar benefits, what informed this design choice?
3. **Could the authors specify if the neuromorphic-specific design applies solely to neuromorphic tasks or to all benchmarks?** This would improve clarity regarding the model's consistency across different tasks.
4. **What is the author’s perspective on improving the model's performance on data whre time-dependance is note relevant?** Given S7’s limited success on LRA, is there a feasible modification that could address these challenges while preserving input-dependence?