Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional State Space Models

> *As the author stated the Classical 2D SSMs are only able to capture linear dependencies, It could be better to compare with Koopman-based deep models [1][2], which are similar to SSMs and excel at tackling nonlinear dynamics. I also encourage the author to discuss this branch in related works.* **Response:** Thank you for bringing these studies to our attention. Indeed they are relevant papers and we will make sure to properly discuss them in the final version of the paper. Following the reviewer's suggestion, we have conducted experiments to compare Koopa with Chimera. Our results show that Chimera outperforms Koopa in 30/32 forecasting settings and provide 9% improvement on average. We will report the full results in the final version of the paper. --- ### References: --- [1] Graph mamba: Towards learning on graphs with state space models. Behrouz et al., 2024. [2] Vision mamba: Efficient visual representation learning with bidirectional state space model. Zhu et al., 2024. [3] Pointmamba: A simple state space model for point cloud analysis. Liang et al., 2024.

### References: --- [1] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. Lio et al., 2024. [2] ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis. Luo et al., 2024. [3] TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. Qiu et al., 2024. [4] TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. Wu et al., 2023. [5] SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention. Ilbert et al., 2024. [6] Frequency-domain MLPs are More Effective Learners in Time Series Forecasting. Yi et al., 2023. [7] Recurrent Convolutional Neural Network for Object Recognition. Linag and Hu, 2015.

We thank Reviewer gRR3 for their response and for engaging with us in the discussion. We are so grateful. > While evaluating models at the target horizon is crucial, it is equally important to assess whether the dataset is appropriate for providing meaningful insights. **Response:** Thank you for mentioning that. In our experiments, we have already evaluated Chimera at the target horizon in Table 8. For the validity of datasets in forecasting task, we kindly bring to your consideration that we have used **all** the datasets that are used in SAMformer (ICML 2024 Oral), SparseTSF (ICML 2024 Oral), WITRAN (NeurIPS 2023 Spotlight), iTransformer (ICLR 2024 Spotlight), FITS (ICLR 2024 Spotlight), and CrossFormer (ICLR 2023 Oral). We have also provided the results for two additional datasets (suggested by Reviewer hxax). Please note that this is only one of the four types of tasks that we have focused on. Most papers in the community, including WITRAN, have focused only on long-term forecasting (5 datasets in WITRAN). Our paper, not only has used 10 datasets for long-term forecasting, including WITRAN's five datasets, but also includes short-term forecasting, anomaly detection, and classification. **We, however, would be happy to report the results of Chimera on any specific dataset that the reviewer believes is necessary.** Please note that our goal is to compare our architecture to the existing literature, which those works have also built on. To this end, and for the sake of fair comparison, it is important to follow the literature. Assuming we would not follow the literature and would not use common benchmarks, our evaluation could be criticized for cherry-picking datasets and baselines. Proposing new datasets, and assessing the validity of commonly-used datasets, is indeed an important direction, but it is out of the scope of our paper. Please also note that all the above studies, including WITRAN, did not assess their datasets for their tasks, and have used commonly-used benchmarks. > I recommend incorporating additional evaluation sets, as exemplified in WITRAN, and using more robust time series forecasting datasets accompanied by thorough analysis. **Response:** We want to kindly bring to your consideration that our evaluation includes 10 datasets, including **all** five datasets that are used in WITRAN. Similar to WITRAN, and following the reviewer's suggestion, we report the results of the robustness to noise of our approach in the following table. Please note that our paper didn't have this robustness claim, and so we didn't report these results in the initial submission. However, following the reviewer's suggestion to follow WITRAN, we report the results (MSE) here: Model (Noise) | ETTh1 | ETTh2 | ECL | ------------ | :-----------: | -----------: | -----------: | Chimera (0%) | 0.405 | 0.318 | 0.154 | Chimera (1%) | 0.409 | 0.327 | 0.156 | Chimera (5%) | 0.418 | 0.331 | 0.165 | Chimera (10%) | 0.429 | 0.342 | 0.169 | Chimera ind. (0%) | 0.471 | 0.372 | 0.203 | Chimera ind. (1%) | 0.479 | 0.383 | 0.208 | Chimera ind. (5%) | 0.494 | 0.407 | 0.221 | Chimera ind. (10%) | 0.530 | 0.441 | 0.233 | Please note that we have conducted **all** the experiments in WITRAN. Additionally, we have reported our evaluation of short-term forecasting, anomaly detection, and classification tasks. **We would be happy to report the results of any specific evaluation that the reviewer believes is necessary.** > While it is acknowledged that not every proposed method will achieve SOTA, it remains important to analyze the reasons behind the observed underperformance. It also indicates that there is space to improve. **Response:** We agree with the reviewer that if there are patterns in the underperformance of a model, it is important to analyze the reasons behind it. However, Chimera achieves SOTA performance in almost all cases, with a few exceptions with marginal underperformance. These exceptions are diverse across the datasets and tasks, and (similar to all other abovementioned papers) they mostly attributed to the diversity of datasets as well as not **extensively** tuning the hyperparameters. We want to kindly bring to your consideration that a thoroughly comprehensive evaluation of an approach requires unlimited resources and time. Even well-known machine learning models with tens or hundreds of follow-up studies are still being evaluated and improved. Our goal is to show *enough evidence* that Chimera can be an alternative architecture to existing methods, with theoretical motivations, and showing good effectiveness and efficiency. We believe in future studies Chimera and data-dependent 2D SSMs can be studied more extensively and be improved.

**Experiments:** >*many (simpler) methods achieve similar results to the proposed approach …* **Response:** We want to kindly bring to your consideration that Chimera is simpler architecture than Transformer-based models, with less computational cost. In Table 8, Chimera outperforms the best Transformer-based model in 32/32, SSM-based models in 30/32, linear models in 32/32, and convolution-based models in 20/32 experimental settings. Similarly, in Table 9, Chimera outperforms all the linear, RNN-based, and SSM-based models by at least 10% improvement on average. In Table 10, Chimera outperforms all the linear, RNN-based, and SSM-based models by at least 5% improvement on average. In Table 11, Chimera outperforms all the linear, RNN-based, and SSM-based models by at least 6% improvement. > *I also found that the paper tends to overstate its claims* **Response:** Compared to RNN-based and Mamba-based models, Chimera provides about 5% improvement in Table 2, about 3% improvement in Table 3, about 90% improvement in Table 4, about 75% improvement in Table 5. On the other hand, it is $\times 2.5$ faster than other recurrent models. We believe this is a significant improvement, but following reviewer suggestion, we will modify these claims and only will claim that Chimera outperforms these models. > *I question whether a fair comparison was made between the methods.* **Response:** We followed the standard pipelines in the literature for all the methods to ensure a fair comparison. Regarding time series decomposition, please note that this (not new) decomposition is a part of our contribution and architectural design. Modifying existing models so they use this decomposition is a non-trivial task and is out of the scope of the paper. We, however, have considered several baselines that use this time series decomposition in their original design (e.g., Autoformer, FEDformer, Dlinear, SCINet, etc.). Chimera consistently outperforms all these baselines. > *Additionally, there is a lack of naive baselines, such as methods that treat the multivariate series as multiple univariate series.* **Response:** We already have several baselines that treat the multivariate series as multiple univariate series. For example, SpaceTime, LSTM, Autoformer, Stationary, etc., are all such models. > *It would also be useful to visualize some time series decompositions of Chimera* **Response:** Following your suggestion, we have added a visualization of the results to the attached PDF. > *Why didn't you confirm this by analyzing the results?* **Response:** Thank you for mentioning that. We have confirmed this by our experiments. That is, when applying on univariate time series, Chimera is equivalent to SpaceTime but with data-dependent parameters and seasonal-trend decomposition of time series using discretization. Accordingly, the superior performance of Chimera over SpaceTime in this experiment confirms our claim. We will further discuss this point in the final version to make it clearer. > *`The results show that all the components of Chimera contribute to its performance`. I do not agree with this statement …. Why did you only choose ETT datasets?* **Response:** Table 6 shows that using bidirectionality, gating, and seasonal-trend decomposition improve the performance by 6% on average. Please note that this also shows removing each of these components can damage the performance of Chimera. Following your suggestion, we also have provided the ablation study results for more datasets in the attached PDF. > *For the forecasting problems, it would be useful to provide the results per horizon* **Response:** We have already provided the results per horizon in Table 8. Comparing different horizons, Chimera even achieves relatively better results on longer horizons. **Data:** We have followed the literature and performed most of our experiments on datasets that are commonly used in the literature. The main reason is to make a fair comparison since each paper has specified their own sets of hyperparameters that work best for these datasets. We also want to kindly bring to your consideration that we have already used new datasets that are not overly used by the literature (Tables 2, 3, 5). However, following your suggestion, we have added 2 new datasets. The results are reported in the attached PDF. Verifying our conclusion, these results also show that Chimera outperforms the baselines. Please note that the message of our paper is not achieving state-of-the-art performance in anomaly detection. We have used this task to show the effectiveness of Chimera over existing methods, and so we followed the literature for these experiments. However, following your suggestion, we have conducted experiments using the provided pipeline in this paper and conclude that Chimera outperforms baselines in most cases. We will report the results and discuss this paper in more detail in our final version.

**Other Comments:** > *This is confusing as there is also the VAR(p) model with a different formulation* **Response:** Please note that our multivariate AR is the same as VAR mentioned by the reviewer. Please note that coefficients in our formulation are vectors and writing the element-wise formulation of our AR process would be the same as the formulation of VAR. We will use VAR instead of AR in the final version to avoid any confusion. > *What do you mean by "series length"? M4 series do not have six observations. Also, it is not long-term vs short-term but rather multivariate vs univariate time series.* **Response:** Series length is the number of time steps. The M4 dataset has different settings, where the resolutions are hourly, daily, weakly, monthly, quarterly, and yearly. In the yearly resolution, the M4 dataset has 6 time steps (years). Please note that we never mentioned that M4 has a long-term version. Please note that this is the original setting of M4 and also has been used in various studies with the same setting [2, 3, 4]. > *Does this mean that Chimera has been used with both univariate and multivariate time series? Do you still need the second dimension in that case?* **Response:** Chimera is capable of modeling univariate time series. Please note that one of the dimensions in Chimera is variates and in the univariate setting, this dimension has only one state. > *The paper uses different terminologies …* **Response:** Thank you for bringing this to our attention. We used these terms in their literal definition not as a formal term in our paper. However, following your suggestion, we will make sure to consistently use the same term and properly define it in the final version. > *This is supervised anomaly detection, which is rare as labeled anomalies are hard to find. How did you handle the imbalance during training?* **Response:** Please note that even in the case of unsupervised setting, anomaly detection is equivalent to unsupervised classification with two classes of `normal` and `abnormal`. For these tasks, we have followed the literature and benchmarks, and adopted the classical reconstruction task and chose the reconstruction error as the anomaly score. **Typos:** Thank you for bringing this to our attention, we will make sure to proof-read the paper and fix all typos in the final version. We hope our above responses have fully addressed your concerns/questions about the paper. We would be more than happy to answer any further questions or discuss any remaining concerns of the reviewer. --- ### References: --- [1] The Illusion of State in State-Space Models. Merrill et al., 2024. [2] ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis. Luo et al., 2024. [3] TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. Qiu et al., 2024. [4] TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. Wu et al., 2023.

We thank Reviewer hxax for their response and for engaging with us in the discussion, and we are glad that their concerns have been addressed. > I believe the paper still requires significant revisions, particularly in enhancing the experimental setup, refining the datasets used, incorporating naïve forecasting baselines (e.g., ARIMA, ETS), and avoiding overstatements **Response:** As we discussed above, and also mentioned by the reviewer in their initial review, we are using datasets that are commonly used in the literature. We have followed the literature and also considered several additional datasets. Our model shows improvement over both sets of datasets. Similarly, in the experimental setup, we have followed the benchmark studies and literature. We, however, would be more than happy to address any specific limitation about the experimental setup or our used datasets. **Regarding naïve baselines:** We have theoretically shown that our model can recover ARIMA. Also, several of our old baselines have already been compared with ARIMA and show improvement. Therefore, in our initial submission, we have compared with more recent models, similar to most studies in the community. **However, following the reviewer's suggestion, we report the results for ARIMA, ETS, and Chimera in the next comment.** > How do you account for such varied improvements? **Response:** Please note that these results are for different tasks. In fact, Tables 2 and 3 (5% and 3% Improvement) correspond to classification, Table 4 (90% improvement) corresponds to short-term forecasting, and Table 5 (75% improvement) corresponds to the classification of brain decoding, which requires data-dependent parameters and is a complex task. We kindly bring to your consideration that the improvements are not varied across datasets or models, but are varied across tasks, which is expected and reasonable as each task has its own hardness. **We indeed have discussed why our approach provides an improvement in each of tasks, and even have supported it with ablation studies.** For example, in Table 5, we attribute this improvement to data-dependency, and so support that with an ablation study (Tables 5 and 6). We showed using data-independent parameters results in about 17% performance drop. > The authors need to conduct a more robust ablation study to clarify these discrepancies. **Response:** We have conducted ablation studies for **all** the components of Chimera. We show that each of them is contributing to its performance. For each of these tables (tasks) we also have ablation studies (e.g., Table 6 and our 1-page PDF attached in the rebuttal). We, however, would be more than happy to provide the results of any specific ablation study that the reviewer believes is helpful. > Numerous factors, such as hyperparameter tuning and optimization, could explain these variations. **Response:** We have used the best hyperparameters that are reported in the original papers. > Additionally, the paper currently attempts to cover too many directions. **Response:** We are grateful and also want to kindly bring to your consideration that the reviewer has mentioned this as the strength of our paper: `Chimera is compared to multiple baselines across various datasets for tasks such as time series forecasting, classification, and anomaly detection`, in their initial review. We also agree that it is important to show the performance over multiple tasks, which can comprehensively evaluate the contributions of the paper. We want to kindly bring to your consideration that recently, it is a common practice in the community to evaluate the model’s performance over different tasks. For example, please see iTransformer (ICLR 2024 Spotlight), ModernTCN (ICLR 2024 Spotlight), FITS (ICLR 2024 Spotlight), and CrossFormer (ICLR 2023 Oral). Assuming we would not follow the recent literature and would not use different benchmarks, our evaluation could be criticized for cherry-picking tasks and being too specific. --- Finally, we want to kindly bring to your consideration that a thoroughly comprehensive evaluation of an approach requires unlimited resources and time. Even well-known machine learning models with tens or hundreds of follow-up studies are still being evaluated and improved. Our goal is to show enough evidence that Chimera can be an alternative architecture to existing methods, with theoretical motivations, and showing good effectiveness and efficiency. We believe we have already provided **enough evidence** by comparing Chimera with **more than 25 baselines** (including traditional methods like LSTM, recent methods in 2024, and SOTA from 2019-2023) on **several tasks and more than 20 datasets**. We have conducted ablation studies for **each Chimera’s component** and each of our contributions like data-dependencies. We believe in future studies Chimera and data-dependent 2D SSMs can be studied more extensively and be improved.