Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching

Summary

This paper introduces the Rough Transformer, a variant of the original Transformer that allows the processing of discrete-time series as continuous-time signals through the use of multi-view signature attention. Empirical comparisons shows that Rough Transformers outperform vanilla Transformers and continuous-time models on a variety of time-series tasks and are robust to the sampling rate of the signal.

Strengths

- Overall the paper is well written and is easy to follow - The idea of using signature transform within an attention mechanism is interesting

Weaknesses

- Empirical evaluations are limited, casting doubts on the true potential of the proposed architecture - The tasks considered are rather simple, and it is not clear whether the proposed architecture will give favorable tradeoffs between accuracy and efficiency in the more challenging tasks (see below) - Missing evaluations on time series forecasting tasks (only classification and regression tasks are considered) - Missing comparisons with recent RNN models (such as https://arxiv.org/abs/2110.04744, https://arxiv.org/abs/2212.00228), Transformer models (e.g., those studied in https://arxiv.org/abs/2011.04006), State Space models (https://arxiv.org/abs/2111.00396 and the more recent variants such as Mamba: https://arxiv.org/abs/2312.00752) and other sequence models (https://arxiv.org/abs/2305.01638, https://arxiv.org/abs/2209.10655) - Missing ablation studies on the components of the proposed architecture, particularly the role of the global and local components in the multi-view signature, truncation level n, etc. - Missing related work; e.g., the papers mentioned above, https://link.springer.com/article/10.1007/s40304-017-0103-z and https://arxiv.org/abs/1710.10121 for continuous-time DL models, https://arxiv.org/abs/2006.12070, https://arxiv.org/abs/2102.04877 for continuous-time RNN models, https://dl.acm.org/doi/abs/10.5555/3546258.3546305 for using path signatures to understand continuous-time RNNs

Questions

- I was wondering how effective is the proposed model in autoregressive generative tasks and time series forecasting (see https://arxiv.org/abs/2311.04147)? These are natural tasks in the domain of sequence modeling - While the proposed method can improve the modified transformer model in sequence modeling, why would the method be attractive for practitioners when they could just use more sophisticated models like SSMs for sequence modeling?

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes

Summary

The paper proposes Rough Transformers, an attention-based model for long continuous-time signals. The model utilizes ideas from rough path theory to extract path signatures from the continuous-time signal (obtained by interpolation of the original signal). Two types of signatures are extracted: global and local. The global signature extracts long-term information from the signal whereas the local signature extracts local information. Self-attention is then used on this "multi-view" signature. Experiments on classification and regression tasks show improved performance over existing model architectures.

Strengths

- The paper views the problem of modeling long continuous-time signals through the lens of rough path theory. While this in itself is not novel, the combination of rough path signatures with attention is a novel combination. - Experiments on classification and regression tasks show that RFormer improves over existing models in terms of accuracy and significantly more compute efficient. (I have some concerns and questions about the empirical analysis, please see Weaknesses)

Weaknesses

- The technical contribution is limited. In such scenarios, the empirical analysis needs to be sufficiently strong. - The empirical analysis has been conducted on a few toyish datasets. While the results are definitely promising, more experimental support is needed to validate the model. - Although the model is motivated from a continuous-time and irregularly-sampled data perspective, the actual investigation of these settings is limited. As per my understanding, all experiments under 4.1 have been conducted on regularly sampled time series (please correct me, if I am wrong). I find it surprising that simple RNN-based methods do not perform well in these settings. If this is indeed the case, some simple CNN-based method should be studied. Recent models based on state space layers (e.g., S4, Mamba) can also be explored as baselines. - OOM for important baselines is not really helpful to draw any conclusions. To highlight efficiency of RFormer, please conducted experiments where you increased the context length or other parameters to show where baselines run OOM and how do they perform before that. - More experiments are needed, particularly for the forecasting task to understand how well the model understands the dynamics. - I took a brief look at the code and it looks like hyperparameter tuning was conducted for RFormer. Was such tuning also performed for the baselines? If no (which is a valid response), how did you selected the baseline parameters? How sensitive is the model to different hyperparameters? - The discussion on related work needs to be moved to the main text and improved. Please contrast RFormer with the related works, particularly the ones that are closely related such as NRDE and ContiFormer. Discussion on some closely related works [1, 2, 3] is missing. Ideally there should also be a comparison with at least of these methods (e.g., CRU). [1] Schirmer, Mona, et al. "Modeling irregular time series with continuous recurrent units." International conference on machine learning. PMLR, 2022. [2] Ansari, Abdul Fatir, et al. "Neural continuous-discrete state space models for irregularly-sampled time series." International Conference on Machine Learning. PMLR, 2023. [3] Oh, YongKyung, Dongyoung Lim, and Sungil Kim. "Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data." arXiv preprint arXiv:2402.14989 (2024). I am happy to update my score, if my concerns are adequately addressed.

Questions

- Can the authors clarify by what they mean by "input sequences must be sampled at the same times, (ii) the sequence length must be fixed"? These do not seem to be limitations of transformers, especially (ii). See weaknesses for other questions.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

There is a brief discussion on limitations.

Summary

The paper proposes Rough Transformer (RFormer), an extention of the Transformer architecture towards operating on continuous-time representations of time-series data. RFormer employs a novel technique called multi-view signature attention, which performs attention on path signatures pre-computed from input data offline, thereby capturing both local and global dependencies across observations. Experiments on various real-world time-series datasets shows that RFormer enjoys superior predictive performance as well as computational efficiency compared to previous methods, while being robust to changes in sequence length and irregular sampling.

Strengths

- [S1] **Good novelty.** To the best of my knowledge, incorporating rough path theory to time-series representation learning is a novel approach, and would be of great interest to the machine learning community. - [S2] **Great empirical performance.** Experiments on a wide variety of real-world datasets show large improvements in both accuracy and efficiency, demonstrating strong utility of proposed multi-view signatures in time-series modeling.

Weaknesses

- [W1] **Questionable motivation of synthetic frequency classification experiments.** The first experimental section tests RFormer on two synthetic datasets with which the task is to classify input time-series based on their ground-truth frequencies. While L233-234 mentions the second setup in particular is designed towards testing the long-range reasoning capabilities of RFormer, but it is unclear whether this indeed the case. For the second synthetic dataset, in particular, how can identifying the frequency be a proxy for long-range reasoning when the frequency $\omega_0$ is used for $t < t_0$ only? The results on Figure 2 showing that methods that are "tailor-made for long-range time series modeling" (L254) such as Neural-CDE and Neural-RDE underperforming significantly also indicates that the designed task is not really representative of long-range reasoning. \ \ More interesting questions to ask could be: What makes RFormer sample-efficient vs. vanilla Transformer particularly on the Sinusoidal dataset and not so much on the Long Sinusoidal dataset? What makes RFormer more robust to changes in sampling frequency compared to Neural-CDE and Neural-RDE? Table 1 of [A] shows Neural-CDE is also quite robust to dropped data, but is this characteristic not emergent for Sinusoidal and Long Sinusoidal datasets? - [W2] **Missing analysis on interpolation methods.** By default, RFormer uses piecewise-linear paths for computing path signatures, but as mentioned in L140, it seems any continuous-time interpolation can be used. As such, it would be interesting to discuss (1) whether any other interpolation techniques can be deployed efficiently similarly to piecewise-linear paths and (2) if they lead to any boosts in predictive accuracy, but these discussions are missing in the current draft (i.e., is the currently used piecewise-linear interpolation "pareto-optimal" under performance-efficiency trade-offs?). [A] Kidger et al., Neural Controlled Differential Equations for Irregular Time Series. NeurIPS 2020.

Questions

- [Q1] **Large overlaps in representation space.** Basec on the multi-view signature formulation in Figure 1 and Equation 8, it seems the earlier observations would be covered by a large number of input tokens. As shown in the right plots in Figure 3, this would result in large "representational overlap" similarly to the oversmoothing phenomenon in graph representation learning [B]. Considering this, could restricting the global view to a few previous points rather than all previous points be a viable option? or does the theoretical and empirical robustness of RFormer to variable lengths and irregular sampling require that all previous points be covered in the global view? - [Q2] **Discussion on input length of MLP and Transformer.** L108 states that the input length $L$ of the MLP and Transformer is fixed by assumption, but is this true? Sequences with different lengths can be processed in a single batch via padding. [B] Rusch et al., A Survey on Oversmoothing in Graph Neural Networks. arXiv 2023.

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations in Appendix B.