ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer

Summary

The paper introduces a masked encoder based transformer model for time series forecasting. It performs masking in the observation space, replacing the prediction horizon with 0s. It further applies an attention mask such that tokens only attend to the context, and do not attend to the masked tokens. The paper also uses a tunable version of rotary positional embeddings, and ensembles predictions over various patch sizes.

Strengths

The paper proposes a method which can adapt to multiple forecast horizons, which is a weakness of many recent deep forecasting models. The paper presents several empirical analyses to verify the claims made, and also presents a detailed look at related work.

Weaknesses

Overall, the proposed approach is very similar to Moirai, with several minor changes: i) Perform the masking operation in observation space ii) Add an attention mask iii) Tunable rotary embeddings iv) Ensembling results over multiple patch sizes However, it seems that even more features which Moirai had were removed, making it a weaker proposition. Such features include probabilistic predictions, multivariate capabilities, and any context length. Furthermore, few experiments are done to show the benefits of the changes, such as (i), (ii), and (iv). Others: 1. The statement, "we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages" (L238-L239) is a little eyebrow raising, in the sense that it is stating the obvious. I believe most researchers expect in-distribution predictions to outperform zero-shot predictions. Such statements should be rephrased. 2. Effects of tuning rotary embeddings in figure 4 seem very marginal, error margins should be provided for such cases.

Questions

None

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

Authors have provided limitations of their work.

Summary

This paper introduces ElasTST, which aimed at addressing the challenge of varied-horizon time-series forecasting. ElasTST integrates a non-autoregressive architecture with placeholders for forecasting horizons, structured self-attention masks, and a tunable rotary position embedding. Furthermore, it employs a multi-scale patch design to incorporate both fine-grained and coarse-grained temporal information, enhancing the model’s adaptability across different forecasting horizons. Experiments in the paper demonstrate the model's performance in providing robust predictions when extrapolating beyond the trained horizons.

Strengths

- The paper attempts to address an important issue in time series applications by enabling the model to adapt to varied forecasting horizons during the inference stage, which can make the trained forecasting model more flexible in usage. - Introducing multi-scale patching operations to construct input tokens for the Transformer can effectively handle local patterns of different granularities. - The paper introduces a tunable Rotary Position Embedding (RoPE) module tailored for time series tasks, showing potential for broader application across various forecasting methods.

Weaknesses

- The newly proposed tunable RoPE module lacks comparative analysis with previous methods. A comparison with methods such as no positional embedding, vanilla positional encoding, trainable positional embedding matrices, and original rotary positional embeddings, would clarify the benefits of different approaches. Since tunable RoPE appears easily transferable, demonstrating its performance enhancement by applying it to previous models like PatchTST and Autoformer would provide more intuition. - Using longer sequence length significantly increases the computational cost in Transformers. Comparing with baselines like PatchTST, the paper uses multiple patch sizes as input processing method, which significantly increases computational costs. Further analysis like how much adding a patch size can increase computational costs or what patch size combination are more reasonable to choose under resource constraints would be beneficial.

Questions

- Given the significant increase in computational cost from longer token sequences, would independently inputting each patch size into the transformer layers, instead of all at once, reduce memory usage while maintaining model performance? - With multiple patch sizes used in the study, how are the RoPE applied? Are they applied independently to tokens from each patch size, or collectively to all tokens from different patch sizes? - Ablation studies indicate that incorporating smaller patch sizes seems to benefit model performance substantially. If my understanding is correct, when the patch size is 1 (excluding tunable RoPE), the structure is similar to the decoder part of models like Informer and Autoformer. Would using a patch size of 1 worsen the model performance? Would adding "1_" to the existing best patch size setting "8_16_32" improve performance? Using patch size 1 turns the model into a channel-independent Informer-decoder-like model, which significantly increases computational costs, hence it you don't have to consider this on large datasets.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

N/A

Summary

The paper introduces ElasTST, a novel approach designed to enhance robustness in forecasting across various horizons. The proposed architecture dynamically adapts to different forecasting horizons, addressing a significant challenge in the literature. Previous methods typically relied on recursive approaches or utilized masks that failed to maintain horizon invariability. ElasTST aims to achieve high forecasting accuracy while ensuring adaptability across different horizons by employing a transformer encoder as its backbone. The approach incorporates structured attention masks through patching and rotary position embeddings (RoPE) to encode relative positional information for improved forecasting.

Strengths

1. The paper effectively tackles a crucial issue in the forecasting literature—enhancing deep learning architectures to produce forecasts for any horizon while maintaining horizon invariability and reducing the concatenation of errors. 2. The approach leverages well-established research, such as patching from the PatchTST paper, and also rotary position embeddings, to handle long-horizon forecasting tasks. 3. The paper considers recent advancements in foundation models for time series forecasting.

Weaknesses

1. The literature review on foundational models is lacking. Notable models such as TimeGPT-1 [1], Chronos [2], TinyTimeMixers [3], and Moment [4] are not discussed. 2. A traditional encoder transformer is not included as a baseline, which would help demonstrate the improvements offered by the proposed method. Additionally, NLP-based architectures such as NHITS [5] and TsMixer [6] are absent from the comparisons. 3. The paper does not adequately address the computational scalability of ElasTST when applied to very large datasets or extremely long time series. 4. The authors modified the traditional metrics used in long-horizon literature (MSE and MAE) to scaled metrics. They argue this adjustment reports fairer results, but it overlooks the fact that these datasets are typically already scaled, making additional scaling in the metric unnecessary. 5. The paper lacks a detailed discussion on potential failure cases or scenarios where ElasTST might underperform, which would provide a more balanced view of its applicability. [1] https://arxiv.org/abs/2310.03589 [2] https://arxiv.org/abs/2403.07815v1 [3] https://arxiv.org/abs/2401.03955 [4] https://arxiv.org/abs/2402.03885 [5] https://arxiv.org/abs/2201.12886 [6] https://arxiv.org/abs/2303.06053

Questions

1. What is the computational complexity added by the Rotary Position Embeddings (RoPE)? 2. In some cases, the results are very similar to PatchTST, such as with the ETTm2 dataset. Is there an intuition or explanation regarding the types of data that explain these similarities and differences in performance? 3. It would be interesting to compare the performance gains for each point in the forecasting window. Are the gains through the whole window or in the earliest/latest points?

Rating

5

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes