Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series

Summary

It presents a lightweight pretrained model for TS with strong performance.

Strengths

Overall, this is a strong paper with extensive experimental work.

Weaknesses

Adaptive patching is a well-conceived design, although I am unclear why it is termed "adaptive" when the design appears to be fixed and pre-set. Perhaps "multiscale patching" would be a more accurate descriptor. Additionally, the writing should more clearly distinguish between the designs used for pretraining and those used solely for finetuning. For the full-shot head probing, it is important to include comparisons with state-of-the-art end-to-end methods, as readers are likely interested in such comparisons.

Questions

In the weakness

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

not applicable

Summary

This paper introduces Tiny Time Mixers (TTM), a new compact pre-trained model for efficient zero/few-shot multivariate time series forecasting. TTM is based on the lightweight TSMixer [1] architecture and incorporates several innovations, which enable effective pre-training on varied dataset resolutions with minimal model capacity. Besides, TTM also employs multi-level modeling to capture channel correlations and fuses exogenous data into the forecasting process during fine-tuning. The authors comprehensively evaluate TTM on multiple datasets and compare their performance with other state-of-the-art models. The results highlight TTMs' superior accuracy, computational efficiency, and lightweight nature.

Strengths

1. **Enhanced Zero/Few-shot Forecasting Performance**: TTMs demonstrate a substantial improvement in zero/few-shot forecasting capabilities, outperforming existing benchmarks by 4-40%. This advancement is particularly notable as it shows that smaller models can achieve high accuracy without the extensive computational resources typically required by larger models. 2. **Specialized Techniques for Pre-training and Fine-tuning**: The paper presents novel techniques such as adaptive patching, diverse resolution sampling, and resolution prefix tuning. These innovations enable robust pre-training on heterogeneous datasets and facilitate effective fine-tuning, allowing TTMs to capture channel correlations and integrate exogenous signals, which is critical for accurate multivariate forecasting. 3. **Innovation in Model Architecture**: The paper introduces Tiny Time Mixers (TTMs), a compact pre-trained model architecture tailored for multivariate time series forecasting. Starting from just 1 million parameters, TTMs offer a lightweight and efficient alternative to larger, more computationally intensive models. This addresses the need for fast and resource-friendly forecasting tools.

Weaknesses

1. The base model TSMixer used for pre-training is not newly proposed. 2. TTM necessitates to train different models for different context length and forecast length, posing some new weaknesses compared to Transformer-based pre-training models. While this paper presents forecast length adaptation strategy to handle the fixed forecast length problem, performance loss still emerges when there is a disparity between the model's forecast length and the actual forecast length. 3. There are not sufficient experiments to prove the scaling law on pre-training models. The paper's results show that the TTM performs best with 5M parameters. Will the TTM perform better with a larger number of parameters? Showing the tradeoff between performance and cost along with parameters size increasing will facilitate understanding of the model and help choose the most suitable model size. 4. Some parts of TTM lack ablation studies to sufficiently prove their effectiveness, such as decomposing the exogenous mixer module and decoder channel-mixing.

Questions

see weakness

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

see weakness

Summary

The paper introduces Tiny Time Mixers (TTMs), a series of pre-trained models designed for efficient zero/few-shot multivariate time series forecasting. TTMs are built on the lightweight TSMixer architecture and incorporate innovations such as adaptive patching, diverse resolution sampling, and resolution prefix tuning to handle varied dataset resolutions with minimal model capacity. These models outperform existing benchmarks in accuracy while significantly reducing computational requirements. The empirical studies demonstrate superior performance across multiple tasks and datasets.

Strengths

1. The paper introduces innovative solutions to overcome the limitations of large pre-trained models in time series forecasting, offering techniques that could potentially be adapted to enhance other time series forecasting models as well. 2. The empirical studies are robust, thoroughly assessing the model's accuracy and efficiency across multiple benchmark datasets. 3. The experimental results are impressive, demonstrating enhanced accuracy and substantially lower computational demands compared to existing methods. 4. The ablation studies are thorough, providing a detailed analysis of the impact of different pre-training datasets and the effectiveness of the proposed training techniques.

Weaknesses

I did not see significant weaknesses that need to be addressed in this paper. As a minor suggestion, given that there is another mixer architecture known as TSMixer[1], it would be beneficial to include a footnote or a mention in the appendix to clarify that the TSMixer referenced in this work differs from the other one. The clarification will help avoid potential confusion and ensure the distinctiveness of the models is clearly understood. [1] Chen, Si-An, et al. "Tsmixer: An all-mlp architecture for time series forecasting." arXiv preprint arXiv:2303.06053 (2023).

Questions

1. Given that the pre-training datasets may include time series with varying input lengths, how do TTMs manage these differences during training? Additionally, how do TTMs handle time series with input lengths not encountered before during inference? 2. Considering that resolution prefix embeddings are learned discretely, how do TTMs accommodate resolutions that were not seen during training when encountered during inference? 3. What criteria were used to determine the patch size for lagged exogenous features in the TTMs? 4. The training and evaluation of TTMs primarily focus on long time series with hundreds of input steps. Does this focus impact the performance of TTMs when applied to shorter time series, such as those with fewer than 30 time steps?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

The limitations regarding the restricted number of downstream tasks are discussed in the appendix.

Summary

This paper proposes a novel time-series pre-trained model TTM that instead of trying to over-parameterize the model, TTM tries to under-parameterize the model for better generalization ability. The model architecture is simple, straightforward, and easy to understand, coming along with advanced training strategies, such as adaptive patching, diverse resolution sampling, etc. Empirical evaluations show the benefit of TTM in both state-of-the-art forecasting performance with other supervised, zero-shot, and pre-trained methods, as well as super-efficiency in training.

Strengths

1. The idea of using an under-parameterized pre-trained model (1M parameters in TTM compared to millions or billions of parameters in other pre-trained time-series models) for better generalization ability seems to be novel and promising. 2. The training strategies are clearly presented, the evaluations are comprehensive, and the results are impressive.

Weaknesses

1. While the idea of using much fewer model parameters can work as presented in this paper, it is counter-intuitive as the current popular pre-trained models tend to be much larger than the proposed TTM. Beyond the empirical results, the paper does not go into deep intuitions of why this manner can work well in generalization, making the idea less convincing. 2. Presentation may be improved, currently the presentation of figures and tables is too crowded.

Questions

1. Following the point about weakness, could the author intuitively explain why such few parameters can work well for time-series forecasting tasks? 2. As the model is under-parameterized, my worry about TTM is that TTM can be a model of garbage in and garbage out (take it easy, I am saying this is bad, I am just very curious). Here is my concern: Let us make a comparison here, the proposed TTM is like a linear regression, very simple, while other over-parameterized models are like a polynomial/kernel regression. We all know that linear regression has less capability than polynomial regression for interpretations. However, for extrapolations, if the extrapolation data still follows the distribution of interpolation, then we can still expect polynomials to work better than linear. However, the reality is that extrapolation data does not follow the distribution of interpolation for most cases in time series (non-stationary problem), so in this case, both linear and polynomials can make mistakes. However because linear is less 'aggressive' in predicting trend changes due to the capability limit itself, it makes less wrong predictions than polynomials in many cases. That is, given the fact that the datasets used in this work, such as ETT, are highly non-stationary, i.e., data distribution can change heavily as time goes. I might want to see two things: First, I want to see whether TTM can still show trend information in long-term forecasting. I wonder if the outperformance is because TTM is making 'conservative' and simple predictions that sacrifice the trend information because of the under-parameterization than other pre-trained models. Second, I want to see some evaluations on some very easy stationary signals, such as sin/cos wave, and see that TTM can still work better than other pre-trained models. 3. Does the training process involve using any synthetic data, such as simple sin/cos waves, which can be common practices in training other pre-trained models?

Rating

7

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

Please refer to the Weaknesses and Questions