Are Language Models Actually Useful for Time Series Forecasting?

Summary

This paper questions the effectiveness of LLMs in time series forecasting. Through a series of ablation studies on 3 LLM-based forecasting methods, the authors find that removing or replacing the LLM component often improves forecasting results. They conclude that LLMs do not provide significant benefits for time series forecasting tasks, despite their significant computational cost. The paper suggests that simpler models or encoders with basic attention mechanisms perform equally well, challenging the current trend of employing LLMs for time series forecasting.

Strengths

- The paper offers an interesting perspective on the use of LLMs in time series forecasting, challenging prevalent assumptions and trends in the field. - The empirical analysis seems to be valid, with well-executed ablation studies and comparisons across multiple methods and datasets. - The paper is easy to read, with well-organized sections and effective use of figures and tables to present relevant results. - The findings may have implications for the research community, potentially redirecting efforts towards more efficient and effective methods for time series forecasting.

Weaknesses

- The scope is limited to time series forecasting and does not explore other potential applications of LLMs in time series analysis, such as imputation, classification or anomaly detection. - The datasets used are all evenly spaced time series, which may not fully represent the variety of real-world time series data, such as those with irregular intervals. - The paper could benefit from a more detailed discussion on the potential reasons why LLMs fail to outperform simpler models in this context, providing deeper theoretical insights.

Questions

Related to my last point in "weakness", can the authors provide more theoretical explanations or hypotheses on why LLMs do not perform well in time series forecasting compared to simpler models?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors acknowledge that the study focuses solely on time series forecasting and uses only evenly spaced datasets. Future research should explore LLMs’ effectiveness in other time series tasks and with non-uniform datasets. Additionally, potential negative societal impacts, such as the environmental cost of training large models, should be considered.

Summary

A recent surge of papers have popularised the usage of pre-trained LLMs for time series forecasting. The paper analyses the claim that LLMs are useful for time series forecasting, by performing a series of ablation studies. Their conclusion is that LLMs bring little to no benefit for the task, and are significantly more costly.

Strengths

1. The paper is very well written, the hypothesis is stated clearly and the experiments and results are also clearly stated. 2. Experimental approach is very sound, experiments are well thought out and very comprehensive. 3. The paper reveals very significant findings regarding a recent trend in time series forecasting, in which many papers at top conferences have been publishing about. It turns out that the performance of many of these methods is not due to a pre-trained LLM as touted by these papers, but due to other factors such as patching and channel independence is the most important.

Weaknesses

1. Experiments in section 4.3 are only presented for LLaTa, it should be presented for all the models. Importantly, the OneFitsAll paper showed that pre-trained weights performed much better than random initialized weights. Can the authors comment on this? 2. The experiments in 4.6 lead to a model that is very similar to PatchTST and some other similar patch-based Transformers, which the LLM-based models claim to be better than. Can the authors comment on this?

Questions

See weaknesses

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Authors have adequately addressed limitations in appendix.

Summary

This paper explores the effectiveness of language models in time series forecasting. The authors substantiate their claim by performing three straightforward ablations of three popular and recent LLM-based forecasting methods. After extensive experiments, the authors find that patching and attention structures perform similarly to state-of-the-art LLM-based forecasters.

Strengths

1. The paper is well-structured and easy to follow. 2. The author question existing LLM-based forecasting models by conducting comprehensive ablation studies and provide very interesting and insightful observations.

Weaknesses

1. Although the paper focuses on the effectiveness of LLM in TSF, existing state-of-the-art forecasting models have been omitted. 2. The authors conduct ablation studies on the patching and decomposition of LLM-based models to find out where the performance comes from. However, there is a lack of discussion on LLaTA, which embeds the input time series by treating each channel as a token instead of using patching.

Questions

1. Why the author choose ETTh1 and ILI datasets in RQ4? Results on other benchmark datasets are expected. 2. How does the author implement the "LTrsf" model, and what is the difference between "LTrsf" and the existing iTransformer model? 3. Can you compare the results between "PAttn" and PatchTST?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

yes

Summary

This paper presents an extensive empirical study on the effect of pre-trained LLMs in time series forecasting tasks. By ablating popular LLM adaptations on widely adopted time series benchmarks, experiments in the paper show LLMs do not benefit from pre-training on text data to gain improvement in forecasting performance. It suggests that the patching and first layer of attention makes most of contribution in forecasting setups.

Strengths

This paper is very well written and populated with extensive empirical evidences to support its main arguments. 1. The experiment setting covers various types of LLM adaptation to forecasting, including simple fine-tuning, PEFT and modality alignment, as well as popular time series forecasting benchmarks, which makes the conclusions fairly solid. 2. The reasoning behind the 6 research questions are convincing with rigorous ablation studies, and it covers multiple aspects of applicability of LLM in forecasting, from efficacy to efficiency. 3. Great reproducibility evidences are provided for an empirical study.

Weaknesses

A questionable piece of results is the random baseline in Table 5. It is counter-intuitive that, while the `woPre+woFT` variant is completely unfitted according to line 208-210, but its metrics are quite close to other variants in all experiments. Minor typos/errors: 1. line 76: One method ~this~ is

Questions

See weakness

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

Limitations are pointed out in the paper by the authors.