Library

AutoTimes: Autoregressive Time Series Forecasters via Large Language Models

Foundation models of time series have not been fully developed due to the limited availability of time series corpora and the underexploration of scalable pre-training. Based on the similar sequential formulation of time series and natural language, increasing research demonstrates the feasibility of leveraging large language models (LLM) for time series. Nevertheless, the inherent autoregressive property and decoder-only architecture of LLMs have not been fully considered, resulting in insufficient utilization of LLM abilities. To fully revitalize the general-purpose token transition and multi-step generation capability of large language models, we propose AutoTimes to repurpose LLMs as autoregressive time series forecasters, which projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. Compatible with any decoder-only LLMs, the consequent forecaster exhibits the flexibility of the lookback length and scalability with larger LLMs. Further, we formulate time series as prompts, extending the context for prediction beyond the lookback window, termed in-context forecasting. By introducing LLM-embedded textual timestamps, AutoTimes can utilize chronological information to align multivariate time series. Empirically, AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5\times$ training/inference speedup compared to advanced LLM-based forecasters. Code is available at this repository: https://github.com/thuml/AutoTimes.

cs.LGcs.CLComputer Science
Source

Paper

References (59)

01Are Language Models Actually Useful for Time Series Forecasting?Mingtian Tan, Mike A. Merrill, Vinayak Gupta et al.2024 · Neural Information Processing Systems · 274 citations In Library
02Unified Training of Universal Time Series Forecasting TransformersGerald Woo, Chenghao Liu, Akshat Kumar et al.2024 · International Conference on Machine Learning · 668 citations In Library
03UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series ForecastingXu Liu, Junfeng Hu, Yuan Li et al.2023 · The Web Conference · 230 citations In Library
04Large Language Models Are Zero-Shot Time Series ForecastersN. Gruver, Marc Finzi, Shikai Qiu et al.2023 · Neural Information Processing Systems · 801 citations In Library
05iTransformer: Inverted Transformers Are Effective for Time Series ForecastingYong Liu, Tengge Hu, Haoran Zhang et al.2023 · International Conference on Learning Representations · 2.1k citations In Library
06TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series ForecastingDefu Cao, Furong Jia, Sercan Ö. Arik et al.2023 · International Conference on Learning Representations · 287 citations In Library
07Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps DomainGerald Woo, Chenghao Liu, Akshat Kumar et al.2023 · arXiv.org · 25 citations In Library
08Time-LLM: Time Series Forecasting by Reprogramming Large Language ModelsMing Jin, Shiyu Wang, Lintao Ma et al.2023 · International Conference on Learning Representations · 1.1k citations In Library
09TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time SeriesChenxi Sun, Yaliang Li, Hongyan Li et al.2023 · International Conference on Learning Representations · 244 citations In Library
10Koopa: Learning Non-stationary Time Series Dynamics with Koopman PredictorsYong Liu, Chenyu Li, Jianmin Wang et al.2023 · Neural Information Processing Systems · 248 citations In Library
11Long-term Forecasting with TiDE: Time-series Dense EncoderAbhimanyu Das, Weihao Kong, Andrew B. Leach et al.2023 · Trans. Mach. Learn. Res. · 610 citations In Library
12Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu et al.2023 · Neural Information Processing Systems · 11k citations In Library

Scroll for more · 38 remaining

Similar papers

Time-LLM: Time Series Forecasting by Reprogramming Large Language ModelsMing Jin, Shiyu Wang, Lintao Ma et al.2023 · International Conference on Learning Representations · 978 citations In Library
SST-LLM: time series forecasting based on large language modelsYiyang Hu, Haibin Liao, Ming Wu et al.2025 · Other Conferences · 4 citations In Library
Time Series Forecasting with LLMs: Understanding and Enhancing Model CapabilitiesMingyu Jin, Hua Tang, Chong Zhang et al.2024 · SIGKDD Explorations · 96 citations In Library
LangTime: A Language-Guided Unified Model for Time Series Forecasting with Proximal Policy OptimizationWenzhe Niu, Zongxia Xie, Yanru Sun et al.2025 · International Conference on Machine Learning · 28 citations In Library
Semantic-Enhanced Time-Series Forecasting via Large Language ModelsHao Liu, Chun Yang, Xiaoxing Zhang et al.2025 · arXiv.org · 3 citations In Library
Adapting LLMs to Time Series Forecasting via Temporal Heterogeneity Modeling and Semantic AlignmentYanru Sun, Emadeldeen Eldele, Zongxia Xie et al.2025 · arXiv.org · 12 citations In Library
Large Language models for Time Series Analysis: Techniques, Applications, and ChallengesFeifei Shi, Xueyan Yin, Kang Wang et al.2025 · arXiv (Cornell University) · 5 citations In Library
Empowering Time Series Analysis with Large Language Models: A SurveyYushan Jiang, Zijie Pan, Xikun Zhang et al.2024 · International Joint Conference on Artificial Intelligence · 99 citations In Library
Logo-LLM: Local and Global Modeling with Large Language Models for Time Series ForecastingWenjie Ou, Zhishuo Zhao, Dongyue Guo et al.2025 · arXiv.org · 1 citation In Library
Context-Aware Probabilistic Modeling with LLM for Multimodal Time Series ForecastingYueyang Yao, Jiajun Li, Xingyuan Dai et al.2025 · arXiv.org · 1 citation In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer k2uw7/10 · confidence 4/52024-07-03

Summary

This paper argues that existing LLM based time series models have not fully exploited the inherent autoregressive property and the decoder-only architecture of LLMs. To address this problem, this paper introduces a novel AutoTimes model, which exploits the autoregressive property of LLMs. Experimental results demonstrate that AutoTimes could outperform the baseline methods.

Strengths

1. This paper exploits the autoregressive property of the LLMs. 2. Comprehensive experiments are conducted to demonstrate the effectiveness of the proposed AutoTimes, including forecasting, zero-shot forecasting and in-context forecasting. 3. This paper is well-written, and is easy-to-follow.

Weaknesses

1. Autoregression seems a little bit trivial. 2. It is not quite clear how much improvements can be brought by auto-regressively generating time series. Either direct comparison or theoretical analysis should be provided. 3. It is difficult to directly draw conclusions about the scaling behaviors from Table 5. The authors draw conclusions by comparing results of OPT-x models, rather than comparing different models e.g., compare OPT with LLaMA or GPT-2.

Questions

Please see the weaknesses.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Authorsrebuttal2024-08-12

Eagerly Await Your Response

Dear Reviewer k2uw, Thanks again for your valuable and constructive review. Would you please let us know if your concerns about autoregression effectiveness and scaling behavior are resolved? **Since the Author-Reviewer discussion period is ending in two days, we eagerly await your response.** Till now, we find that your rating is still "reject". In a very worried mood, we respectfully remind you that **we have comprehensively evaluated the improvement of autoregression and validated the scaling effect on different LLMs categories, which should help you better assess our work.** Also, **we have clarified the contributions of the method and paradigm renovation, and how they differ from previous approaches.** **We do hope you can consider our rebuttal and kindly let us know if our rebuttal has addressed your concern.** Sincere thanks for your dedication! We are looking forward to your reply. Authors

Authorsrebuttal2024-08-13

We Are Looking Forward to Your Reply

Dear Reviewer k2uw, Thank you again for your valuable and constructive review. We kindly remind you that there is **only a half day left** before the Author-Reviewer discussion period ends. We find that your rating is still "reject", so **we are eager to receive your response to our rebuttal**. We respectfully ask if you have any further concerns. Please let us know if you have other questions about our paper. We will be more than happy to engage in more discussions and improvements to the paper. Thank you sincerely for your dedication! We eagerly await your reply. Authors

Reviewer EAH77/10 · confidence 4/52024-07-10

Summary

The authors present AutoTimes, a method that utilizes large language models (LLMs) for time series forecasting. One of the key underexplored research topics addressed by the authors is the lack of models and pre-training mechanisms that result in foundation models capable of handling lookback and forecasting horizons of arbitrary length. This is achieved by adapting the LLM forecasting framework to autoregressively forecast time series segments. Furthermore, the paper outlines techniques such as in-context learning to further improve prediction performance. Compared to previous works, the methodology requires only a small number of trainable parameters compared to previous LLM fine-tuning techniques. As far as I am aware, this work is the first to be capable of handling multimodal input and producing an autoregressive forecast in the domain of LLMs and time series.

Strengths

- The paper is well-written and easy to follow. I appreciate how the authors clearly outlined their contributions. - The observation that non-autoregressive methods may contradict decoder-only LLMs and the shortcomings of prior methods is well-motivated. The solution proposed in the context of LLMs and time series is novel. - The introduction of a One-for-all benchmark, which involves prediction horizons transfer learning instead of dataset transfer learning, is innovative. - The authors provide strong motivation and experimental proof to support their claims. - The flexibility and scalability of their method are demonstrated by successfully swapping out different LLMs with varying numbers of parameters. - The continuous improvement over previous SOTA methods on multiple datasets, along with their thorough ablations, strengthens their claims and illustrates the method's flexibility.

Weaknesses

- I'm not a fan of your chosen color scheme. While I appreciate that you try to include colors for different entities, it is too vivid (jump from Figure 3 to 4 to 5), making the figures tough to read, especially when skimming through them quickly (which most first readers will do initially). It is not a major objection, but improving this aspect could enhance your manuscript. - Typo: - L21: etc [22, 42], missing dot. - The paper makes certain simplified assumptions, such as treating time series segments independently for embedding. This might overlook complex inter-dependencies present in real-world time series data. Addressing these inter-dependencies could enhance the robustness and applicability of the proposed method.

Questions

1. **Claim of missing data for foundation models:** I am aware that there is not a plethora of datasets available, but did you look into [1] (datasets) or [2] (models)? 2. While I see the autoregressive part as a great contribution, I am not entirely convinced that the proposed embedding scheme, which only fine-tunes a small portion, brings a competitive improvement. There are multiple works [4, 5] that do not require any fine-tuning of the LLMs. 3. L106: "Unlike previous works, our proposed method regards time series itself as the instructive prompt. It avoids the modality gap caused by concatenating time series and language tokens directly." This statement gives the impression that your approach is superior. I would be interested to see how your model compares to more of these learnable-prompt methods, especially how it compares to *Test* [4] performance-wise, which appears to be superior to Time-LLM. 4. How does AutoTimes handle missing data or irregular time series intervals? Your claimed improvement over [6], which does not use language directly, has the disadvantage that you cannot simply overcome NaNs in the time series directly. 5. "Unlike previous works, our proposed method regards time series itself as the instructive prompt. It avoids the modality gap caused by concatenating time series and language tokens directly." I am a bit confused. What kind of model checkpoint are you using? Instruction tuned? An instructive prompt is a directive where the user provides explicit instructions to guide the response. I thought you were skipping the language level. Could you outline how you embed your time series and text and how exactly this is fed into the model? --- [1] Goswami et al. "MOMENT: A Family of Open Time-series Foundation Models." ICML 2024. [2] Ekambaram et al. "Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series" arXiv:2401.03955 [3] Chang et al. "LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters" 2024 [4] Sun et al. "TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series" ICLR 2024 [5] Wang et al. "Xinglei Wang, Meng Fang, Zichao Zeng, and Tao Cheng. Where Would I Go Next? Large Language Models as Human Mobility Predictors" 2023 [6] Gruver, Nate, et al. “Large Language Models Are Zero-Shot Time Series Forecasters.” _Advances in Neural Information Processing Systems_, 2023

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

While the paper provides a limitations section, I don't think it is sufficient to bury it in the appendix. During my first read, I thought the authors entirely skipped the limitations until I found it in the checklist. Here are some specific points that I believe are missing or need improvement: 1. Although the paper claims that the embedding and projection layers only account for 0.1% of the total parameters, the scalability of these layers for very large datasets or extremely long time series is not thoroughly discussed. An evaluation of how the method performs with significantly larger datasets would strengthen the paper. 2. While the authors mention that they leave out real-world datasets for future work, I think the approach was not tested on datasets with missing values. This is especially important as their tokenization scheme seems to be based on non-overlapping patches, which are known to lose their locality assumption when missing values occur. An analysis of how the model handles missing data would provide a more comprehensive evaluation of its robustness and applicability. In addition to that, the authors answered broader societal impacts sufficiently.

Reviewer 1PWH7/10 · confidence 4/52024-07-12

Summary

This paper proposes a model named AutoTimes to repurpose LLMs for time series forecasting. Different from previous methods that use flattening and linear projection to get a prediction, this model repurposes LLMs in an autoregressive way, which is closer to the pre-training process of LLMs. Specifically, the main backbone of LLMs is frozen, and new patch embedding/output layers are added like in previous works. Absolute timestamps are embedded through LLMs to serve as position embeddings. Experiments show that the proposed model achieves SOTA performance and is more efficient.

Strengths

1. The main body of the thesis is clear and easy to understand. 2. Reproposing LLMs in an autoregressive way is intuitive and more reasonable than previous linear projection ones. 3. Numerous experiments were conducted to demonstrate the effectiveness of the proposed method.

Weaknesses

1. The most I concerned is whether this type of methods truly leverages the capabilities of the pre-trained LLMs. Please conduct the following ablation study: randomly initialize a large model, freeze its parameters, train it using the proposed method, and compare it with the pre-trained ones. 2. The description of multimodal in Table 1 seems to be overselling, as the proposed model only uses texts for timestamps embedding and does not have capability to leverage natural languages. 3. The in-context forecasting part is confusing: 1)What are the use cases for such a method? Table 16 shows that it's more effective to extend the lookback window. So what is the case that we can not get a longer lookback window but can get a window a long time ago? 2)Such a method can not even ensure the input series is continuous, so where does the promotion come from?

Questions

1. Is LLM necessary for timestamp embedding? What if replacing it with ordinary nn.Embedding like Informer? 2. How does the proposed model reduce the error accumulation of autoregressive models, given that there is no specifical design for this? 3. Please report the number of learnable parameters in Table 5. Larger models have larger hidden states so that patch embedding/output layers for them have more learnable parameters. Therefore, it is uncertain whether the performance improvement comes from the scaling behavior of LLMs or from having more learnable parameters for tuning.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

No potential negative societal impact.

Reviewer acJd4/10 · confidence 4/52024-07-15

Summary

The authors present in this paper an interesting approach where LLMs are leveraged to be fledged as time series forecasters. This proposed approach is based on freezing the LLM backbone to update a small amount of parameters to generate suitable time series embeddings which, together with time stamps as positional embeddings, provide suitable forecasts. Moreover, the authors provide an interesting notion of in-context forecasting, where the corresponding model is taught with examples how to forecast without gradient updates. Finally, the authors present numerical evaluations on diverse datasets.

Strengths

The authors provide an interesting view and current challenge in the field of time series forecasting: how to leverage LLMs for time series forecasting with minimal effort and minimal model modification. The authors provide an approach that basically freezes the LLM backbone to only update parameters related to a suitable embedding of time series, together with the usage of timestamps prompts as positional encoders. Moreover, the emphasise that the number of updated parameters is around 0.1% of those of the LLM backbone. The authors provide an interesting paradigm that naturally arises from LLMs: how can we do forecaster fine-tuning without gradient updates. This brings in-context forecasting as a contribution of the authors, which seems to have potentially an interesting resourceful idea for the community.

Weaknesses

Perhaps the main weakness comes from the evaluations. The main points that I would invite the authors to aggressively address are the following: - the amount of baselines used for comparison is very limited. It is completely understandable that the authors miss some baselines because the field is just moving extremely quickly. Nevertheless, the authors should at least cite a relevant fraction of these recent works, for instance : - - Woo et al, 2024: Unified Training of Universal Time Series Forecasting Transformers - - Ansari et al, 2024: Chronos: Learning the Language of Time Series - - Dooly et al, 2023: ForecastPFN: Synthetically-Trained Zero-Shot Forecasting - - Goswami et al 2024: Moment: A family of open time-series foundation models - The amount of datasets used is limited. It is understandable that the authors provide a limited amount of datasets for evaluations, as one of the main hindrances in the field is the lack of publicly available data (at least in comparison to other fields). Yet, one can see that other contributions have shared datasets and made them freely available. See for instance: - - Woo et al, 2024: Unified Training of Universal Time Series Forecasting Transformers - - - and the corresponding HF source: https://huggingface.co/datasets/Salesforce/lotsa_data - Evaluations of in-context forecasting are limited. Following the previous points, I think this is one of the most exciting points of the paper and the reader would appreciate to see more systematic evaluations and explorations of this notion. Currently there is only one evaluation which involves M3 and M4 datasets, perhaps following the approach presented in One-Fits-All paper.

Questions

- In Section 3 the authors mention that they aim to forecast covariates as well. Is this correct? In general, one can assume that certain kind of covariates are available in the future, like timestamps or boolean variables indicating that something specific will happen in the future. But in general, I am not sure that the authors truly want to forecast covariates.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

The authors have devoted a section describing sensible limitations of their work.

Authorsrebuttal2024-08-12

Request of Reviewer’s Attention and Feedback

Thank you for your feedback, but we respectively disagree that the discussion would correspond to almost a major revision of the paper: **1. The original baselines compared in our paper are the most up-to-date and advanced deep-learning approaches**. * For LLM4TS methods, we compared AutoTimes with the state-of-the-art models: TimeLLM (ICLR 2024, cite 157), and FPT (NeurIPS 2023 Spotlight, cite 147). * For deep time series forecasters, we compared with the most prevalent ones, covering various architectures with state-of-the-art performance: iTransformer (ICLR 2024 Spotlight, cite 189), DLinear (AAAI Oral, cite 978), PatchTST (ICLR 2023, cite 615), TimesNet (ICLR 2023, cite 466). **2. Our method is evaluated on diverse tasks and extensively analyzed in the original submission**. * Our evaluations include long-term time series forecasting, short-term time series forecasting, zero-shot time series forecasting, and in-context time series forecasting, covering 10 datasets beyond those in previous LLM4TS methods. * Our analysis covers method generality, scaling behavior of LLMs, method efficiency, and hyperparameter sensitivity, which are hardly explored in previous LLM4TS approaches. * Our ablations cover almost all components in the proposed method: textual timestamp embedding (Appendix D.5), the LLM backbone ( LoRA Adaptation in Section 4.4), and autoregressive forecasting (Figure 7 and Table 9). Even if it is indispensable to include the suggested evaluations (datasets of LoTSA, comparison with concurrent Large Time Series Models, and exploration of our proposed in-context forecasting) that have not been included in any previous works in the LLM4TS direction, **these experiments can be easily added by lines in existing tables and incorporate them in the Appendix, such that the work do not need a major revision**. We believe that such adjustments can enhance the integrity of the paper without too much impact on the overall structure. We hope you will reconsider this point. Thank you for your understanding and support.

Reviewer EAH72024-08-08

Confusion

I thank the authors for their thorough answers. I have to admit that I am, unfortunately, a bit confused. Would you mind double-checking your response and ensuring a consistent numbering of my questions? I asked five questions, but you referred to Q6, likely because you enumerated my outlined weaknesses as questions. Additionally, you seem to have combined Questions 1 and 2 into Q3; however, you appear to have only answered Question 2: "Based on the mentioned works, we also provide an evaluation of AutoTimes on larger datasets (Time-Series Pile[1])." Where did you do that? Before I attempt to decipher and remap the questions to their original form, I could be mistaken. To avoid any wrong conclusions, I kindly ask the authors to restructure it for me.

Authorsrebuttal2024-08-08

Restructed Response (Part 1)

Thank you for your thoughtful feedback and for bringing these points to our attention. We apologize for any confusion caused by the numbering and organization of our responses. Based on the original context of the rebuttal, the response is restructured as follows: **W1 & W2**: Suggestions for improving the presentation of the paper. Thanks for your valuable feedback regarding the color scheme and the mentioned typo. We will use a more subdued scheme and fix the typo in the revision. **W3**: Address inter-dependencies in real-world time series data. We acknowledge that real-world multivariate time series often exhibit complex inter-dependencies that can significantly impact analysis. In terms of that, AutoTimes adopts Channel Independence like previous methods and further uses the timestamp as the position embedding to implicitly align different variates. As you insightfully point out, it is necessary to explore the complex inter-dependencies, which is a hot topic in current deep time series models. It is also an essential problem for LLM4TS methods since the gap between natural language (1-D discrete token sequence) and time series (multi-dimensional continuous sequence) poses increasing challenges for LLMs to explicitly utilize the relationship between sequences. We will explore several potential approaches: **integrating textual descriptions of variates** and **employing adaptors for variate correlating**. Your suggestion will guide us in refining our methodology. **Q1**: Claim of missing data for foundation models. Thanks for your mentioned works, we are excited to see recent works advancing the development of datasets and pre-trained large models in the field of time series. We will cite them in the related work section and polish your mentioned claim. **Q2**: About the effectiveness of the proposed embedding scheme. We appreciate your concerns about the effectiveness of our approach, especially in light of several works that use LLMs for time series without fine-tuning. The mentioned works indeed provide an out-of-the-box experience that is free from fine-tuning. To boost the performance further, AutoTimes intends to **leverage the general token transition of LLMs while tailoring it to the specific characteristics of the dataset**, which is achieved by freezing the LLM backbone (keeping the token transition) and training new embedding layers (learning the dataset-dependent embeddings of time series). Please also refer to the detailed code and scripts provided in the $\underline{\text{supplementary material}}$, by which we ensure all the results of the paper are reproducible. **Q3**: Comparison with the performance of TEST[1]. We compare TEST performance-wise. The averaged results among four prediction lengths {96, 192, 336, 720} are reported. AutoTimes achieves better performance on the majority of datasets. | Datasets (MSE\|MAE) | AutoTimes | TEST | | ------------------- | -------------------- | -------------------- | | ETTh1 | **0.389**\|**0.422** | 0.414\|0.431 | | Weather | 0.235\|0.273 | **0.229**\|**0.271** | | ECL | **0.159**\|**0.253** | 0.162\|0.254 | | Traffic | **0.374**\|**0.264** | 0.430\|0.295 |

Authorsrebuttal2024-08-08

Restructed Response (Part 2)

**Q4**: How does AutoTimes handle missing data or irregular time series intervals? Thank you for your insightful question. At this stage, AutoTimes does not specifically address missing data or irregular intervals. It is the same with current works focusing on regular forecasting scenarios where time series are complete and consistently sampled. We will add this as a limitation and conduct evaluations on well-acknowledged datasets in the revision. **Q5**: How does AutoTime embed and feed time series and texts? The claim: "our proposed method regards time series itself as the instructive prompt..." refers to the following: * As depicted in $\underline{\text{Figure 1(b)}}$, previous LLM4TS methods feed (language tokens | lookback time series) to handle multimodal input. * As depicted in $\underline{\text{Figure 3}}$, AutoTimes feed (time series prompt | lookback time series) to enable in-context forecasting, where time series is self-prompted. * As shown in $\underline{\text{Equation 5}}$, AutoTimes uses textual timestamps as position embeddings and adds them with the corresponding series embedding of each token. Therefore, AutoTimes presents **prompting time series in two directions**. Horizontally, AutoTimes append the time series prompt at the front of the lookback series and regards it as the task demonstration. Vertically (series embedding + timestamp embedding), merging token-wise embeddings can use timestamps in natural language and align multiple variates. **L1**: An evaluation of how the method performs with significantly larger datasets. Based on your suggested works[2], we provide an evaluation of AutoTimes on larger datasets (Time-Series Pile[2]) to address your concern about the scalability of the trainable layers. Please note that **Time-Series Pile is an aggregation of time series datasets, which also includes ETTh1, Weather, ECL, and Traffic as the subsets**. We first train the LLM-based forecaster by AutoTimes on Time-Series Pile, and report the performance on the above well-acknowledged subsets respectively. Therefore, the experiment evaluates the capacity of our embedding/projection layers to accommodate diverse time series. | Performance of Subset (MSE\|MAE) | nn.Linear | 3-layer MLP | | -------------------------------- | ------------ | -------------------- | | ETTh1 | 0.724\|0.586 | **0.363**\|**0.395** | | Weather | 0.288\|0.335 | **0.166**\|**0.211** | | ECL | 0.856\|0.764 | **0.135**\|**0.231** | | Traffic | 1.393\|0.799 | **0.351**\|**0.247** | As we delve into layer scalability, the essence of designing the embedding scheme to address larger time series datasets is highlighted, such as increasing more layers and using MoE modules to increase the data generality, which provides great guidance for our future research. **L2**: An analysis of how the model handles missing data. We appreciate your highlighting the importance of this issue, especially about our tokenization scheme based on non-overlapping patches. Currently, our research has not included testing on datasets with missing values, as our focus has been on repurposing LLMs on regular time series. According to your suggestions, we will consider moving the limitation to a more prominent position within the main body of the paper to ensure that readers can easily access and engage with this critical information. [1] Sun et al. TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series. [2] Goswami et al. Moment: A Family of Open Time-Series Foundation Models. We appreciate your patience as we work to restructure our responses for clarity. Your feedback is invaluable in helping us improve the communication. We'd be very happy to answer any further questions.

Reviewer EAH72024-08-10

I thank the authors again for their response and willingness to restructure their answer. I have read the other reviews and your response, and I am mostly satisfied. Regarding Q4, while I acknowledge that the authors left it out, I still believe that, given the nature of this work, it could provide an interesting insight (at least the missing data part). For Q5: Thank you for clarifying this. Based on your response, it might be a good idea to make this clearer in the manuscript, as it wasn't immediately clear to me at first glance. However, with the newly conducted experiments and the forthcoming manuscript revisions, I believe this work could have a great impact on at least one subfield of time series prediction. As I have no further concerns, I will maintain my initial score. Great work!

Authorsrebuttal2024-08-10

Thanks for Your Response

Thank you for your prompt response and valuable comments on our paper, which have been of great help to us. Here is a brief revision plan based on your suggestions: (1) We understand your perspective on the insights that our method could gain from delving into the missing data. We will conduct evaluations to enhance the depth of our paper. (2) We will elaborate on the pipeline of time series and text embedding and make it clearer to improve readability for future readers. Thanks again for your response and support for our work. We promise to follow your suggestion to improve the quality of our paper.

Authorsrebuttal2024-08-10

Complete Evaluation Results

Dear Reviewer acJd: We sincerely appreciate your insightful pre-rebuttal review, which has inspired us to improve our paper further substantially. According to your suggestions, we have made every effort to complete the evaluations, including more baseline models, benchmark datasets, and evaluations/explorations on in-context forecasting. Experimentally, we verify that our method can achieve the best performance on new baselines and benchmarks, an in-context forecasting paradigm deserves a meaningful notion for real-world forecasting. **Due to the word limit of the rebuttal, we provide the complete results of the previous questions here**: **1. More baseline models for evaluation.** | ETTh1 (MSE\|MAE) | AutoTimes | MOIRIA | MOMENT | Chronos | | - | - | - | - | - | | Pred-96 | **0.360**\|**0.400** | 0.384\|0.402 | 0.387\|0.410 | 0.571\|0.464 | | Pred-192 | **0.388**\|**0.419** | 0.425\|0.429 | 0.410\|0.426 | 0.654\|0.504 | | Pred-336 | **0.401**\|**0.429** | 0.456\|0.450 | 0.422\|0.437 | 0.712\|0.530 | | Pred-720 | **0.406**\| **0.440** | 0.470\|0.473 | 0.454\|0.472 | 0.774\|0.570 | | Average | **0.389**\|**0.422** | 0.434\|0.439 | 0.418\|0.436 | 0.678\|0.517 | | ECL (MSE\|MAE) | AutoTimes | MOIRIA | MOMENT | Chronos | | - | - | - | - | - | | Pred-96 | **0.129**\|**0.225** | 0.158\|0.248 | 0.136\|0.233 | - | | Pred-192 | **0.147**\|**0.241** | 0.174\|0.263 | 0.152\|0.247 | - | | Pred-336 | **0.162**\|**0.258** | 0.191\|0.278 | 0.167\|0.264 | - | | Pred-720 | **0.199**\|**0.288** | 0.229\|0.307 | 0.205\|0.295 | - | | Average | **0.159**\|**0.253** | 0.188\|0.274 | 0.165\|0.260 | - | | Traffic (MSE\|MAE) | AutoTimes | MOIRIA | MOMENT | Chronos | | - | - | - | - | - | | Pred-96 | **0.343**\| **0.248** | - | 0.391\|0.282 | 0.770\|0.552 | | Pred-192 | **0.362**\|**0.257** | - | 0.404\|0.287 | OOM | | Pred-336 | **0.379**\| **0.266** | - | 0.414\|0.292 | OOM | | Pred-720 | **0.413**\| **0.284** | - | 0.450\|0.310 | OOM | | Average | **0.374**\|**0.264** | - | 0.415 \|0.293 | OOM | **2. More benchmark datasets for evaluation.** | Australian Electricity Demand (MSE\|MAE) | AutoTimes | PatchTST | iTransformer | DLinear | | - | - | - | - | - | | Pred-96 | **0.150**\|**0.228** | 0.163\|0.242 | 0.153\|0.233 | 0.167\|0.250 | | Pred-192 | **0.203**\|**0.268** | 0.216\|0.284 | 0.214\|0.270 | 0.211\|0.283 | | Pred-336 | **0.236**\|**0.293** | 0.255\|0.312 | 0.244\|0.295 | 0.237\|0.302 | | Pred-720 | **0.264**\|**0.315** | 0.289\|0.343 | 0.267\|0.318 | 0.269\|0.332 | | Average | **0.213**\|**0.276** | 0.231\|0.295 | 0.220\|0.279 | 0.221\|0.292 | | Bdg-2 Panther (MSE\|MAE) | AutoTimes | PatchTST | iTransformer | DLinear | | - | - | - | - | - | | Pred-96 | **0.537**\|**0.458** | 0.565\|0.476 | 0.546\|0.462 | 0.581\|0.499 | | Pred-192 | **0.663**\|**0.511** | 0.707\|0.543 | 0.694\|0.524 | 0.693\|0.547 | | Pred-336 | **0.741**\|**0.544** | 0.807\|0.584 | 0.774\|0.564 | 0.781\|0.584 | | Pred-720 | **0.802**\|**0.575** | 0.911\|0.649 | 0.832\|0.597 | 0.829\|0.615 | | Average | **0.686**\|**0.522** | 0.748\|0.563 | 0.712\|0.537 | 0.721\|0.561 | | Oikolab Weather (MSE\|MAE) | AutoTimes | PatchTST | iTransformer | DLinear | | - | - | - | - | - | | Pred-96 | **0.603**\|**0.577** | 0.635\|0.603 | 0.630\|0.591 | 0.663\|0.611 | | Pred-192 | **0.643**\|**0.602** | 0.678\|0.630 | 0.660\|0.609 | 0.694\|0.633 | | Pred-336 | **0.666**\|**0.615** | 0.685\|0.634 | 0.677\|0.620 | 0.711\|0.643 | | Pred-720 | **0.697**\|**0.632** | 0.710\|0.647 | 0.698\|0.633 | 0.727\|0.654 | | Average | **0.652**\|**0.607** | 0.677\|0.629 | 0.667\|0.613 | 0.699\|0.635 | **3. Systematic evaluations of in-context forecasting.** | Dataset (MSE\|MAE) | In-Context(Prompt-384 + Input-288) | Zero-Shot (Input-288) | Zero-Shot(Input-672) | | - | - | - | - | | ETTh1-OT | **0.0645**\|**0.1951** | 0.0673\|0.1996 | 0.0657 \|0.1969 | | ETTh2-OT | **0.1513**\|**0.3009** | 0.1637\|0.3133 | 0.1538 \|0.3026 | | ETTm1-OT | **0.0399**\|**0.1512** | 0.0424\|0.1567 | 0.0415 \|0.1534 | | ETTm2-OT | **0.1629**\|**0.3143** | 0.1669\|0.3137 | 0.1701 \|0.3197 | | Dataset (MSE\|MAE) | Ahead-Period | Ahead-Random | Fixed Prompt | Other-Variates | Baseline (Zero-Shot) | | - | - | - | - | - | - | | ETTh1-OT | **0.0645**\|**0.1951** | 0.0666\|0.1988 | 0.0769\|0.2109 | 0.1263\|0.2796 | 0.0657 \|0.1969 | | ETTh2-OT | **0.1513**\|**0.3009** | 0.1621\|0.3141 | 0.1859\|0.3346 | 0.1780\|0.3338 | 0.1538 \|0.3026 | | ETTm1-OT | **0.0399**\|**0.1512** | 0.0407\|0.1529 | 0.0512\|0.1733 | 0.0852\|0.2284 | 0.0415 \|0.1534 | | ETTm2-OT | **0.1629**\|**0.3143** | 0.1719\|0.3216 | 0.2104\|0.3649 | 0.2297\|0.3738 | 0.1701 \|0.3197 | Given the limited timeframe for author-reviewer discussion, please kindly let us know if our response has addressed your concerns. Your feedback is invaluable in helping us improve the communication. We'd be very happy to answer any further questions. All the best, Authors

Reviewer acJd2024-08-12

I would like to thank the authors for putting this significant amount of effort in such a small amount of time. Given that LOTSA is a fairly large collection of time series datasets, the authors are presenting results on a subset of these datasets. For instance, some of them belong to the monash repository. It is not expected to conduct experiments in all datasets. Yet, I would like to ask the authors: what was the criteria to choose these datasets? was it limitation on time execution, memory, compute in general?

Authorsrebuttal2024-08-12

Additional Response to Reviewer acJd

Thank you for your prompt response and for acknowledging the effort we put into this work. We appreciate your interest in our dataset selection in the rebuttal. The criteria for choosing the datasets from the LOTSA collection are based on the size of the datasets. Given the limited time for rebuttal, we are only able to cover partial datasets in LOTSA, since the overall time points of this collection are up to 27 billion. We aimed to select datasets that would provide the diversity of evaluation while ensuring that our experiments remained manageable within the given timeframe. Concretely, the time points of the selected datasets range from 0.5~2 million, which is comparable to existing datasets. If the evaluation is favored, we will be glad to continue to run evaluations on larger datasets and include them in the final revision. We hope this clarifies our approach, and we appreciate your understanding of the limitations inherent in working with such a large collection and other experimental analyses. Thank you again for your thoughtful feedback!

Reviewer acJd2024-08-12

Thanks for this clarifying answer. I will increase my score to the next rating value, i.e. 4: Borderline Reject. The main reason to provide this score is that on one side I believe the idea is interesting and it shows a path of interest in the field of time series. On the other side, the amount of experiments and following results and discussions would correspond to almost a major revision of the paper.

Reviewer 1PWH2024-08-11

Thanks for your response. The rebuttal addressed my concerns about the role of LLM and timestamp embedding, as well as the practical application of in-context forecasting. Therefore, I decided to raise my score to 7 and recommend this work to be accepted. Moreover, I hope the authors can include these experimental results for rebuttal in the final version to make the work more comprehensive, especially for the Q6 Table regarding scaling behavior.

Authorsrebuttal2024-08-11

Thanks for Your Response and Raising the Score

Thank you for your positive feedback and for raising the score to 7. We are glad to hear that our rebuttal addressed your concerns regarding the true ultilization of LLMs in our method, effectiveness of timestamp embeddings, and the practical application of in-context forecasting. We appreciate your suggestion to include the experimental results in the final version, particularly for the Q6 Table on scaling behavior. We will ensure that these results are incorporated to enhance the comprehensiveness of our work. Thank you once again for your support and constructive feedback. We look forward to finalizing the manuscript.

Reviewer k2uw2024-08-13

Thanks for the response. After reading all the comments and responses, my concerns are mostly addressed. I'm happy to raise my scores.

Authorsrebuttal2024-08-14

Thanks for Your Response and Raising the Score

Thanks again for your response and raising the score to 7! In the final version, we will elaborate more on clarifying contributions and novelty, and include the additional experiments to the paper.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC