TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series

Summary

The authors propose an interesting way to do time series forecasting and classification with pretrained Large Language Models (LLMs). Whereas LLMs are designed to be used via prompts, the authors propose to find time series embeddings that are introduced in the model instead of finding suitable prompts. In this way, by considering an end-to-end approach, where the embeddings are adjusted according to the given dataset keeping the LLM weights frozen, the authors show that the proposed methodology does present a competitive performance. In particular, the larger the model, i.e. LLaMa-13B, the better the performance, potentially outperforming SOTA models like transformers. Finally, the authors present a brief analysis on the embeddings learn by the model, which suggests that these are mainly composed by adjectives/sentiments.

Strengths

The authors motivate the proposed approach by clarifying that there are basically two ways to extend LLMs to time series: either by 1/creating a LM from a large enough time series corpus, which would imply an non-negligible amount of computational resources, or 2/ adjusting time series so that one can leverage available LLMs. Given that the later approach is more accessible to practitioners, the authors then start providing a way to embed time series to leverage existing LLMs for time series. The strongest point of the paper is the notion of introducing time series embeddings to LLMs rather than finding the right prompt for it. Interestingly, the authors consider that by freezeing the weights of the LLM one can obtain embeddings that are leading to a competitive performance in terms of forecasting and classification. Perhaps the most interesting point would be to understand the kind of embeddings that are obtained from the proposed methodology.

Weaknesses

1. Given that the proposed approach relies on contrastive learning, I would suggest the authors to provide a more self-contained and better written text when it comes to contrastive learning. Currently the paper assumes a non-trivial amount of familiarity to the topic and makes it difficult to follow. For instance, what does the following term mean: _jitter-and scale strategy_? 2. There is no code included. This is relevant as I wonder if one can reproduce the experiments given the current writing quality of the paper. 3. There are no evaluations for zero-shot evaluations. According to the motivation of the paper, it might be relevant as well to consider the case where the time series embedding is developed in a particular dataset, and then tested in another one. Although this might not be necessarily too far from the few-shot evaluation already presented. 4. There is no analysis on time execution. The authors do not report how long it takes to obtain the time series embeddings. 5. The results are presented in an aggregated way across datasets. It would be of interest for researchers and practitioners how the proposed approach performs per dataset.

Questions

1. What is jitter-andscale strategy? 2. can the authors please expand a bit on what the following line means? _In this work, we choose P representative text embedding tp as pivots/prototypes, and map TS embedding to them_ 3. typo: Figure 3.g: spesific → specific 4. typo: Figure 2: Fine-turn → Fine-tune. 5. it seems that a general trend is that the LLaMa-13B performs best. It I understand correctly, this is as well the largest pretrained model considered. Further, one can see that for GPT2, the larger the model, the better the performance. Do we know, or can we do a conjecture, on what is the limiting performance in terms of model size (number of parameters)? 6. Why is LLM-QA not showing any variance? Or why is it depicted in this way? 7. Why is one-fits-all not used in *all* forecasting experiments? If I understand correctly, one can use it for comparison. 8. Why are not other models considered here? for instance DeepAR, MQCNN, Wavenet, NHiTS, NBeats, etc for these experiments? Even models like ARIMA, ETS, seasonal naive would be interesting to consider. 9. From 3.d it seems that the prompt length should be restricted to not be particularly large, but suprisingly the smaller the number of parameters (GPT-117M) the better the performance. How does this fit with the previous observation that larger models perform better? 10. On Embeddings (section 4.3): how does the embeddings change per model? For instance, what are the embeddings when using BERT-335M, GPT2-774M, etc? 11. When it comes to forecasting, is the proposed approach a point-wise or a probabilistic forecaster? 12. What is the performance if the LLM taken is the one of the first checkpoint, i.e. when the weights of the model are random? What is the gap between these case those taken for the current experiments? 13. What is the time execution/complexity of the proposed approach. Does this depend on the pretrained model in consideration? For instance, would it take longer of LLaMa-13B than for BERT-110M? 14. What is the performance per dataset when it comes to time series forecasting? The current experimental setting only reports results aggregated across different datasets.

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Summary

This research presents two approaches to handle Time-Series (TS) tasks with Large Language Models (LLMs): one that trains or fine-tunes a model for TS data, and another that transforms TS data for compatibility with existing LLMs. The focus is on the latter approach, introducing a method named TEST, which casts TimeSeries data into a format that LLMs can understand and then utilizes the LLM for TS tasks with the help of soft prompts. Experiments reveal that frozen LLMs equipped with the TEST strategy match or surpass current state-of-the-art TS models, enabling them to process TS data without losing language capabilities.

Strengths

The paper made clear the difficulty of using LLM for TS data: "In this way, the core is to create embeddings that the LLM can understand." TEST method proposed a "similarity-based, instance-wise, feature-wise, and text-prototype-aligned embedding for TS tokens" for this problem. The explanation and writing about the method, as formally shown in Algorithm-1, is straight-forward clear. The theretical analysis about casting Time-Series data into a pattern, and then translating the pattern into text, is an interesting hypothetis. The experiment design, which includes 7 dimensions and 5 baseline models, is comprehensive and convincing. The improvements of TEST can be clearly visualized by the shaded pink area in top-left Figure-3, which is quite significant.

Weaknesses

A picture to illustrate the connection between the "Time-series" data, and then the corresponding "text data to LLM", will be helpful for readers to understand the input and output of the TEST method. And it would be interesting to investigate the connection between resulting target text's semantics and the actual content of "Time-series" data.

Questions

Do you have concrete examples of how a Time-Series data is transformed into text/(any format consumable by LLM), and the output of an LLM to such time-series data(of course, with the soft-prompt included)? It would be helpful to include such information, if not in main text, in the Appendix. How scalable is the proposed method. Will you be releasing the encoder model, along with the soft-prompt for the task that the paper used?

Rating

8: accept, good paper

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Summary

The paper explores two ways to accomplish Time-Series (TS) tasks using Large Language Models (LLMs): LLM-for-TS (model-centric) and TS-for-LLM (data-centric). The authors focus on the TS-for-LLM approach and propose a method called TEST (TimE Series tokens to align the Text embedding space of LLM). TEST tokenizes TS data, embeds it using instance-wise, feature-wise, and text-prototype-aligned contrast, and aligns the TS embedding space with LLM's embedding layer space. Soft prompts are then created to make LLM more receptive to the embeddings. The paper demonstrates the feasibility of TS-for-LLM through theoretical analysis and experiments on TS classification, forecasting, few-shot, and representation tasks. The results show that pre-trained LLMs with the TEST strategy can achieve better or comparable performance to state-of-the-art TS models and offer benefits for few-shot learning and generalization. The authors hope this study will serve as a foundation for future research in the TS+LLM domain.

Strengths

1. The organization of this paper is well-structured, making it easy to read and comprehend. 2. The paper proposes a novel embedding method (TEST) that aligns Time Series tokens with the text embedding space of Large Language Models, effectively enabling LLMs to handle Time Series data without compromising their language abilities. This study will serve as a foundation for future research in the TS+LLM domain. 3. Through experiments on various Time-Series tasks, the paper demonstrates that the TEST strategy can achieve better or comparable performance to state-of-the-art models, offering benefits in few-shot learning and generalization, thus validating the feasibility of the TS-for-LLM approach.

Weaknesses

1. Based on the experience with LLM and multimodal fusion, it is possible to integrate LLM with image/video/speech representations using techniques such as adapters once good representations are obtained. Similarly, LLM4TS and TS4LLM can be fused with each other, for instance, when we obtain a good TS token embedding using TS4LLM, we can directly fine-tune the large model using LLM4TS, thereby achieving good performance. Therefore, is it appropriate to isolate LLM+TS into LLM4TS and TS4LLM as two distinct categories? 2. Time series prediction is a crucial task, and as observed from the experiments in this paper, there is a significant gap between TS4LLM, LLM4TS, and traditional methods when performing this task. Have the authors considered analyzing the reasons for this discrepancy? 3. In the statement "...we choose P representative text embedding tp as pivots/prototypes...", what specific type of text does P represent?

Questions

see above

Rating

5: marginally below the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Summary

The paper introduces TEST, a TS-for-LLM method that leverages LLMs in time series (TS) data analysis without the need for fine-tuning. TEST focuses on transforming TS data into LLM-understandable embeddings. It shows potential in enhancing few-shot learning performance by harnessing the inherent knowledge within LLMs. The method involves training an encoder using contrastive learning, comprising three objectives: instance-wise, feature-wise, and text-prototype objectives. The instance-wise objective pulls closer to similar instance embeddings. The feature-wise objective is applied to each feature column to further discriminate instances. The text-prototype objective aims to align the TS embeddings and prototype embedding using feature-wise loss. Learnable soft prompts are also employed to assist LLMs in understanding time series data. Experimental results demonstrate that TEST can rival or outperform current SOTA methods.

Strengths

1. The exploration of using language models for time series data without extensive pre-training or fine-tuning is an intriguing avenue of research. 2. I am impressed by the non-fine-tuning approach to achieve performance comparable to a supervised baseline. It demonstrates the effectiveness of the proposed method on the time-series problem. Additionally, exploring the TS-for-LLM method is a promising direction for few-shot settings.

Weaknesses

1. The paper lacks clarity regarding how the representative prototypes are chosen. There are three groups of prototypes (value, shape, frequency), and it's unclear how these are determined. Does their selection depend on the dataset, and what criteria are used to assess their effectiveness? 2. I would like to see an examination of the performance impact of removing the decoder component. It looks like the decoder is not a necessary component in the proposed method. 3. The paper could be stronger from an ablation study to thoroughly investigate the effectiveness of the proposed components and objectives. For example, what's the performance without using soft-prompt or feature-wise objectives? This would help readers assess the contributions of each component.

Questions

Have you considered the potential for parameter-efficient fine-tuning methods, such as LoRA, to outperform the use of soft prompts? Similar to using soft-prompts, LoRA does not require updating the model's entire parameter set. Reference: LoRA: Low-Rank Adaptation of Large Language Models, Hu et. al.

Rating

5: marginally below the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good