Time Series as Images: Vision Transformer for Irregularly Sampled Time Series

Summary

This paper explores a very interesting direction to represent time series as plotted images and then stack vision transformers to perform representation learning. Following this idea, this paper has conducted empirical studies on some classification benchmarks of both irregularly sampled time series and regularly sampled ones. By leveraging this new data representation, the proposed method has obtained the state-of-the-art performance on three irregular time-series benchmarks and also produced competitive results on multiple regular time-series dataests.

Strengths

In my view, the biggest strength of this paper is the first successful demonstration of applying image-driven methods on representation learning of irregularly sampled time series. While after reading the related work section, I have also learned that this paper is not the first to represent time series as real images in general. Neverthess, I think the successful demonstration of image-based time-series representation learning on time-series classification tasks is interesting and has sufficient novelty.

Weaknesses

However, I do have several concerns on the experimental results at the current stage. - Given the relatively high performance variations when employing different plotting configurations and specific early stopping epochs on different datasets, I would like to know the detailed hyper-parameter tuning procedure when you develop ViTST. Are you tuning these options directly based on the feedback from the test set? If not, can you elaborate on how to divide the validation part and please report the details of performance on both validation and test sets if possible. - This paper includes some ablation tests, but I think the existing results are far from enough to convince me about the robustness of ViTST. For example, as table 3 shows, this method is sensitive to the existences of interpolation, markers, colors. I want to know that whether ViTST is robust to 1) different line styles and widths given the existence of interpolation, 2) different marker sizes and types given the existence of markers, and 3) different color permutations assigned to multivariate time series. My central confusions lie in which aspects play indispenable roles in making ViTST work and whether ViTST is robust to those irrelevant options. Moreover, I would like to see more ablation results about grid layouts and image sizes. Tables 2, 3, 4 only cover a few, and there seem to large performance variations, too. - Another question bothering me is that why ViTST can obtain the best performance for irregularly sampled time series but is only roughly comparable to existing solutions for regular time series. - I have tried the code but cannot reproduce the results.

Questions

See the weakness part.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The major limitation is that we are not clear many other critical factors ensuring the success of ViTST and whether it is robust to different plotting setups.

Summary

The paper investigates the use of pre-trained Vision Transformers (ViT) for both regularly and irregularly sampled time series classification. Two primary aspects are discussed: Transforming time series into images and performing time series classification using pre-trained ViT. Through a series of experiments, the authors illustrate the feasibility of this research direction.

Strengths

The paper is well-organized and presents an engaging perspective on time series classification, particularly focusing on the potential of ViT in relevant tasks. The authors conducted extensive empirical evaluations to showcase the effectiveness of the proposed framework, which is a strong aspect of the work.

Weaknesses

+ The paper falls short in providing an in-depth analysis of several critical aspects, such as the advantages of using ViT over other foundational models for time series, factors contributing to successful cross-domain transfer when employing ViT for time series classification, and how key design factors impact performance. See the listed questions for details. + In addition, some crucial technical details are missing. For example, it is unclear which layers in ViT are frozen during downstream fine-tuning. Justification for the use of line graphs over other visualization strategies (e.g., frequency maps) is not adequately discussed in Section 3.1. The model's efficiency compared to specialized models in time series modeling is also not clear. + There are some writing issues. For example, the title should be “…for Irregularly Sampled Time Series Classification” unless the effectiveness of ViTST is evident on other time series tasks such as forecasting and imputation. Besides, it is unknown whether the backbone ViT model are frozen in Fig. 2.

Questions

+ What distinguishes the use of ViT for time series modeling from other foundational models like language or acoustic models (e.g., Voice2Series)? Are there unique advantages, beyond performance, to using ViT? + What theoretical factors contribute to such a successful cross-domain transfer? According to the experiments, some factors like line colors may significantly affect the model performance. A follow-up question is how to claim the robustness of the proposed framework in general time series classification when changing line shapes, colors, or even time series datasets to other domains other than healthcare. + What types of interpolation are used in the experiments? + What is the motivation behind Section 4.5? Is it for a second pre-training based on pre-trained ViT or is it for pre-training from scratch? + In the transformation of time series to images, why do the authors opt to combine different univariate time series into a single image, as opposed to potential alternative implementations like introducing an additional layer to fuse different input images before feeding them into ViT?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors have not adequately addressed the limitations and potential negative societal impacts of their work.

Summary

The paper focuses on learning from irregularly sampled time series data. The paper presents a simple approach that converts irregularly sampled time series into an image where different input channels are line graphs. The converted images are then modeled using a standard Transformer model. Experiments show that the proposed approach outperforms the SOTA approaches on several datasets.

Strengths

- The paper focuses on the task of learning from irregularly sampled data which is important in many domains. - Experimental results show the effectiveness of the approach when compared to baselines and other recent approaches. - The paper is clear and well written.

Weaknesses

- The irregularly sampled time series data are represented by a line graph where observed points are connected using a straight line which is a very adhoc way to deal with missing values and irregularity. - The novelty is marginal as the proposed approach mainly uses a standard Transformer model where time series data is fed as images. - The standard deviation of results in Table 2 seems very high. It is not immediately clear if the proposed approach achieves statistically significant performance or it's just noise. - The proposed approaches uses static features present with the data. It is not immediately clear how these static features were used with the baseline approaches. - The paper notes that all the methods were trained for 20 epochs. Did they all converge in 20 epochs? The baseline approach should be trained till convergence. - The paper is missing experiments on MIMIC-III dataset, most commonly used dataset in this space. - The paper is missing comparisons with recent ODE based approaches which achieve SOTA performance. - Does the size of image change if the sequence length changes?

Questions

I have mentioned my concerns in the weaknesses section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

No.

Summary

This paper describes a surprisingly simple and effective approach for applying computer vision Transformer models to time series classification. Multivariate time series inputs are plotted in a grid to produce images that are used to fine-tune pretrained vision Transformer models. This approach achieves state-of-the-art results on irregularly sampled datasets and competitive results on regularly sampled datasets. Aside from the empirical efficacy, the results are relevant to understanding the capabilities of Transformers and the effectiveness of cross-domain pretraining.

Strengths

I think the basic finding of the paper - that pre-trained vision Transformers are effective at performing time series classification - is original and significant. It raises very interesting questions about the general efficacy of pre-trained Transformers and the relationship between time series and visual data that could inform future work. In a practical sense, I have some concerns detailed below but I generally appreciate the simplicity and efficacy of the proposed approach. The basic results are convincing and the ablation results are extensive and informative. Most of the questions I would have had about the model are already addressed in the main paper or Appendix. The paper was well-written and clear. Results are reproducible since full code was provided.

Weaknesses

To use the proposed method requires making a number of somewhat arbitrary decisions on how to convert time series to images: what layout to use, what order to put the time series in, what colours to use, line width/style, etc. The results in Table 3 and the Appendix show that decisions like these can affect performance even though they don't strictly change the information contained in the input. While these factors can be empirically tuned, the lack of robustness to seemingly arbitrary choices is concerning. In particular, the use of matplotlib to generate inputs makes the input construction somewhat poorly controlled. While I appreciate the simplicity of using an existing library, this leaves many implementation details about how exactly plots are rendered up to the library. Again, given that purely visual variations can affect model performance, it's not clear how important these implementation details are. I would have preferred to see a more precise approach that formally determined what numeric value is assigned to each pixel, making these details apparent. The range of irregularly sampled time series datasets is somewhat small to draw general conclusions from, but this is in line with existing work in this area.

Questions

How were line colours selected? If random, what distribution of colours was used? Is there any justification for the specific choice made? For your self-supervised learning experiments, did you start with a pretrained model?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are not explicitly discussed. No societal impact concerns come to mind, but I think technical limitations should have been addressed. These could include comparing training and inference resource requirements to other models and acknowledging points like those I raised in the Weaknesses section if the authors agree they are limitations.

Summary

For the classification task based on irregular time series data, the authors introduced Vision Time Series Transformer (ViTST) approach where irregular time series data is displayed as line graph then fed to pretrained transformer type models. The authors tested this approach using several time series data from medical domain and human activity domain, and the result shows confident advantages of the proposed approach.

Strengths

Simple idea but very interesting. Could contribute to many domains dealing with time series data.

Weaknesses

Time series data is everywhere but the authors tested only a few dataset from medical domain and human activity domain. Thus "any shape" in line 357 sounds too strong at this time.

Questions

Have you tested multi-class classification tasks and/or regression tasks?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Since this approach could have a potential to expand widely, it would be better to state "Limitation" of this paper.

Summary

The authors propose a method to model irregularly sampled time series. The method is based on transforming numerical time series data to line graphs and then applying pretrained vision transformers to that data. For multivariate datasets, every variable is plotted separately, and plots are aggregated in a grid. The authors provide exhaustive evaluation on different datasets, including leaving-sensors-out settings and regularly sampled time series data.

Strengths

The paper is well written and easy to understand. The method proposed by the authors is extremely simple, yet surprisingly effective on many time series modeling tasks, as shown by exhaustive experimental evaluation.

Weaknesses

- I don’t see the authors claims made in the abstract and introduction backed by their empirical results. It is difficult to judge, whether their proposed method indeed outperforms previous approaches “significantly” (L. 10, L. 43, L. 67) as such an analysis is missing. On most of the metrics for the P19 and P12 data at least, improvements do not look significant to me. On the PAM dataset results are likely significant (analysis is also missing here). - The PAM dataset (the only one where we likely actually see a significant improvement over Raindrop) is also the one with the fewest number of variables (17 vs 34 & 36 for P19 and P12, respectively). I assume that the proposed method performs much worse for datasets with many variables, as also shown in the authors experiments on the FD, and PS datasets, where ViTST performs indeed much worse than other methods. - The authors do not comment on the computational cost of their method, which is likely much higher than for most non-vision based approaches. - Recently, Semenoglou et al. proposed a very similar method, albeit limited to single-variable time series. They plot a univariate time series and feed this plot into a CNN to predict future observations of the same series. While I think that the present work improves on the work by Semenoglou et al. in several regards (multivariate, pretrained ViT outperforming pretrained ResNet, …), the authors should include a reference to Semenoglou et al. in their related work. Additionally, I don't think statements such as "This paper studies the problem from a whole new perspective by transforming irregularly sampled time series into line graph images and leveraging powerful vision transformers for time series classification in the same way as image classification." (L. 4-7) are valid, given that the idea to transform time series to images and then applying image classification networks is not novel. - The authors do not comment on limitations of their method.

Questions

- For the PS dataset, the authors must have used much smaller (I assume 12x12) sizes for their plots, am I correct? - What is the computational cost of the proposed method? - Did the authors conduct experiments on whether the order of the plots in the grid matters? - Based on the ablation studies provided by the authors it seems likely that most of the performance comes from pretraining of the transformer models. Wouldn’t a large pretrained transformer for numerical time series modeling be likely to perform similar or better than the method proposed by the authors? If the authors agree, then I would like to see a brief discussion of this in an updated version of the manuscript.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Unfortunately, the authors do not comment on limitations of their work. I see the following possible limitations and would like to see an open discussion of those in an updated version of the manuscript: - Ability to handle extremely large number of variables - Computational costs of the method

Thanks for the reply

I think this paper is interesting so I try to reproduce your results with your code. However, I find some crucial issues and would like to hear your answers. - When deciding how to plot time-series images, especially the y-axis ranges, your code seems to use all data (including both train and test), which is improper because this step is just like doing normalization for time series, you should only use the training data to determine these hyper-parameters. - It is hard for me to reproduce your results reported in the paper. I mainly follow your provided code to run the experiments. Here is what I have obtained on P12 using ViT. Accuracy = 71.8 +/- 9.3 AUPRC = 16.1 +/- 1.2 AUROC = 54.0 +/- 2.1 Precision = 20.0 +/- 9.0 Recall = 22.6 +/- 13.6 Would you like to provide detailed instructions for me to replicate your results? I think tuning such a vision model must be very tricky. So, I change the score to 'Reject' unless I can reproduce the results.

Response to author rebuttal

Dear reviewer, The author rebuttal appears to have presented several targeted responses to your questions. Are your questions appropriately addressed? If they are, would you consider re-assessing your score in light of them. If not, please do provide additional context and feedback to the author. In either case, please provide an acknowledgement of the effort the authors put in, why your questions have (or have not) been addressed and what your assessment of the work is in light of this evidence with a view to reach consensus with the other reviewers on this work. -AC

Response to author rebuttal

Dear reviewer, The author rebuttal appears to have presented several targeted responses to your questions. Are your questions appropriately addressed? If they are, would you consider re-assessing your score in light of them. If not, please do provide additional context and feedback to the author. In either case, please provide an acknowledgement of the effort the authors put in, why your questions have (or have not) been addressed and what your assessment of the work is in light of this evidence with a view to reach consensus with the other reviewers on this work. -AC

Reviewer's reply to the rebuttal

I would like to express my gratitude to the authors for their detailed responses, which have helped to clarify several of the issues that were previously raised. After carefully reviewing the manuscript, considering the comments of other reviewers, and evaluating the authors' rebuttals, I am inclined to adjust my score to borderline reject. However, I must emphasize that the paper, in its current form, is not yet in a condition suitable for publication at a conference of NeurIPS' caliber. There are several aspects that require more comprehensive analysis, and these issues have not been fully resolved in the authors' rebuttal: + What theoretical factors contribute to such a successful cross-domain transfer? As this is the main argument in this research, answering this question is crutial. + I see no experiements related to model efficiency comparisons. The aforementioned results (e.g., Tab. 2 and Fig. 3) are not related to this. Otherwise, it makes no sense to claim the *simplicity and compression* in this research. + The authors' rebuttal does not convincingly justify the impact of line shapes and colors on the robustness of the proposed method. + Regarding the motivation of image combinbation, I see no results and discussion related to the mentioned preliminary experiemnts. Please consider to clarify this: *In our preliminary experiment, we tested obtaining separate image representations for each sub-image and concatenating and feeding them into a prediction layer. However, this approach consistently underperformed the unified image processing technique.*