T-Rep: Representation Learning for Time Series using Time-Embeddings

Response to Rewiewer 9kuw - Part 1

We sincerely appreciate the positive feedback regarding the presentation of our paper. It is gratifying to hear that the content is smooth and easy to follow. We are also pleased that the limitations of existing methods and the novelty of our proposed approach are clearly communicated. We totally agree that the biggest flaw of the paper is the lack of ablation study, as explained in the global answer we are working on it and we will update the submission with these results in a few days. **Q1. While the authors claim that time embedding tau are induced to learn such time-related features as trend, periodicity etc. , they are function of the time indices and are unaware of the time series values. How could the aforementioned features be captured from indices only?** **A1.** T-Rep is a parameterised function we are learning, containing a sub-module for learning time-embeddings from time indices, an a module for feature extraction from the time series, with all modules interconnected. Because the loss function used to train T-Rep is a joint function of the produced representations and time-embeddings, the loss backpropagates through the feature extractor as well as the time-embedding module. This makes the time-embedding ‘aware’ of the time series values, through the loss function’s gradient (the loss is a function of the time series and the time indices). The loss function is given by the pretext tasks, which have been designed to encourage the time-embeddings to contain meaningful information about the time-series (particularly the ‘Time-embedding conditioned forecasting’ task, see section 4.2.2 for more details). This consequently pushes the time-embedding to contain time-related features. Now the nature of the features (e.g. trend, periodicity etc.) is, we believe, not only influenced by the pretext tasks but also by the choice of time-embedding architecture. For example, for forecasting and anomaly detection, we use Time2Vec (https://arxiv.org/abs/1907.05321). This architecture is quite interpretable: the first component of the time-embedding vector is a linear function, which Time2Vec authors claim learns the trend of the time-series. The other components are sinusoidal features with learned period and phase-shift, aiming to capture periodicity in the data at different scales. We hope this addresses your question. If you want more details, we encourage you to look at Time2Vec (https://arxiv.org/abs/1907.05321), which was the first paper to introduce this idea of learning a time-embedding module or function alongside the feature extractor/predictor. **Q2. I understand that tau is normalized so that their difference can be measured by JSD. But there are alternative ways to measure the divergence without confining the norm of time embeddings.** **A2.** Thank you for this very relevant question, which we thoroughly addressed during the model design process. Indeed, there are alternative ways to measure divergence without restricting the norm of time embeddings, but we haven't found any that provide as much satisfaction as the Jensen-Shannon Divergence (JSD). We explored the use of Kullback-Leibler Divergence, but it has some significant drawbacks: it is not symmetric, sensitive to zero values, and not defined on disjoint distributions. We also attempted to employ Wasserstein distance, but it proved computationally expensive, especially in high-dimensional spaces, leading us to quickly dismiss this option. The JSD appeared to be the best compromise, but we are open to suggestions if you believe that alternative approaches would be more relevant.

Response to Reviewer 5x3c

We appreciate your very detailed review and the attention given to our submission. We are pleased that you found our presentation excellent and the paper easy to read. Regarding the ablation study, as mentioned in the overall response, we are in the process of finalizing it to add these results to the article because we completely agree that it is a crucial experiment to include in our submission. In the same manner, we are in the process of adding the baselines you suggest. Having started the ablation study, we can already provide you with results of the ablation study on the forecasting task. We observe that the two pretext tasks complement each other well: if only one of them is used, the performance is lower than using none, and significantly lower than using both. Additionally, we can see that the time-embedding (TE) module effectively enhances performance, and it is the combination of these elements (TE + 2 proposed pretext tasks) that allows us to outperform the state-of-the-art TS2Vec. | | Forecasting | |-----------------------------------|-------------| | | Avg. MSE | |-----------------------------------|-------------| | **T-Rep** | **0.986** | |-----------------------------------|-------------| | *Pretext tasks* | | | w/o TE-conditioned forecasting | 1.022 (+3.7%) | | w/o TE divergence prediction | 1.003 (+1.7%) | | w/o New pretext tasks | 0.999 (+1.3%) | |-----------------------------------|-------------| | *Architecture* | | | w/o TE module (=TS2Vec) | 1.004 (+1.8%) | *Ablation results on ETT forecasting dataset. The percentage changes are calculated as the relative difference between the modified model's performance and T-Rep's performance.* **Q1. Appendix A.2 mentions that each models are run 10 times but also notes "Also, we do not use any random seed, (...)". I'm unsure what this means. Are all experiments started from the same seed or are ten (fixed) seeds used here? I would kindly ask the authors to clarify that.** **A1.** Thank you for raising this point, which seems to lack clarity in our article and was also highlighted by reviewer rWLt. Instead of fixing a seed, which has the disadvantage of not accounting for the variance that may exist between two seeds, we chose to run each experiment between 10 and 20 times to address variance and obtain reliable results. The reported results represent the average of these runs. This approach maximizes the reliability and robustness of our results, avoiding the selection of a specific seed that might produce the best results. During our 10 or 20 runs, the seed is set to "None." When the seed is set to None, PyTorch utilizes the system time or another unpredictable source to initialize the random number generator. Consequently, each time you run the program, you are likely to get different random numbers. *If our responses and revisions meet your expectations, we would greatly appreciate it if you could consider revising your evaluation accordingly. Thank you for your time and thorough review.*

Response to Reviewer znEu - Part 1

Thank you very much for all your highly relevant suggestions, motivating experiments that significantly enhance the quality of the article. We appreciate your considerations regarding interpretability and the quality of representations, which align with those of reviewers 99iS and rWLt. Currently, we are conducting experiments to demonstrate the meaningful time series properties of latent dimensions, and we will send you the results in the next few days. Similarly, we are incorporating comparisons with the baselines you suggest, and we are working on highlighting the robustness of our model to missing data, with a new real-world anomaly detection dataset. **Q1. The choice of time-embedding architecture is not well motivated or analyzed. Different architectures are used for different tasks, but it is unclear why they perform best in each case. More ablation studies on the time-embedding design could strengthen this key component.** **A1.** Thank you for pointing out this issue. The field of time-embeddings is very recent and little literature exists. The most famous time-embedding architecture is Time2Vec (https://arxiv.org/abs/1907.05321), which we use in forecasting and anomaly detection tasks. This architecture is actually relatively interpretable: the first component of the time-embedding vector is a linear function, which Time2Vec authors claim learns the trend of the time-series. The other components are sinusoidal features with learned period and phase-shift, aiming to capture periodicity in the data at different scales. This focus on learning a dataset’s inherent trend and periodicity explains why it helps the T-Rep outperform its counterparts in the forecasting and anomaly detection tasks. Generally speaking, constrained time-embedding architectures (i.e. sinusoidal form, linear, RBF features) will tend to be more interpretable than their very expressive but more ‘black-box’ MLP-based counterparts. We believe MLP-based time-embeddings outperform sinusoidal ones in classification (and other instance-wide tasks) because they can capture more ‘global’ and coarse features of a time-series instance’s temporal structure, allowing T-Rep differentiate different instances of a dataset more easily in downstream tasks. This is a result of the expressivity of MLPs and the lack of constraint on the features they can extract, compared to sinusoidal or RBF-based time-embedding architectures. It is very hard to give more rigorous explanations, as the learned features are not human-interpretable. Overall, we agree that there is limited understanding of why time-embeddings behave differently depending on the task. This is because it is a new niche of machine learning which has not been studied in great depth, and few people have experimented with thus far (we haven’t come across any papers using them in classification). **Q2. The pretext tasks using time-embeddings seem somewhat ad-hoc. While they demonstrate the utility of time-embeddings, developing more principled pretext tasks derived from intrinsic properties of time series could be beneficial.** **A2.** Thank you for your highly relevant question, which happens to be one of the main avenues of our future work. We completely agree that these pretext tasks are ad-hoc, and our intention was to experimentally verify their relevance before delving into the theoretical foundation of these pretext tasks using time embeddings. We are particularly interested in this aspect, which will be the focus of our upcoming work. **Q3. The comparison to previous methods like TS2Vec is not entirely fair, as T-Rep uses a larger encoder architecture. Comparisons with a TS2Vec model of comparable complexity could better isolate the benefits of the time-embeddings. Besides, the composition on TimeNet and PatchTST on SOTA ETT dataset is necessary.** **A3.** That is absolutely true, thank you for noting this point. We use a layer of 128 instead of 64 to facilitate the concatenation of the time embedding to the time series (an issue TS2Vec does not have to deal with). We considered that these dimensions, while larger, are still "small" (we are not in the range of 2048, for instance), making it acceptable. We have begun implementing the pipeline to compare PatchTST on the state-of-the-art ETT forecasting dataset. This will be included in the final version of the article, and we hope to present the results to you by the end of the discussion period. Based on our understanding, TimeNet cannot be included in this comparison as it has been developed for classification, and not for tasks requiring timestep granularity, such as forecasting. Do you think this comparison is really necessary despite this aspect?

Response to Reviewer rWLt - Part 1

Thank you for taking the time to review our article, and we sincerely appreciate your thoughtful comments. Your positive remarks on the motivation and clarity of our paper are encouraging, and we are pleased that you found the overall presentation to be well-written and easy to follow. We are currently working on the final experiments for visualizations and the ablation study, as you recommended and as explained in the common answer. In the meantime, here are the answers to your various questions: **Q1. How many datasets are used for anomaly detection (sec 5.1)?** **A1.** The Yahoo Dataset, consisting of 367 synthetic and real univariate time series featuring outlier and change point anomalies, was used for the anomaly detection task. To address the comments from the reviewers, especially reviewers 99iS and znEu, we are currently replicating our experiments on the 'Sepsis' dataset from The PhysioNet/Computing in Cardiology Challenge 2019, which contains multivariate data sourced from 40 000 Intensive Care Unit patients in three separate hospital systems. We chose this dataset because it contains real-life data, includes missing data, and anomalie appear by segment. Link of the dataset can be found here: https://journals.lww.com/ccmjournal/fulltext/2020/02000/early_prediction_of_sepsis_from_clinical_data__the.10.aspx. We will have the results shortly, providing us with a second dataset for anomaly detection. **Q2. How does the issues of similarity/dissimilarity play out in multivariate settings? What if the switching behaviour is only present in a few of the channels?** **A2.** This issue of similarity and dissimilarity in contrastive learning is treated the same way for univariate and multivariate settings. The semantic choice of which instances should be similar/dissimilar is entirely dependent on the method developed and the problem at hand. For example, in time-series it is common to want to represent two samples from nearby timesteps similarly. However, the way in which the similarity/dissimalirity of representations is evaluated is the same for univariate/multivariate: a dot product is computed between two representation vectors as a measure of similarity. With regards to the state switching issue, this happens regardless of the number of affected channels. The issue with contrastive learning and finite-state systems is the pair selection process: in the introduction of our paper, we explain that contrastive learning tasks “define positive pairs by proximity in time, and negative pairs by points that are distant in time (Banville et al., 2021; Franceschi et al., 2019), which can incur sampling bias issues. Points of a negative pair might be far in time but close to a period apart (i.e. very similar) and points of a positive pair might be close but very different (think of a pulsatile signal for example).” Whether the switching state behaviour is present in a few or all channels, this information will be lost during training because of the pair selection process. **Q3. Why does T-Rep perform better in some cases and not it others? For instance, in Table 5 in ETTh_2 T-Rep performs better at larger horizons, while being significantly worse at lower ones. This is counter-intuitive, since in section 4.2.2. the paper mentions that the pretext forecasting task specifically aims to predict shorter horizons.** **A3.** We appreciate this question, which we also considered during our experiments. We lack both theoretical and experimental evidence for why T-Rep performs better on longer horizons. However, our intuition leads us to believe that on a short horizon, we are still within a Markovian process, with the major dependence on the preceding moment. On a longer horizon, periodic processes come into play, and the model excels through its handling of time embeddings. This, however, remains speculative, and extensive experiments will need to be conducted in future work to confirm this. **Q4. It seems that the methods requires supervision in pretest task 4.2.1 and not in 4.2.2 for the pretext task, can the authors comment on potential use of T-Rep in completely unsupervised pretraining?** **A4.** It may not be expressed clearly enough in the article, but we confirm that both pretext task (4.2.1 and 4.2.2) are supervised. T-Rep as a whole is trained on datasets without labels (there is an ‘X’, but no ‘y’), so the training of this model can be regarded as unsupervised. However, it relies on the creation of artificial supervisory signals constructed from the given ‘X’. These artificial supervisory signals are what we call ‘pretext tasks’, and do not require any data or information beyond the given dataset ‘X’. This duality between not using any labels for the dataset but creating artificial supervised signals (=pretext tasks) to train the model resulted in the field’s name, “Self-supervised learning”: the supervision in pretext tasks originates from the input data.

Response to Reviewer 99iS

We appreciate your insightful review of our paper. We have diligently addressed each of your comments point by point, aiming to enhance the overall quality of our work in response to your valuable feedback. **Q1. The effectiveness of the proposed method relies on the evaluation done mainly on UCR/UEA and Yahoo Datasets which are infamous for their incorrectness, and triviality in labels. Refer: https://arxiv.org/pdf/2009.13807.pdf. This questions the effectiveness of the proposed method and how well would it work in real-world scenarios? Q2. Similar to Pt.1, I feel the downstream benchmarks are bit too trivial to test the true effectiveness/usefulness of a representation learning method. For e.g., in the task of anomaly detection, the authors propose to use a simple point-based evaluation method which is not realistic, as real world anomalies are usually segment based and need more rigorous evalution to demonstrate usefulness. Refer: https://arxiv.org/pdf/2109.05257.pdf.** **A1 and A2.** We acknowledge the concerns you raised regarding the datasets used for anomaly detection evaluation and the behavior of the model in real-world scenarios. We are aware of the limitations of these datasets, and we have recently become acquainted with them. We completely agree with the necessity of working with less trivial datasets. To address those 2 questions, we are currently testing our model on the 'Sepsis' dataset from The PhysioNet/Computing in Cardiology Challenge 2019, which contains multivariate data sourced from ICU patients in three separate hospital systems. We chose this dataset because it contains real-life data, includes missing data, and anomalies appear by segment. We are, in fact, implementing the PA%K protocol presented in the article you suggested (Towards a Rigorous Evaluation of Time-series Anomaly Detection, Kim et al., https://arxiv.org/pdf/2109.05257.pdf.), which helps alleviate the overestimation effect of point-based evaluation methods. This allows us to address both your concerns regarding the Yahoo Dataset while demonstrating that our T-Rep model is effective in real-world scenarios with messy data. Link of the dataset can be found here: https://journals.lww.com/ccmjournal/fulltext/2020/02000/early_prediction_of_sepsis_from_clinical_data__the.10.aspx We will communicate our results to you as soon as possible and include them in the article **Q3. In Sec 5.5, I get that one can manually look at the heatmaps to infer similarities between the patterns in raw time series signal and the learned representation but I didn't fully understand the usefulness of that? Can I just look at the heatmap and use that information? if so, how? Manually matching pattern is pleasing but I don't see any usability, and hence don't see any true interpretability coming out of it.** **A3.** What this heatmap shows is two-fold. Firstly, it serves as a valuable sanity check by offering a qualitative means of verifying the preservation of information and properties from the original signal in the representations. To give an example, this can be used by a data scientist working on a change point detection model to see if there's an abrupt change in the probability distribution of the signal. Secondly, it shows that the features extracted by the model are interpretable: we can visually see that variations in the representation vectors correspond to human-interpretable features such as periodicity, variance, anomalies, distribution shifts etc. One could imagine that the model learns features which aren’t human interpretable. For instance, in a CNN, the first few layers tend to capture high-level features like edges or textures, but the last few layers learn more abstract and complex features, which we can’t visually make sense of. It is very positive to us to see that this is not the case with T-Rep and that the extracted features can be visually interpreted and matched with properties of the original signal. Whilst we agree one shouldn’t rely on visual pattern matching, this provides advantages by providing visual support. We agree this visual aid is not sufficient for downstream tasks, where we rely on comparing representations quantitatively rather than qualitatively. *If our responses and revisions meet your expectations, we would greatly appreciate it if you could consider revising your evaluation accordingly. Thank you for your time and thorough review.*

Response to Reviewer 99iS - Experiments

We have now conducted the experiment you requested, which you can find in section 5.1 of the paper in blue. The results on the Sepsis dataset confirm that T-Rep performs better than TS2Vec and a linear baseline on a dataset different from UCR/UEA and Yahoo Datasets. The proposed method has a great effectiveness in a real-world scenario and on non-trivial downstream benchmarks, even when using an evaluation method that is not point-based. We have also added the two references you suggested. Thank you for proposing this improvement. *If our adjustments meet your expectations, we would greatly appreciate it if you could consider revising your evaluation accordingly. Thank you for your time and thorough review.*

Rebuttal submitted

Dear reviewers, We have finished taking into account all your comments. You can find the updated version of our article, with all the modifications we made highlighted in blue. These modifications include: - Improving the colors of Figure 1 to enhance visibility - Adding experiments on the Sepsis dataset for segment-based anomaly detection on a real-life dataset - Introducing the Minirocket baseline for the classification task - Adding a linear baseline for the forecasting task - Conducting a comprehensive ablation study - UMAP visualization of learned representations from T-REP and TS2VEC - Mathematical formalizations of contextual consistency and hierarchical loss We believe that these additions significantly enhance the quality of our submission, and we sincerely thank you for guiding us through their implementation. In addition to these various additions, we have addressed all of your questions. Please see each individual response for more details.