Learning to Embed Time Series Patches Independently

**W1. Support to the claim that PI is superior to PD for TS representation learning** Our observation is consistent across seven real-world datasets from various domains in the following four aspects, which is the main support to our claim (all table and figure indices below are based on the *new version* of our submission): - (1) Efficiency: **Table 13**, **Table I.1** - (2) Performance: **Table 2--8** - (3) Interpretability: **Figure 7** - (4) Robustness to distribution shift : **Figure 5** - (5) Robustness to patch size : **Figure 6** Nonetheless, we agree that this observation might not be extendable to some cases. We hope our proposed PI strategies provide a strong baseline for future TS representation learning works. For example, a future work could outperform ours by proposing more sophisticated PD strategies or benchmarks that require learning patch-dependence to perform well on downstream tasks. However, based on our empirical observation, it seems real-world datasets for TS analysis that we commonly use do not require learning patch-dependence.   **W2. Related works on the complementary masked strategy** As noted by Reviewer oFTu, several complementary masked strategies have recently been proposed [A,B,C]. We initially omitted them as 1) our focus is primarily on the PI strategies, and 2) the role of the proposed CL is to boost the performance further as an auxiliary task, without hurting the PI task. However, we agree that discussion on those works is indeed interesting and helpful to remind audiences of recent progress, so we added a brief discussion on them in **Section 2**. SdAE [A] employs a student branch for information reconstruction and a teacher branch to generate latent representations of masked tokens, utilizing a complementary multi-fold masking strategy to maintain relevant mutual information between the branches. TSCAE [B] addresses the gap between upstream and downstream mismatches in the pretraining model based on MM by introducing complementary masks for teacher-student networks, and CFM [C] introduces a trainable complementary masking strategy for feature selection. However, our proposed complementary masking strategy differs in that it is not designed for a distillation model, and our masks are not learnable but randomly generated. [A] Chen, Yabo, et al. "Sdae: Self-distillated masked autoencoder." ECCV (2022). [B] Ye, Shaoxiong, Jing Huang, and Lifu Zhu. "Complementary Mask Self-Supervised Pre-training Based on Teacher-Student Network." ACCTCS (2023). [C] Liao, Yiwen, et al. "Deep Feature Selection Using a Novel Complementary Feature Mask." arXiv preprint arXiv:2209.12282 (2022).   **W3. Clarity of sentences** We apologize for any confusion caused by certain sentences. For the example you provided "Secondly, we propose to utilize the simple PI model architecture (e.g., MLP), as opposed to the conventional PD architecture (e.g., Transformer), **which** is not only more efficient but also performs better.", **which** refers to **the simple PI model architecture**. We revised the sentence in the revision to clarify it. We will also keep improving the writing to make the final version clearer.   **Q1. Unit of training time** We apologize for the unclear expression. The unit of training time **Table 2 (of old version)** was sec/epoch, i.e., the time spent for one epoch of training in seconds. In the extended version of **Table 13 (of new version)**, we changed it to the total training time in minutes.

**Q1. Explanation on why the PI architecture is superior to the PD architecture** We believe the PI arch is superior for a similar reason as pretraining with the PI task: **learning to embed time series patches independently is better.** The difference between the PI arch and PI task is in the stage where we force PI, i.e., by the arch design for the PI arch, or the learning objective for the PI task. Apart from the performance, another prominent advantage of the PI arch over PD arch is its efficiency, in terms of the number of model parameters and inference time shown in **Table 13 (of new version)**. As we correlated the advantage of PI arch and task above, here also we remind the qualitative intuition/analysis on why the PI task is superior: - 1. **Figure 1** provides us an intuition that pretraining with the PI task is more robust to distribution shifts. - 2. **Figure 5** provides us a more systematic analysis in terms of different trend and seasonality, where we observed that pretraining with the PD task becomes even worse when the slope is flipped and amplitude is increased. - 3. Though it is not about the performance, **Figure 7** shows that the PI arch is more interpretable than the PD arch.   **Q2. Details of experimental settings** We added **Appendix B (of new version)** to provide the detail on experimental settings. We also clarify that we will release the code upon acceptance for reproducibility. However, if you find some hyperparameter or setting is still not clear, please feel free to share it. We also note that all hyperparameters are chosen by validation, where we added the information about validation in **Section 4.2**, and we provide ablation studies and analyses for our design choices through **Section 4.4 and 5**.   **Q3. Comparison with self-supervised learning approaches other than MTM and CL** Unfortunately, we could not find other self-supervised learning approaches competitive to MTM and CL in recent works. We note that our ablation study in **Figure 4** includes **the comparison with the vanilla autoencoder, i.e., the one "w/o Dropout"**. However, following the conventional knowledge, autoencoders can learn good features when the size of the hidden dimension is small; otherwise, autoencoders might learn the trivial solution. **Figure 4** shows that the vanilla autoencoder is generally not better than ours as a pretext task. We agree that exploring or revisiting other self-supervised learning approaches would be an interesting future work.   **Q4. Explanation of limitations and potential challenges** Thanks for bringing this to our attention. Though our extensive experiments demonstrate the effectiveness of the proposed PI strategies across multiple benchmark datasets that we have evaluated, this observation might not be extendable to some cases. For example, in our opinion, if a dataset has long-term dependencies, the dataset size is sufficiently large to learn such dependencies, and capturing the dependency is crucial for a downstream task, then a sophisticated PD strategy may outperform PI. In fact, the PD architecture can be seen as a generalization of the PI architecture, as learning to ignore patch-dependence completely in the PD architecture makes it operate as like PI. Hence, the PD architecture should be able to replicate the performance of the PI architecture given that other conditions are ideal (though we could not empirically find such ideal cases). We hope our proposed PI strategies provide a strong baseline for future TS representation learning works.

**W1. The training and inference efficiency analyses are brief or missing** We agree that the information provided in **Table 2 (of old version)** was somewhat brief, so we extend and move it to **Table 13 (of new version)**. To highlight the additional information, **Table 13 (of new version)** now compares the inference time and performance. **PITS shows ~2.3x faster inference time than PatchTST**, thanks to the efficient PI architecture. Also, we **differentiate our method with or without CL**, such that **Table 13 (of new version)** also shows how much the pretraining time increases to achieve better performance when incorporating the naive or hierarchical CL. Regarding the concern on **"training time compared to other supervised learning methods"**, we compare the training time of PITS and its supervised learning version (used the PI arch but no pretraining with the PI task) in **Table I.1 (of new version)**. Self-supervised PITS pretrains the model for 100 epochs, trains the linear head for 10 epochs, and fine-tunes the entire model for 20 epochs in sequence, and supervised PITS trains for 100 epochs to achieve good performance. Given that pretraining is already done, fine-tuning the pretrained model is **x3~4 faster** than training from scratch, while providing better performance.   **Q1. Size of input horizon for the TSF task (Table 3 of old version; 2 of new version)** We tuned the input horizon size with 3 candidates via validation: $L \in \\{336, 512, 768\\}$, where we added the information about validation in **Section 4.2**. We realized that the detail on experimental settings is missing, so we added **Appendix B (of new version)** to provide them. We also clarify that we will release the code upon acceptance for reproducibility. However, if you find some hyperparameter or setting is still not clear, please feel free to share it.   **Q2. The number of runs for Table 3 (of old version; 2 of new version)** We ran the experiments 3 times with different random seeds, and they gave us consistent results: the standard deviation is mostly in the order of $10^{-4}$ to $10^{-3}$, while we observe the performance gain in the order of $10^{-2}$ to $10^{-1}$. For more details, we provide the standard deviation of the performance of PITS in **Appendix L (of new version)**.

**W1-a. Contribution of our work** As noted by Reviewer ZBCM, autoencoding is a well-established pretext task for self-supervised representation learning, prior to masked modeling (MM). Our contribution is on the fact that we **revisit autoencoding by adapting the task and architecture (to be PI) for better TS representation learning**, and our extensive experiments show that the proposed method achieves **SOTA performance while being more efficient than Transformers pretrained with MM** [A,B,C]. We believe our findings shed light on the effectiveness of self-supervised learning through simple pretraining tasks and model architectures in various domains, and provide a strong baseline for TS representation learning. [A] Zerveas, George, et al. "A transformer-based framework for multivariate time series representation learning." SIGKDD (2021) [B] Nie, Yuqi, et al. "A time series is worth 64 words: Long-term forecasting with transformers." ICLR (2022) [C] Dong, Jiaxiang, et al. "SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling." NeurIPS (2023).   **W1-b, W2. References for combining CL and MM** Thanks for the suggestion, we added a brief discussion on recent works on combining CL and MM in various domains [A,B,C,D,E]. We initially omitted them as our main focus is on PI strategies, but we agree that discussion on those works is indeed interesting and helpful to remind audiences of recent progress. Among them, we note that SimMTM [E], which is a concurrent work to ours, proposed an MM task with a regularizer in its objective function in the form of a contrastive loss (though they did not claim that the regularizer is about contrastive learning but "series-wise similarity learning" and "point-wise aggregation"). This regularizer enforces that the original TS and its masked TS exhibit close representations and are distinctly different from representations of other series. However, it differs from our work in that it performs CL *between TS*, while our proposed CL operates *within a single TS*. Nevertheless, since our primary emphasis is on PI tasks and architectures, we removed the claim in **Section 1** that "our work is the first to integrate MTM and CL." [A] Jiang, Ziyu, et al. "Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations." ICLR (2023). [B] Yi, Kun, et al. "Masked image modeling with denoising contrast." ICLR (2023). [C] Huang, Zhicheng, et al. "Contrastive masked autoencoders are stronger vision learners." arXiv preprint arXiv:2207.13532 (2022). [D] Gong, Yuan, et al. "Contrastive audio-visual masked autoencoder." ICLR (2023) [E] Dong, Jiaxiang, et al. "SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling." NeurIPS (2023).   **Q1. About Figure 5** The purpose of **Figure 5** is to see ***how robust PI task is under distribution shifts than PD task***. For better understanding, we added further details regarding the explanation of **Figure 5** in **Section 5**. **[Left Panel]** The **left panel of Figure 5** depicts a toy dataset comprising 98 time series exhibiting varying degrees of distribution shift, where **we assign a different color to each time series**. The degree of shift is characterized by changes in slope (trend) and amplitude (seasonality). About how the toy dataset is exactly made, if the slope and amplitude of the training data are denoted as $s$ and $a$, respectively, then the slope and amplitude of the test data are represented as $r_1 \times s$ and $r_2 \times a$, respectively. Consequently, $r_1$ and $r_2$ serve as indicators of the slope and amplitude ratios, reflecting the degree of distribution shift. **[Right Panel]** The **right panel of Figure 5** illustrates the difference in MSE between the model trained with PI and PD tasks using the 98 time series depicted in the left panel (i.e., MSE of PD task - MSE of PI task). The x-axis and y-axis correspond to $r_1$ and $r_2$, respectively. For each $r_1$ and $r_2$, the color intensity shows how much PD is worse than PI (the darker, the worse.) The observations from this figure are: - 1. As the MSE difference (PD-PI) is non-negative for all cases, PI is more robust to distribution shifts than PD. Basically, PD becomes worse than PI as the change of $r_1$ and $r_2$ is larger (out of the center marked with the black rectangle). - 2. The color gradient shows in which direction PD becomes even worse: when the slope is flipped (right to left) and the amplitude is increased (top to bottom). If you have further questions, please feel free to share them. We are happy to address them and improve the clarity of our work.