Deep Unsupervised Domain Adaptation for Time Series Classification: a Benchmark

Thank you for your valuable comments and suggestions. We particularly appreciate all the strengths you mentioned about our work. Here are our replies to the weaknesses you raised. **Presentation** - P1. We acknowledge the reviewer's suggestion to incorporate an *additional critical difference diagram* for a comprehensive comparison of all experiments, presenting an overall perspective on performance. This point has been addressed in the Appendix A.2.1. The average accuracy rank across all algorithms and model selection methods delineates three clusters: \ i) the first cluster comprises most of the algorithms trained using the target-only method, \ ii) a second cluster includes OTDA and VRADA, identified as the less performing algorithms across the three hyperparameter selection methods, and \ iii) a third cluster encompasses the remaining algorithms, fairly/visually distributed between the source-only and IWCV methods. Due to space constraints, we kept this further analysis in the Appendix only. - P2. We have included a comprehensive table, where each column corresponding to a specific method (Source Risk, IWCV, Target Risk), in the Appendix A.1.2. The table aims to provide *a summary by averaging accuracy scores across datasets for each classifier*, accompanied by reporting of standard deviations. - P3. We have added *variations of Figure 4* with different datasets and UDA algorithms in Appendix A.2.5, as well as an analysis of these graphs. They confirm a certain correlation of the source risk and IWCV with the target risk, but show a plateau for the source risk, and some different optimum for IWCV, thus explaining the gap in performance compare to target risk. -P4. We agree with the reviewer's comment regarding the effectiveness of the plots in Figure 8 for conveying results. Due to space limitations, these additional plots have been placed in the appendix alongside all the extra graphs. **Significance** - S1. We completely agree with this point and we would like to answer with the following elements. First, the results are part of the git repository in a CSV format, and will be available when we open-source it for checking reproducibility and going further in the analysis. Furthermore, the implementation of our benchmark relies on existing tools to strengthen its *reliability, scalability and modularity* and that we believe will help other research to easily add datasets and/or algorithms. These tools include Hydra for the configuration of the pipelines and parameters, HuggingFace for the datasets and evaluation of the models, PyTorch with HuggingFace, scikit-learn and POT for training the algorithms. As a proof of its modularity, we can mention that it only took us a few hours to add an additional algorithm (SASA) to the benchmark and run the experiment with it during the current rebuttal phase. **Questions** - Q1. We appreciate your highlighting of this analytical aspect. To address this, we have included a new section in the Appendix A.1.3 specifically dedicated to emphasizing the *dataset imbalances*. In this section, we introduced a scoring metric, denoted as I, which quantifies the degree of imbalance, where a score of 0 indicates a highly balanced dataset, and a score of 1 signifies high imbalance. The results are presented in a table format, summarizing the number of classes, samples, and I scores for both the source (train, test) and target (train, test) datasets. It is important to note that only three datasets exhibit significant imbalance, posing additional challenges to the classification task. Note that two out of these three datasets are part of the original established set of datasets for UDA TSC.

Thank you for your valuable comments and suggestions. Here are our replies to the weaknesses you raised: 1. We agree with the reviewer that the technical novelty is limited, but we think this is expected from any benchmark study, as the goal is to assess the quality of existing algorithms on data which should be available openly to the community. Still, we have not limited ourselves to existing TSC UDA dataset and we adapt 7 diverse time series datasets to UDA settings which have never been used before for UDA. Moreover, we have not limited ourselves to simply comparing existing methods, as this could be expected from a benchmark, but we proposed the usage of the Inception backbone that, not only achieves the best performance, but also allows a fair comparison between SOTA methods having different backbones. 2. We thank the reviewer for pointing out these references. We believe the work on UDA for video data [ref1] is a particular field and should not be included here, as it is a combination of image (spatial) and time series (temporal) problems and therefore goes beyond the scope of our study. Nevertheless we have added the latter paper as part of our references of UDA for computer vision tasks. Furthermore regarding the Gaussian Process Regression [ref2], although the paper tackles time series and tabular data, it focuses on regression (data extrapolation) rather than classification tasks and does not tackle the unsupervised DA setting. Nevertheless we have now mentioned regression DA task in the introduction section. Finally, regarding the SASA model [ref3], we strongly agree with the reviewer that this reference is relevant to our benchmark. We therefore have added the Section A.4 in the appendix where we present the results of SASA (with both LSTM and Inception backbones) over all 54 datasets (accounting to almost 1296 hours of gpu time during the current rebuttal phase). However, we decided to keep it outside the main benchmark as it is more suited for the task of detecting an event within a time series, and not classifying the complete input time series, since it is making the decision based on the last few time stamps, which probably explains its bad performance in our study. Comments were also added in the main paper to mention SASA. 3. We strongly acknowledge the significance of quantifying the shift between various source and domain pairs, as it holds the potential to offer valuable insights into when domain adaptation proves beneficial and which algorithm is most effective. However, given that quantifying the shift in DA is already difficult, adding to that the complex dynamics of a time series data, we believe that this domain shift quantification is an ongoing area of research that we plan to address in our future work. The complexities stem from the unique nature of time series data, where shifts can manifest in time, features, or a combination of both. Measuring domain discrepancy, e.g. through metrics like Maximum Mean Discrepancy or Wasserstein distance, on different domain pairs may not provide sufficient information about time series shift. As an illustration of this difficulty, OTDA leverages this Wasserstein distance, which quantifies the shift, but does not work well for time series data in our benchmark. We found that using the performance of Inception (i.e. with no adaptation) was a quick and easy proxy to compute the shift, which can be seen in Figure 3, although we cannot assess how good a proxy it is. We also added a paragraph in Appendix A.1.1 to explain some of the discrepancies for specific cases (e.g. when we know that one domain is more noisy than others). 4. We believe that there is already some interesting insight for the community even if they are indeed quite general. We cannot really enter too much into details of combination of algorithm/dataset.\ i. The limited impact of IWCV compared to Source Risk, which is now additionally supported by the added Figure 6 that mix the methods.\ ii. The limited impact of backbone, additionally supported by the comparison for different dataset theme and method.\ iii. The fair comparison between algorithms with the same common Inception backbone, and the good performance of InceptionRain.\ We have also added some more analysis in the Appendix following the other comments.

Thank you for your valuable comments and suggestions. Here are our replies to the weaknesses you raised: 1. Note that we compare Inception with 3 different architectures that are proposed in the literature (CoDATS, Raincoat and CoTMix). Because it is hard to compare the different backbones of different methods directly, we chose the Inception backbone as reference. Following your remark, we added a section (Appendix 2.6) and three plots in Figure 11, that compare the 2 additional backbones individually with Inception. The same conclusion holds for each of them. Furthermore With this result, we believe that there is enough argument for the limited impact of backbone in time series UDA. Additionally, a proper comparison of another classical backbone would lead to around 5 months of GPU’s time. 2. We agree with the reviewer about the necessity of a deeper analysis. However, we believe part of this analysis relies on an estimation of the shift between the datasets to understand how much shift in the data each method can support. The estimation of the shift being still an open issue and difficult to tackle, the best we proposed here is an analysis about the difficulty of a dataset, which we can assess with accuracy scores as well as standard deviation from pairs of source-target. 3. Firstly, the InceptionTime algorithm has been found to be the best deep learning algorithm in a recent benchmark for Time Series Classification [ref1], which is the reason why we chose this backbone for a common comparison of the UDA approaches. Following your comment, we added in Section A.2.6 the Figure 12 to provide a better comparison about the differences between various types of dataset (Machinery, Medical, Motion and Remote sensing) and add comments, especially on a potential bias toward Motion dataset for existing methods due to existing and already established UDA TSC datasets. The latter observation justifies the need to expand the list of UDA TSC datasets and improve generalization to different TSC domains, which is one of the main contributions of this paper. 4. Indeed, both figures have similar finding and can be merged but we still think that focusing specifically on Raincoat, which is the best method, is still important to support our claim: it is the Raincoat’s UDA method behind the better performance rather than its backbone. [ref1]: Ruiz, A.P., Flynn, M., Large, J. et al. The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Min Knowl Disc 35, 401–449 (2021)