Con4m: Context-aware Consistency Learning Framework for Segmented Time Series Classification

Summary

This paper proposes a time-series classification method which exploits temporal consistency (implemented by contextual information). Also, the proposed method can handle noisy class boundaries.

Strengths

S1. This paper presents a novel problem formulation, time-series classification with noisy class boundaries. S2. The proposed method takes full advantage of temporal consistency, which is an inherent property of time series. S3. The authors created their own proprietary dataset, SEEG, for extensive experiments.

Weaknesses

W1. First of all, the writing in Introduction and Figure 1 are very misleading. "**Inconsistent** boundary labels" (with "different annotators") and Figure 1(a) indicate that multiple inconsistent annotations coexist for the same time-series instance. However, after reading a few pages, I found a discrepancy between my understanding and the presentation in this paper. Subsequently, I realized that the authors were referring to scenarios in which the class boundaries could be inaccurate. That is, there is **one** possibly erroneous boundary label for each transition. This presentation issue really affected my negative impression on this paper. W2. I do not agree that existing studies largely focus on the assumption of i.i.d. It is very easy to find the existing work that considers temporal consistency. For example, please see https://cs.stanford.edu/people/jure/pubs/ticc-kdd17.pdf. More recently, please see https://openreview.net/pdf?id=gjNcH0hj0LM. That is, many existing methods consider that consecutive timestamps tend to have the same class label. W3. Con-Attention is the same as Anomaly-Attention. Please see Equation (2) in https://arxiv.org/pdf/2110.02642. Therefore, along with W2, the technical novelty for the first contribution is not very significant. W4. Section 2 is not tightly connected to subsequent sections. In my opinion, Theorem 2 is too obvious, and it does not need to be formulated as a theorem. In fact, it does not look like a formal theorem. W5. The Tanh function fitting seems to be needed for each segment boundary and is not straightforward (involved with many parameters). Thus, the training efficiency could be an issue. It would be good to report the training cost of the proposed method. W6. The authors assume that the center of a class interval has the highest confidence. This assumption could be true or false depending on the domain. As in the motivating example, if the annotators have a difficulty in identifying the end times of seizure waves, the highest confidence will appear earlier than the center. W7. The authors made several strong assumptions or heuristics. For example, it is assumed that consecutive segments almost span at most two classes; a class interval is divided into five levels; a training cap is given as five epochs. Thus, this paper lacks rigorousness and generalizability. W8. It is not trivial to set up the hyperparameter value, especially, $E_\eta$. The default value is not guaranteed to work well for other datasets. A detailed guideline is necessary. W9. The degree of noisy labels in each dataset should be analyzed in detail. First, it is not clear why there is no noise in fNIRS and Sleep, while there is real noise in SEEG. Is the annotation strategy different in these datasets? How do you know the ground-truth labels in SEEG? Second, can you measure the degree of real noise just like the noise rate in the image classification domain? Can you approximate the corresponding value of $r$ for this degree of noisy labels in SEEG? Overall, without detailed information on noisy labels, it would be difficult to analyze the effect of noisy label learning. --- I have adjusted my rating to 5 during the discussion period.

Questions

See W1~W9.

Rating

5

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

In my opinion, the generalizability of the proposed method is not clearly verified. In this regard, the authors did not properly discuss the limitations of the proposed method. Please see my comments to further improve the manuscript.

Summary

This paper addresses the challenge of segmented time series classification (TSC) for Multiple classes with Varying Duration (MVD) data. The authors propose Con4m, a consistency learning framework that leverages contextual information with a focus on inconsistent boundary labels. The method incorporates continuous contextual representation encoding, context-aware coherent class prediction, and a label consistency training framework. Experiments on 3 datasets demonstrate Con4m's superior performance compared to state-of-the-art baselines, especially in handling inconsistent labels.

Strengths

- Originality: The paper introduces a new approach to segmented TSC for MVD data, addressing challenges often overlooked in existing TSC models. The Con4m framework creatively combines ideas from curriculum learning, noisy label learning, and temporal action segmentation. - Quality: The theoretical analysis in Section 2 and throughout provides a solid foundation for the proposed method. The experiments are comprehensive, including comparisons with various baselines, label disturbance experiments, and ablation studies. - Clarity: The paper is well-structured and clearly written. The figures, especially Figure 2 and Figure 3, effectively illustrate the proposed method's architecture and workflow. - Significance: The work has potential implications for various domains dealing with MVD data, such as healthcare and activity recognition. The label harmonization approach could be particularly valuable in scenarios where obtaining consistent labels is challenging or costly.

Weaknesses

- Limited exploration of hyperparameters: The paper doesn't thoroughly discuss the sensitivity of the method to key hyperparameters, such as N_l = 5 in the label consistency framework or E_g = 5 for curriculum learning. - Scalability concerns: The paper doesn't address how the method would scale to larger datasets or longer time series. This could be a limitation for practical applications with high-dimensional or long-duration data. - Comparison with semi-supervised methods: Given that the method deals with inconsistent labels, it might be beneficial to compare it with semi-supervised learning approaches that are designed to handle partially labeled or noisy data. - Generalization to other domains: While the method is tested on three datasets, they are all from the healthcare domain (all 3 of them focusing on brain signal analysis). It would be valuable to see how well the approach generalizes to other domains with MVD data, such as activity recognition or financial time series. For example, initiatives like WOODS (https://arxiv.org/abs/2203.09978/) offer diverse timeseries data for evaluations.

Questions

1. In the label consistency framework, why is N_l = 5? Was this value determined through hyperparameter search? How sensitive is the method to this choice? 2. Why was fNIRS not used in the ablation studies? Is there a specific reason for excluding this dataset from the detailed analysis? 3. How does the computational complexity of Con4m compare to the baseline methods? Is there a significant increase in training time or memory requirements? 4. Have you explored the potential of applying Con4m to semi-supervised or few-shot learning scenarios, where only a portion of the data has reliable labels? 5. The paper mentions that Con4m modifies approximately 10% of the training labels in the SEEG data. How does this percentage vary across different datasets or disturbance ratios? Is there a way to estimate the optimal percentage of labels that should be modified? 6. Could you provide more insights into the choice of the hyperbolic tangent function for prediction behavior constraint? Have you experimented with other monotonic functions, and if so, how do they compare?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

See the questions above. I would especially emphasize the narrow benchmarks and tasks. Learning across segments is a broader problem within the timeseries domain.

Summary

The paper proposes Con4m, a novel framework for time-series classification and temporal action segmentation that leverages contextual information. The framework is designed to improve the prediction accuracy by incorporating context from surrounding data segments. The proposed method combines time-series classification (TSC) and temporal action segmentation (TAS) in a unified framework, making it versatile for different types of sequential data. It tries to address the issue of common time series classification methods which overlook the dependencies between consecutive segments. The method demonstrates robustness to label disturbances, showing significant improvements over baseline models across different datasets (fNIRS, Sleep, and SEEG).

Strengths

- The paper proposes context-aware boundary detection and classification for long multi-class time series. The topic is very interesting and focuses on a real challenge in processing time-series data. - Con4m integrates contextual information to improve the coherence and accuracy of predictions. This approach helps in better recognizing boundaries and transitions in time-series data. - The paper is well-written and presented with several example illustrations. - The model is Compared against different time-series classification baselines. The evaluation shows great performance across Sleep and SEEG datasets and demonstrates robustness to label disturbances

Weaknesses

1. What if combining other models with a change-detection model to detect segment boundaries first and then classify the windows within the segments? That would be great to see the impact of Con4m in comparison with other baselines in this setup. 2. Since Con4m is capable of doing segmentation and classification together, it will be necessary to evaluate and compare the impact of each with other SOTA. Minor change: - Figure 4(b) has been referred to before Figure 4(a). Please fix that 3. The experiments are limited, and somewhat unfair. The chosen baselines do not claim ability in segment detection. There are several works that have done segment or change point detection, and they need to be included in baselines. Also, evaluations are limited to three datasets. - Minor suggestion: It would be nice to visualize the whole framework to show each module.

Questions

See the initial two comments (no 1 and no 2) in weaknesses. In addition: 3. In Label Disturbance Experiments: It is not clear how the authors evaluated the models. Is it based on the new disturbed boundaries or the original boundaries? What is the evaluation setup for the other baselines as they are not considering the labelled boundaries in their calculations (I assume they should be boundary-agnostic). 4. While section 4.3 shows an interesting set of experiments, I believe it only measures the degree of agreement between the baseline models and Con4m about the modified labels. Unless authors can ensure all the labels modified by Con4m are correct and accurate

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The following limitations are not considered/discussed: - Although I appreciate the novelty of the method, the final improvement is subtle compared to other baselines given they have not claimed any ability in segment detection. Also, the evaluations are limited to 3 datasets. - As the authors mentioned in the conclusion the model heavily relies on the availability of labels even for the segmentation part. While most segmentation models are unsupervised, I suggest including comparison by combining other baselines with unsupervised segmentation methods.

Summary

The authors proposed a learning framework called $\textit{Con}4\textit{n}$ that leverages contextual prior of Multiple classes with Varying Duration (MVD) to enhance the discriminative power of consecutive time series segments while harmonizing inconsistent labels associate to these later. The authors stated and shown through extensive experiment that their framework enables to encode underlying temporal dependence between consecutive time series segments and mitigate the data annotation error that may be due to discrepancy of expertise.

Strengths

- Originality: The authors propose a sophisticated architecture for encoding the temporal dependency between successive time series segments during a classification task. The originality of this work lies in the fact that the classification process is reinforced by a prior contextual information encoded by a Gaussian kernel. Their approach is more realistic than previous work, which assumes an independent and identical distribution of successive time series segments. - Quality: The article is well-written and structured. The experiment is consistent enough; - Clarity: Although the document is fairly clear overall, there are still a number of areas for improvement (please refer to weaknesses and questions); - Significance: The proposed solution, once mature, will undoubtedly benefit many domains in which time series classification can be very challenging due to the complexity of data structure and divergence of experiences.

Weaknesses

- Figure 2 is not intuitive enough to understand the overall architecture of the model; - $\hat{p}$ and $\tilde{p}$ are mentioned without explanation. Even though for $\hat{p}$ it is quite obvious, it must be explained clearly (see paragraph $\textbf{Neighbor Class Consistency Discrimination}$. - Space and Time complexity: Although efficient, the proposed model may require significant memory and computational time in large-scale applications. - The model may fail when confronted with seasonal time series data. This is because the Gaussian kernel used is not suitable for capturing the periodic pattern of the data; - Execution time is not included in the results. This is an important measure to report, as it plays a crucial role in sensitive areas such as medicine, where accuracy and prediction time are just as important.

Questions

- What features do you use in positional encoding? Is it the order index of time series segments or timestamps? Please mention it in the main text; - Can the model work with multivariate time series data, i.e. using the observation of various features in a time segment to perform a classification task? If so, how will you encode the multivariate time series segments? - Do the authors think that their model will still work correctly when faced with seasonal time series data, given that the Gaussian Kernel used is not appropriate for capturing the periodic pattern of the data? - Why did the authors choose the attention mechanism to merge $z_s^l$ and $z_g^l$? Although probably less efficient, have they tried simple techniques such as concatenation, Hadamard product or addition, which have the advantage of reducing spatial and temporal complexity?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

Although the authors have discussed the limitations of their work, the space and time complexity of the model is an additional limitation. Furthermore, unless the reviewer has missed some points, it appears that the proposed model has only been evaluated on univariate time series data (please correct the reviewer if he is wrong). The reviewer suggests investigating these points in future work.

Results for unsupervised segmentation model ClaSP

Apologies for the delayed results. For the F1 score provided by ClaSP, we input each time interval into ClaSP to obtain a score, and then average the scores of all time intervals. Additionally, we transform Con4m's predicted results from time segments to timestamps and use them to calculate the F1 score proposed by ClaSP. The specific scores are as follows: ClaSP: 0.854; Con4m: 0.930. The comparison of these results is quite fair. To align with our evaluation metrics, we assign all possible class combinations to the timestamps on either side of the change points detected by ClaSP. We then select the combination with the greatest overlap with the true test labels for ClaSP. Finally, the same segmentation and evaluation processes are executed to obtain the results. The specific f1 scores are as follows: ClaSP: 0.824; Con4m: 0.720. While this approach provides a test result, it clearly involves test label leakage, rendering it unfair and unsuitable for real-world applications.

Satisfied with the answers.

Dear authors, thank you for your detailed responses. My concerns were clarified.