Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series

Contrastive representation learning is crucial in medical time series analysis as it alleviates dependency on labor-intensive, domain-specific, and scarce expert annotations. However, existing contrastive learning methods primarily focus on one single data level, which fails to fully exploit the intricate nature of medical time series. To address this issue, we present COMET, an innovative hierarchical framework that leverages data consistencies at all inherent levels in medical time series. Our meticulously designed model systematically captures data consistency from four potential levels: observation, sample, trial, and patient levels. By developing contrastive loss at multiple levels, we can learn effective representations that preserve comprehensive data consistency, maximizing information utilization in a self-supervised manner. We conduct experiments in the challenging patient-independent setting. We compare COMET against six baselines using three diverse datasets, which include ECG signals for myocardial infarction and EEG signals for Alzheimer's and Parkinson's diseases. The results demonstrate that COMET consistently outperforms all baselines, particularly in setup with 10% and 1% labeled data fractions across all datasets. These results underscore the significant impact of our framework in advancing contrastive representation learning techniques for medical time series. The source code is available at https://github.com/DL4mHealth/COMET.

Paper

References (53)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer p5Bk6/10 · confidence 4/52023-06-30

Summary

This paper investigates the contrastive learning for medical time series and develops multiple contrastive objectives at different levels (patient, trial, observation, sample). The downstream applications is the classification on EEG, ECG, EMG, EOG signals. The proposed method has obtained the best or competitive performance on two EEG datasets and one ECG dataset.

Strengths

- It is interesting to see a hierarchical view for a specific type of medical time series, such as ECG and EEG. This hierarchical multi-level view captures the unique characteristics of this type of time series. - Given this novel view, the proposed contrastive learning at all levels is straightforward and reasonable. - The paper is well written and easy to understand.

Weaknesses

- My first concern is the discrepancy between the title and the actual methodology. "multi-level contrastive learning" sounds a better choice than "multi-granularity representation learning" because event at the patient or trial level, the representation learning still happens at the sample-level input, and you do not squeeze or aggregate representations at higher levels.Thus, I feel there is no multiple granularity. Please correct me if I misunderstand your method. - The second concern is that the proposed method is only evaluated on limited datasets. It seems that the method only works on EEG-related datasets (AD and TDBrain), beating other baselines significantly, but is modest on PTB, an ECG dataset. I would like to see broader empirical demonstrations and analyses. - Moreover, the ablation tests are only conducted on the AD dataset, on which the proposed method has achieved the most performance improvements. I would like to see more detailed ablation tests and analyses on other datasets.

Questions

Check the weakness part.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Lack of sufficient verifications on large-scale and more diversified datatsets.

Reviewer jbn15/10 · confidence 4/52023-07-02

Summary

This paper proposes a multi-granularity contrastive learning method (named COMET) on medical time-series data. The methods build postive and negative pairs from patient-level, trail-level, sample-level, and observation-level. The proposed method is evaluated on two EEG, and one ECG dataset.

Strengths

1. The paper is well organized and clearly written. Figure 2 is very informative. Related works are also comprehensive. 2. The performance of the proposed method is decent based on the experimental tables.

Weaknesses

1. The proposed method is not new and many multi-granular contrastive learning models have been proposed in the past few years. Based on Figure 1, CLOCS, TS2Vec, TS-TCC are also multi-granular contrastive learning methods specifically in medical timeseries domain. The contribution and extension (from existing two granular methods to the proposed four granular method) of this paper seems trivial given these exisiting methods. 2. Overall, the proposed COMET method can be viewed as "a more finer-granular way of doing data augmentation". In that sense, the data augmentation methods for all models (including baselines) seem different, then does the experimental design fair? 3. The considered two EEG and one ECG datasets all seems small in medical timeseries domain. What is the number of parameters in the model? Usually, the training data size is expected to be 10 times larger than the number of parameters. Contrastive learning methods naturally need a lot more unlabeled data to train. SHHS is a large EEG dataset, and the ECG 2020 challenge datasets are also large https://physionet.org/content/challenge-2020/1.0.2/. 4. Although the authors have gave another example on satellite sensor applications, the generalizability of the proposed model is still questionable. Given that many multi-granular contrastive learning models have been developed in other domains as well, could the authors distinguish this paper and highlights the key features of COMET if generalizable to other domains?

Questions

see above

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NA

Reviewer RVrY8/10 · confidence 5/52023-07-03

Summary

Medical time-series data, unlike domains such as CV and NLP, lack data labeling but contained more layers of information corresponding to observation, trial, and individual physiologies. Unlike previous methods that overlooked the multi-granularity, the authors proposed COMET to trained the combined contrastive losses from all 4 granularities and presented COMET’s elevated performance on multiple datasets, outperforming many up-to-date contrastive baselines. With ablation studies and disclosed code repo, the authors evaluated the effectiveness of proposed COMET and concluded it as a pioneering framework tailored for medical time series.

Strengths

• Figures are well done and make it much easier to understand what the authors are talking about when they mention the different levels of granularity for medical time series data • The paper is very well written and easy to understand. The method seems simple but this is likely due to the authors’ explanation of information granularity. The paper follows a clear logical flow. • Capturing multi-granularity information with contrastive learning is a very good idea backed by performance improvements. • They backed their proposed COMET with comprehensive experiments on multiple datasets with many up-to-date baselines • The performance of their method is very high in comparison the methods they consider to be related work.

Weaknesses

• While the generalization is likely to follow from this work - this method is somewhat limited to ECG data. ECG data has many structural patterns in medical time series data that come from the cyclical nature of a cardiac cycle that other medical time series might not. As a result, the methods might actually be limited to pulsatile-type medical time series, which would still be a contribution. For example On page 2, line 71, the statement that medical time series is "low cost and non-invasive." would not necessarily be correct for data collected from invasive A-line, for example. However, I mostly agree with the paragraph on EEG, ECG, etc., and how they benefit healthcare practices. I would suggest authors better approach this limitation which does not take away from the contribution of the work overall. • While many works are compared, other ECG-related methods, one of which they mention in the paper, CLOCs have not been compared. How does this method compare to ECG specific methods? • It was unclear how the time series features were fed into the COMET and how they were broken into different granularity and evaluated by different contrastive losses (see first bulletins of questions). • It would be better to include ablation study on the process of discovering the optimal set of hyper-coefficients for each dataset. • On Page 6 line 215, D is defined as dataset, but it should be better defined before referenced, namely in Section 4.2 or earlier?

Questions

• Can authors address the limitations? is this work specific to cyclical time-series data? • How do the results compare to other ECG contrastive pretraining methods? • Reading the “comet.py” and “models/encoder.py’s TSEncoder”, it seemed like the same inputs “x” were used to calculate 4 levels of contrastive losses. Then what exactly is “x” (the time series input of the COMET), is it observation-level? Sample-level? Can authors better describe the actual inputs to the system?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

• Only addresses one modality of medical timeseries data • Does not compare to other ECG contrastive pretraining methods These have not been adequately addressed.

Reviewer Trt34/10 · confidence 5/52023-07-07

Summary

This paper proposes a multi-granularity framework leveraging data consistencies at different levels inherent in medical time series data. The model learns with contrastive loss designed at every data granularity, i.e., observation, sample, trial, and patient levels. The method is evaluated with three binary classification downstream tasks, showing significant gain over SOTA timeseries contrastive learning methods.

Strengths

1. The idea of leveraging data consistencies are multi-granularity is interesting and seemingly effective. 2. The writing and presentation is easy to follow.

Weaknesses

1. The contrastive learning for each granularity is simple and straight forward, making the novelty of the paper limited. 2. Majority of the medical tasks are fine-grained in nature. That is to say, the difference between two samples of difference classes may be very subtle and local. It is unclear how the data augmentation method for each granularity is designed to ensure not disturbing the sample label. 3. The evaluation is a bit unpersuasive. The downstream tasks are all simple binary classification problems (e.g. for dementia detection, Myocardial infarction, or Parkinson’s disease). 4. For medical diagnosis tasks, accuracy and F1 shall not be used as the main metric as the data distribution is usually drastically imbalance. I would suggest either focus on AUROC or AUPRC. In this case, COMET shows inferior performance on PTB than TS2vec. Any explanation for this results?

Questions

1. The difference between ‘sample’ and ‘trial’ are a bit unclear. It seems that we can also consider a trail as a sample with a longer time span? How to segment trials into distinct samples? ”Each sample is a one-second interval with 256 observations. ” – Any justification for choosing one-second interval here? 2. I am curious about the implementation details for Observation-level data consistency. How $t^-$ is chosen? Shall it be close to $t$ or distant from $t$? How to guarantee that $x_{i, t^-}$ is more different to $x_{i, t}$ than $\tilde{x}_{i, t}$, as the time series data often display repetitive periodicity? 3. When jittering and masking are used for data augmentation, would it broken the continuity of the normal time series data? 4. The decision of using InfoNCE at observation and sample blocks while using NT-Xent loss at trial and patient level seems to be very arbitrary. Any explanation for it? 5. SOTA comparison: why NCL and CLOCS are not considered? 6. Any ambulation study on the hyperparameters $\lambda_i$ is missing.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

the limitations are presented in the appendix.

Reviewer RVrY2023-08-15

Thank you for your comments. I think these clarifications - and the new baselines - are valuable additions.

Area Chair SinJ2023-08-15

please engage in discussion

Hello reviewers. It looks like right now the reviews are a bit divergent. If you have not already done so, please read the authors' response to your reviews *and to the other reviewers' reviews*, and discuss. The authors have provided reasonably detailed rebuttals so it would be great if you engage in discussion as soon as possible, especially to indicate if your opinion of the paper has changed or if you would like any sort of additional comments or clarifications. Thanks, Area chair

Reviewer p5Bk2023-08-17

Thanks for the reply

"Hierarchical contrastive learning" better aligns with the underlying approach of this paper. This paper has nothing to do with multi-granularity representations. Using "multi-granularity representation learning" can be very misleading. More datasets and more ablation results look good. I would only like to raise the score to "weakly accept" because the current version needs to be modified a lot to fit for the new theme, "hierarchical contrastive learning".

Authorsrebuttal2023-08-17

Thanks for the reply

Thank you again for improving the score! Thank you for your constructive suggestion on our paper’s title. We will change our title to “Hierarchical Contrastive Learning for Medical Time-Series.” Besides, we will thoroughly modify our terms/notations/figures/text to keep the paper consistent and easy to follow. For example, we plan to update all “granularity” into “level” to reduce the potential misunderstanding caused by “granularity” in the camera-ready version.

Reviewer Trt32023-08-17

Thank the authors for the detailed response as well as the added experiments, which have answered part of my concerns. However, first, I do not agree with authors that "the majority of real-world applications of medical time series for disease diagnosis are binary classification tasks". In fact, real-world medical applications must be able to handle multi-class classification. Secondly, I would suggest the authors to refine their notations. In particular, it is quite confusing to use the same notation x to indicate both observation, sample, and trial. Thirdly, both the sample-level consistency and the trail-level consistency are comparing at sample level. The difference lies in how the triplets are generated. Why the former use InfoNCE loss while the later use NT-Xent loss? The decision seems to be quite ad hoc. Given the above, I have raised my rating to 4.

Authorsrebuttal2023-08-18

Further explanations for your three comments

We thank the reviewer for the active feedback and give us the opportunity to further explain to your concerns. **New Q1**: I do not agree with authors that "the majority of real-world applications … are binary classification tasks". In fact, real-world medical applications must be able to handle multi-class classification. **New A1**: 1. We highly agree with the reviewer that real-world applications should be able to handle multi-class classification. Although the current dominant trend of AI research for medical time series centers around simple binary tasks [1-4], yet complexity of medical data and the diversity of clinical applications call for more comprehensive approaches. 2. In response to your concern, we have demonstrated the efficacy of our model in addressing multi-classification challenges. We have conducted an additional experiment over a large-scale ECG dataset (PTB-XL, 17596 patients, 191400 samples) with 5 classes (See A4 and R-table 2). 3. We acknowledge it's helpful to have more studies and analyses in terms of multi-class tasks. Will discuss it in our limitations of this paper, and list it as an important future work. [1] Liang, H., et al., 2019. Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. Nature medicine,. [2] Hicks, S.A., et al., 2022. On evaluation metrics for medical applications of artificial intelligence. Scientific reports. [3] Buch, V.H., et al., 2018. Artificial intelligence in medicine: current trends and future possibilities. British Journal of General Practice. [4] Soenksen, L.R., et al. 2022. Integrated multimodal artificial intelligence framework for healthcare applications. NPJ digital medicine. **New Q2**: Refine their notations. In particular, it is quite confusing to use the same notation x to indicate both observation, sample, and trial. **New A2**: We appreciate your attention to the notations used in our work. First, we clarify we didn’t use the same notation to indicate all of observation, sample, and trial. In this paper, our $x$ consistently represents a sample, while $r$ and $p$ stand for trial and patient, respectively. - Within the sample level, $x_i$ denotes a sample with index $i$, and in the observation level, $x_{i, t}$ signifies an observation within sample $x_i$ at timestamp $t$. The subscripts $_i$ and $_t$ denote the $i$-th sample and the $t$-th observation. - However, at the patient and trial levels, the subscript $i$ is unnecessary as the sample index doesn't matter here because we mainly care about how the positive/negative sample pairs are formulated. Here, the focus shifts to whether the sample $x$ originates from the same patient or trial. So, to simplify the notations, we omitted the subscript $_i$ in patient and trial levels, Instead, we use superscript $^+$ and $^-$ to denote the positive and negative samples, respectively, to emphasize the sample pair construction is more important than sample index. Second, we struggled in designing our paper's notation for a long time. We explored various plans, such as using different subscripts and superscripts to represent different levels (e.g., let $x_{s,o}^{p,t}$ denote the $o$-th observation from the $s$-th sample of the $t$-th trial of the $p$-th patient). However, we found such notations are overly complex and difficult to comprehend. Consequently, we opted for a simpler notation scheme (i.e., the one we presented in the paper) to make the whole model easier to follow. Your attention to our notation design is greatly valued, and we truly welcome any suggestions and advice to help us further polish the notations. We remain committed to refining our notations to ensure they are accessible and intuitive. **New Q3**: ....Why the former use InfoNCE loss while the later use NT-Xent loss? ... **New A3**: This is the Q8 of your original comments, which we have answered in rebuttal. Due to the space limitation in response, we have presented the answers to your Q7-Q10 in the global response (prior to the references). For your convenience, we paste our answer here: - We observed that TS2Vec performs effectively for observation and sample levels. Concurrently, CLOCS introduced contrastive learning on ECG data, utilizing patient-level information, which bears similarity to our implementation at the trial level. Thus, we adopted their loss functions as a foundation for designing our own loss functions across distinct granularities. - Moreover, we clearly understand InfoNCE and NT-Xent are all derived from NCE loss family. We conducted preliminary experiments to compare using InfoNCE at four granularities, using NT-Xent in all granularities, and exchangeable use InfoNCE or NT-Xent at different granularities. The experimental results show InfoNCE and NT-Xent will lead to very similar outcomes while the NT-Xent can converge a bit faster. We will discuss this in our limitations and future work.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC