Previous deep learning approaches for survival analysis have primarily relied on ranking losses to improve discrimination performance, which often comes at the expense of calibration performance. To address such an issue, we propose a novel contrastive learning approach specifically designed to enhance discrimination \textit{without} sacrificing calibration. Our method employs weighted sampling within a contrastive learning framework, assigning lower penalties to samples with similar survival outcomes. This aligns well with the assumption that patients with similar event times share similar clinical statuses. Consequently, when augmented with the commonly used negative log-likelihood loss, our approach significantly improves discrimination performance without directly manipulating the model outputs, thereby achieving better calibration. Experiments on multiple real-world clinical datasets demonstrate that our method outperforms state-of-the-art deep survival models in both discrimination and calibration. Through comprehensive ablation studies, we further validate the effectiveness of our approach through quantitative and qualitative analyses.
Paper
Similar papers
Peer review
Summary
This work concerns deep learning for survival analysis. Deep learning models for survival analysis are typically desired to have good discriminative performance, where a model can differentiate between patients of different risk profile, and calibration performance, where the time-to-event is accurately predicted by the model. Methods which seek to improve discrimination performance have often led to poorer model calibration. Consequently, this paper proposes an auxiliary contrastive loss to improve the discriminative performance of deep survival models, while still achieving high calibration performance. The novelty in this method is adapting the SOTA contrastive learning method for tabular data, SCARF, by weighting the negative pairs according to the difference in time-to-event. Hence, samples with greater difference in time-to-event are considered more important negative pairs, enforcing an inductive bias to differentiate the latent representations of samples with different time-to-event. After providing a detailed problem formulation, the contrastive loss and generation of positive and negative pairs is described. The results of several experiments are provided which compare the proposed method with current state-of-the-art methods when applied to a number of real world datasets, and demonstrate the proposed model performance under ablation of each term in the loss function. From these experiments it is concluded that the auxiliary contrastive loss achieves superior discriminative and calibration performance.
Strengths
The strength of this work lies in its originality, quality and clarity. The paper proposes an interesting adaptation to SCARF, a contrastive method for tabular data, by weighting the negative pairs such that samples with greater difference in time-to-event are considered more important negative pairs. Recent work on contrastive learning and deep survival methods are cited and appropriately discussed. Technically, the paper is sound, with a good mathematical foundation for the importance sampling in the weighted contrastive loss term, and a thorough set of experiments which establishes performance against important baselines and provides good support for modelling choices via ablation studies. Furthermore, the clarity of the paper is for the most part good: the description of the background, methodology, and experiments are clear and detailed.
Weaknesses
While the technical parts of this paper are clearly explained, the organisation and presentation of the paper could be improved. In particular, the introduction uses quite a lot of terminology specific to survival analysis that is unexplained or explained in detail only later in the paper. As a specific example, censoring is first mentioned on line 67, but is not clearly explained. Explaining this terminology, which may not be familiar to all of NeurIPS wide readership, at first use would improve the clarity of the paper. Presentation and clarity of the paper would also be improved if all captions were self-contained. In some cases, for example in Table 1, it is difficult to interpret the results as presented, and this could have been avoided if the captions were more detailed. Specifically, it is not clear what the standard deviations represent in this table i.e. variation across model initialisations or data splits. This work should be praised for providing a list of experiments that should provide the evidence needed to make the claim that the proposed model provides superior discriminative performance while maintaining calibration performance. However, the clarity and quality of these experiments is sometimes below expectations, and this ultimately reduces the overall significance of this work. Table 1 is used to report that the proposed model is the best performing model. However, it seems that the model is within 1 standard deviation of the second best performing model, and hence the statistical significance of this result is unclear. As previously mentioned, while mean and standard deviation of performance metrics are reported, it is not entirely clear what distribution they refer to – whether it be different data splits, or model initialisations via random seed. This makes it difficult to interpret the significance of these results, and also makes it more difficult to reproduce. The paper, therefore, addresses an important problem, proposes a novel and well thought out method to address this issue, but the significance is undermined by a lack of clarity and ambiguity surrounding the statistical significance of reported results.
Questions
The questions below focus on the details of the experiments that were unclear. Clarity on these questions may change the overall opinion of the paper. Table 1: In the caption you refer to the standard deviation - are the reported values in this table the mean? Table 1: Does the standard deviation refer to variation over different data splits, model initialisations or something else? Table 1: What do bold and underlined values represent? Table 1: How do you determine that the superior performance of ConSurv is statistically significant? Line 528: Can you confirm whether min-max normalisation was performed before or after data was split into train, test and validation sets? Line 236: What is the marginal corruption process? By what augmentation were positive pairs generated? Line 197: What does the temperature coefficient scale? What impact does this have on model training? Line 104: Does ref. [18] show that model training is destabilised by the contrastive loss? If not, what supports this statement? Could this model have been included in your benchmarks? NOTE: Due to the discussion in the author-reviewer discussion period, I feel that many of these points were clarified. In particular new results have better demonstrated the statistical significance of the performance of ConSurv. I raise my score from a 4 to a 6 to reflect this.
Rating
6
Confidence
4
Soundness
2
Presentation
3
Contribution
3
Limitations
The authors have addressed the potential negative societal impact of their work, stating that model predictions from such work should be under scrutiny by domain experts before being applied in a clinical setting. They stress that incorrect use could lead to inequitable access or discriminatory practices. While this is welcome, it is not specific to this work, but is a concern of all works proposing machine learning for clinical practice. A potential impact more specific to this work might consider why the inductive bias introduced by the auxiliary contrastive loss would lead to unintended consequences for AI-supported decisions on treatment and intervention. For example, the unsupervised nature of the contrastive learning procedure could align patients based on features which may act as a proxy for protected traits and lead to inequitable model performance amongst different demographics. Additionally, a discussion of limitations is provided but is limited. The authors correctly identify that a significant number of patients are censored in real world datasets, and this undermines the importance sampling based on difference in time-to-event, as weights are assigned according to a function in equation (5). It would improve the discussion of the limitations if it was pointed out that seeking alternatives or amendments to (5) would be a specific avenue to improve upon this work. The limitation section could benefit from greater specificity. The authors propose “modifying models to account for such uncertainties or developing alternative learning methods”. The uncertainties they refer to are unclear, and a preferred statement might be: future avenues of work would be to develop a model which retains the benefit of contrastive learning with respect to discriminative performance, whilst also being agnostic to the number of censored samples within the training set. They then state “there is a need for new evaluation metrics or model developments that consider the characteristics of such data”. The clarity of this proposal might be clearer if stated with more specificity i.e. "to develop a model that retains high discriminatory and calibration performance for datasets with arbitrary proportions of censored samples, new metrics must be established that can compare the same model applied to different proportions of censored samples."
Response to Reviewer VjbS
# [ Response to Reviewer VjbS ] We appreciate the reviewer for the feedback and acknowledging our efforts in addressing the reviewer’s comments. Here, we would like to provide further clarification on the following points: **Normalization and Data Leakage**: We apologize for any confusion caused by our initial explanation of the min-max normalization process. To clarify, we performed the min-max normalization **after** splitting the dataset, **not before**. This inadvertent miscommunication may have raised concerns about potential data leakage. We assure you that our method prevents any leakage, as normalization is conducted based solely on the training and validation sets. For verification, this procedure can be confirmed in the supplementary $\texttt{dataset.py}$ file that was submitted at the initial submission deadline. **Statistical Significance of the Performance Gain**: We would like to start by emphasizing that our work introduces a deep survival model leveraging contrastive learning to enhance discriminative power without sacrificing calibration. Therefore, a fairer comparison showcasing our contribution would involve comparing our model with i) variations of our model trained solely with the NLL loss or with both the NLL and ranking losses, and ii) deep survival models utilizing traditional ranking loss. In this context, we have evaluated the statistical significance of our results based on the p-values calculated by the Welch’s t-test. As shown in Table G, ConSurv achieves superior discriminative power (while maintaining its calibration) over its variant trained solely with the NLL loss. The results are statistically significant except for the NWTCO dataset. On the other hand, our variant trained with both the NLL and ranking losses often loses its calibration power. Contrarily, when compared with the variant trained with the NLL and ranking losses, ConSurv significantly outperforms in terms of calibration power for all the evaluated datasets while providing gains in discriminative power (but not statistically significant for the NWTCO dataset) except for the SUPPORT dataset. These results are consistent with our motivation described in the Introduction. A similar trend can be observed when compared with traditional ranking-based deep survival models (i.e., DeepHit and DRSA). While the gain in discriminative power is sometimes not statistically significant, our method provides statistically significant improvements in terms of calibration power, as shown in Table G. *Table G. p-value for the variants of our method* |Model|Metric|METABRIC|NWTCO|GBSG|FLCHAIN|SUPPORT|SEER| |-|-|-|-|-|-|-|-| |**NLL**|CI|**0.000**|0.308|**0.001**|**0.000**|**0.000**|**0.008** | ||IBS|**0.000**|**0.001**|0.174|0.194|**0.034**|**0.000**| |**NLL+Rank**|CI|**0.000**|0.424|**0.000**|**0.000**|-|**0.000**| ||IBS|**0.000**|**0.003**|**0.000**|**0.000**|**0.000**|**0.018** | **Analysis of Time-Dependent Performance Metrics**: While the CI and IBS provide valuable overall assessments of a given survival model (over the entire time horizon), these metrics cannot fully capture variations in model performance across different time points. To address this, we included time-dependent performance evaluations, namely the time-dependent C-index [R1] and the time-dependent Brier score [R2], in Appendix C.2, at three different time points (i.e., 25%, 50%, and 75% percentiles of time-to-events as in [R3, R4, R5]). It is worth highlighting that utilizing these time-dependent metrics may reveal subtle differences that might be obscured when using CI and IBS alone. Table H shows the number of statistically significant performance improvements over each survival model across different datasets. The improvement is considered significant if ConSurv provides statistically significant gains at at least one of the three time points. Again, we use the Welch's t-test to calculate the p-values for both time-dependent discriminative and calibration performance throughout the datasets. ConSurv offers superior calibration performance compared to ranking-based deep survival models (DeepHit and DRSA) and outperforms calibration-focused survival models (DCS and X-CAL) in discriminative power. Overall, compared to the deep learning baselines, ConSurv achieves statistically significant improvements in at least 4 out of 6 datasets, demonstrating its robust and consistent performance gains across various scenarios. *Table H. Number of Significant Performances Across Datasets* |Model|C-index|Brier score| |-|-|-| |CoxPH|3|1| |DeepSurv|6|6| |DeepHit|4|6| |DRSA|5|6| |DCS|5|5| |X-CAL|5|5| |SupWcon|5|4|
Response to Reviewer VjbS
**Evaluating Calibration from Different Perspectives**: We further provided the comparison in the model calibration by evaluating alternative calibration metrics -- Distributional Divergence for Calibration (DDC) [R6] and D-Calibration (D-CAL) [R7] -- that assess model calibration from a slightly different perspective. These metrics provide a more nuanced perspective on calibration quality, directly evaluating the alignment of predicted survival probabilities with uniform distributions. For the assessment of statistical significance of the performance gain in terms of model calibration, we have additionally included the p-values for the DDC in Table I. (Please note that the D-CAL test already incorporates p-values by its definition and thus is not considered for comparing the statistical significance of the performance gains.) Table I highlights that ConSurv achieves significant calibration gain over the benchmarks -- especially the ranking-based deep survival models -- across most of the datasets. *Table I. p-value for DDC* |Dataset|M|N|G|F|SU|SE| |-|-|-|-|-|-|-| |CoxPH|-|**0.012**|-|**0.018**|**0.000**|**0.017**| |DeepSurv|**0.030**|**0.000**|-|**0.018**|**0.000**|**0.000**| |DeepHit|**0.000**|**0.000**|**0.000**|**0.000**|**0.000**|**0.005**| |DRSA|**0.013**|-|**0.000**|**0.049**|**0.000**|**0.001**| |DCS|0.341|-|**0.000**|0.318|**0.000**|-| |X-CAL|0.926|-|0.964|**0.016**|**0.002**|-| |SupWcon|-|**0.005**|-|**0.043**|**0.045**|**0.000**| --- ## *Reference* [R1] H. Uno et al., “On the C‐statistics for evaluating overall adequacy of risk prediction procedures with censored survival data,” Statistics in Medicine, 2011. [R2] E. Graf et al., “Assessment and comparison of prognostic classification schemes for survival data,” Statistics in Medicine, 1999. [R3] Z. Wang et al., "Survtrace: Transformers for survival analysis with competing events," ACM-BCM, 2022. [R4] C. Nagpal et al., "Deep survival machines: Fully parametric survival regression and representation learning for censored data with competing risks," IEEE Journal of Biomedical and Health Informatics, 2021. [R5] C. Lee et al., "Temporal quilting for survival analysis," AISTATS, 2019. [R6] F. Kamran et al., "Estimating calibrated individualized survival curves with deep learning," AAAI, 2021. [R7] H. Haider et al., “Effective ways to build and evaluate individual survival distributions,” JMLR, 2020.
Thank you to the authors for taking the time to reply again and provide further discussion and clarification. I have no further questions. Thank you for pointing me to the relevant code - I agree that min-max normalisation has not led to data leakage. And thanks for clarifying how your results demonstrate a statistical significant improvement over current baselines. I find the arguments above convincing.
Thanks
Dear Reviewer VjbS, We are sincerely grateful for your time and energy in the review process. In light of your satisfaction with our response, we wonder whether the reviewer would kindly consider revising the rating. Thank you, Paper 15583 Authors
I have happily raised my score from a 4 to a 6 in agreement with one of the other reviewers.
Thanks
Dear Reviewer VjbS, Thank you so much for raising the score from 4 to 6. We kindly ask if you could please ensure that the updated score is reflected in the main system. We greatly appreciate your consideration. Thank you, Paper 15583 Authors
reminder from your area chair to respond to this author feedback
Hello reviewer VjbS, Thanks for already engaging in discussion with the authors on this paper. The authors have responded in detail to your most recent comments. Please try to provide a response before the end of the author/reviewer discussion period (Aug 13 11:59pm AoE). Thanks, Your AC
Summary
The authors study discrete-time survival analysis, proposing to train models using a loss function that combines the NLL loss and a modified NCE loss. This NCE loss is modified to take the survival outcomes (event times) into account, mitigating the effect of potential false negatives that have small event time differences. The method is evaluated on 4 tabular clinical survival datasets, comparing against 6 baseline models and 3 method variations. The proposed method performs well compared to all baselines.
Strengths
- The paper is well-written overall. - The proposed method is described well in Section 4. - The proposed method is conceptually quite simple and intuitive. To add a contrastive loss term, and for this use a modified contrastive loss that takes the survival outcomes into account, makes intuitive sense. - The proposed method seems to perform well compared to baselines, both in terms of discrimination and calibration. In particular, it performs well compared to the baseline of replacing the proposed modified NCE loss term with standard NCE.
Weaknesses
- The technical novelty/innovation is perhaps somewhat limited ("just" adding a contrastive loss term). - The experimental evaluation could perhaps be more extensive, include more datasets. Would have been nice to see the method being applied also to some non-tabular dataset. Summary: - Somewhat limited technical novelty, but this is a well-written paper with a simple/intuitive method that seems to perform well compared to reasonable baselines. Thus, I am definitely leaning towards accept.
Questions
- Could you perhaps apply the proposed method also to some non-tabular dataset? - The visualization in Figure 2 is really neat, could you add this for the three other datasets as well (to the appendix)? Minor things: - 130: "by minimizing the following negative NLL loss", remove "negative"? - Perhaps add 1-3 sentences to the start of Section 3, just to describe what is covered here (and why this is covered, why this is relevant for your method)? - 248: "We compare our proposed method and the benchmarks with...", "benchmarks" --> "baselines"? The same also for line 251?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes.
Summary
The paper proposes a contrastive learning loss to regularize maximum likelihood learning of discretized survival times. The main contribution of this work is the utilization of the Laplace kernel to weigh negative pairs inversely proportional to their time difference from the anchor sample, while also accounting for censoring. Specifically, negative pairs with event times close to that of the anchor sample contribute less to the contrastive loss. Experimental results on four datasets demonstrate that the proposed approach achieves a high Concordance Index (C-Index) without a significant loss in calibration when compared to baselines.
Strengths
The use of contrastive learning to improve model calibration in survival analysis seems interesting. Experimental results demonstrate the efficacy of this approach in maintaining a high C-index without significant loss in calibration, a trade-off that is difficult to achieve in practice.
Weaknesses
*Given that the main contribution of this work is the application of contrastive loss to survival analysis, the paper lacks complete details for reproducibility:* - There are no details on how the model functions $ f_{\theta}, f_{\phi}$, and $g_{\psi} $ are parametrized. - Details on the construction of positive pairs are very sparse. No motivation is provided in terms of why the SCARF approach is chosen and why it should be expected to work with survival data. - The paper proposes the Laplace Kernel for weighing samples; it is unclear if alternative kernels were also considered. - The notation seems sloppy; symbols ${f}$ and $Z$ are overloaded. - I expect calibration and the C-Index should be sensitive to $\beta $; the paper does not discuss how to choose $\beta $ and what values resulted in the performance results reported. This phenomenon has been discussed in detail in [1]. I encourage the authors to include a comprehensive discussion on this trade-off. - It is unclear how the margin parameter $\alpha$ was chosen. I encourage the authors to provide visualizations in the main paper to demonstrate sensitivity to both $\alpha$ and $\beta $. *The proposed approach relies on discretizing event times, which is sensitive to the time-binning approach. Additionally, it seems the contrastive loss could be an indirect way to achieve consistency in event times. I encourage the authors to also benchmark with approaches that directly predict event times such as [2].* *The four datasets used seem to be of low sample size. I encourage the authors to also consider larger datasets such as SUPPORT and SEER.* **References** - [1] Qi et al., "Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration", ICML 2024 - [2] Chapfuwa et al., "Calibration and Uncertainty in Neural Time-to-Event Modeling", IEEE TNNLS 2020
Questions
- What are the values of $\alpha$ and $\beta $ for the reported results? - Were alternative weighting kernels considered? - Were alternative approaches for constructing positive pairs considered?
Rating
4
Confidence
4
Soundness
2
Presentation
1
Contribution
2
Limitations
- The paper does not discuss modeling assumptions; for example, the assumed independent censoring mechanism could be violated in practice.
reminder from your area chair to respond to this author feedback
Hello reviewer Lfcy, The author/reviewer discussion period ends very soon (Aug 13 11:59pm AoE). It would be great if you could respond to the authors as they've taken the time to respond to your review. Thanks, Your AC
Response to Reviewer Lfcy
# [ Response to Reviewer Lfcy ] **Regarding the Comparisons with DRAFT and DATE**: We appreciate the reviewer's insightful comments regarding the comparison of ConSurv with DRAFT and DATE. We opted to include DRAFT, not DATE, in our comparative analysis due to the following considerations: DRAFT posits a log-normal distribution for the underlying time-to-event process, $p(t∣x)$. This allows for a direct calculation of the survival function, $S(t∣x)$, facilitating a clear and equitable comparison using the performance metrics presented in our study. Conversely, while DATE represents a significant advancement in deep survival models by directly generating time-to-event outcomes, it employs a generator that learns the underlying distribution, $p(t∣x)$, implicitly. Consequently, computing $S(t∣x)$ necessitates sampling a substantial number of samples, complicating the comparison against the reported performance metrics. Therefore, we restricted our comparison to DRAFT as a deep learning counterpart that explicitly focuses on AFT models. **Regarding the comparison with CSD (Qi et al.’s Method)**: We appreciate the suggestion to compare ConSurv with CSD (Qi et al.'s post-hoc calibration method), but we believe a direct comparison is not within the scope of our work. Our focus lies in improving discriminative power while preserving calibration, a distinct advancement from traditional ranking losses. CSD addresses a post-hoc approach to improve calibration of survival models, making a direct comparison less relevant. Furthermore, Qi et al.'s work was published after our submission deadline and thus couldn't be included in our original analysis. However, we acknowledge its value and will incorporate a comparison in the appendix of our final version by applying their post-hoc calibration to existing ranking-based models. **The Connection Between Contrastive Learning and Model Calibration**: As the reviewer pointed out, Kamran et al. and Qi et al. have discussed the trade-off between discriminative power and calibration, where the trade-off arises because discrimination is typically assessed at an individual level, whereas calibration is evaluated at the population level. However, the parameter, $/lambda$, in the loss term proposed by Kamran et al. does not directly explain this trade-off. Instead, it controls the balance between two losses in the composite loss function, where each loss influences both the discrimination and calibration of the survival model. To the best of our knowledge, there is no theoretical proof on why utilizing the ranking loss may lead to poor calibration. We hypothesize that directly modifying the output of survival models (whether the hazard function in DRSA or the PMF of event times in DeepHit) can bias the network towards optimizing the approximated C-index, potentially sacrificing accurate modeling of event time distributions, as conjectured by Kamran et al. In contrast, our approach employs a contrastive learning framework that adjusts latent representations based on the similarity in the event times, implicitly promoting discriminative power without directly impacting the distribution of event times. Consequently, we were not able to observe a clear trade-off between the two loss functions, a phenomenon more commonly seen with traditional ranking losses. **Clarification on Notations**: We've taken steps to improve clarity by clarifying notations and providing comprehensive implementation details in both the appendix and this rebuttal. If you have specific areas that you feel still require attention, please point them out and we'll be happy to address them further.
Summary
This paper presents an approach to handle survival analysis datasets using a method to improve both calibration and discrimination. This adds a bunch of novelties to the field like handling right-censoring and an SNCE loss to handle calibration and ranking simultaneously (NLL + contrastive loss). This paper presents a reliable way to look at survival analysis, write a hazard function which is quite reasonable and also present a ranking loss to take care of the domain-specific problem of stratification by risk-levels using generated patient rank list.
Strengths
-> novelties added quite a lot: SNCE loss and handling right-censoring -> ranking loss introduced can be generally used for any method aiming to generate or classify along ranked lists -> well presented math for different loss functions and easy to follow
Weaknesses
-> lack of using unstructured data -> more clarity of censored data
Questions
I understood why there is censored data, but if the authors could explain with an example of a time series what censored data looks like that would make it easier.
Rating
8
Confidence
4
Soundness
4
Presentation
4
Contribution
4
Limitations
maybe added unstructured data like doctors notes
Thank you for the response. I have read the other reviews and all rebuttals. While two of the other reviewers were leaning towards reject, I think the authors responded well overall to their concerns. I don't see any clear reason to lower my score.
Thanks to all authors for their rebuttal. They have clearly gone to great lengths to address the comments that all reviewers have made. I have read the reviews from each of the reviewers, and the authors' response to them, as well as the response of the authors to my own review. I thank the authors for their clear explanation of the temperature coefficient, censoring, and the generation of positive and negative pairs. The manuscript would be improved by the inclusion of this text. I also thank the authors for performing further statistical analysis, specifically by providing the p-values associated with the Welch's t-test, to assess the performance of ConSurv relative to the other baselines. Unfortunately, I remain unconvinced by the performance of ConSurv. The proposed benefit of this model is to enhance discrimination, without sacrificing calibration. However, table E shows us that ConSurv outperforms the following models on **both** discrimination (CI) and calibration (IBS): - METABRIC: DRSA, DCS and X-CAL - NWTO: DeepSurv and X-CAL - GBSG: DeepHit - FLCHAIN: DRSA and DCS - SUPPORT: DRSA and DCS - SEER: CoxPH and DeepSurv In short, it seems that for 36 model-dataset pairs, only 12 show evidence for the proposed property. I also believe the choice to perform min-max normalisation before data splitting may have inadvertently led to data leakage, and I would expect the performance reported to be an overestimate were normalisation applied using the train/validation sets only. My score would change if I were convinced that the statistical performance was significant, and may be changed after further discussion.
Response to rebuttal
Thanks for the rebuttal. Agree with the future work.
Official Response by Reviewer Lfcy
Thank you for your response and for providing additional experimental results on DRAFT, large datasets, and sensitivity analysis. - Unfortunately, I don't think the comparison to DRAFT is adequate; a fairer comparison would be against DATE, since Chapfuwa et al. have already demonstrated that DATE is superior to DRAFT. - The connection of contrastive learning to model calibration appears to be superficial. The paper does not provide theoretical guarantees, only experimental results. Given that the experimental results highlight that $\beta$ controls the calibration-discrimination trade-off, this is a key weakness of the work. This weakness is not shared by the conformalized post-processing approach detailed by Qi et al. Hence, revisions should include comparisons to CSD and the Kaplan-Meier estimator. - The paper will require significant revisions to address the sloppy mathematical notations and provide complete details on the modeling choices. Considering these issues, as well as the other reviews, I am inclined to maintain my score.
Decision
Accept (poster)