Neural Frailty Machine: Beyond proportional hazard assumption in neural survival regressions

We present neural frailty machine (NFM), a powerful and flexible neural modeling framework for survival regressions. The NFM framework utilizes the classical idea of multiplicative frailty in survival analysis to capture unobserved heterogeneity among individuals, at the same time being able to leverage the strong approximation power of neural architectures for handling nonlinear covariate dependence. Two concrete models are derived under the framework that extends neural proportional hazard models and nonparametric hazard regression models. Both models allow efficient training under the likelihood objective. Theoretically, for both proposed models, we establish statistical guarantees of neural function approximation with respect to nonparametric components via characterizing their rate of convergence. Empirically, we provide synthetic experiments that verify our theoretical statements. We also conduct experimental evaluations over $6$ benchmark datasets of different scales, showing that the proposed NFM models outperform state-of-the-art survival models in terms of predictive performance. Our code is publicly availabel at https://github.com/Rorschach1989/nfm

Paper

Similar papers

Peer review

Reviewer nR175/10 · confidence 4/52023-06-30

Summary

The paper proposes to parametrize frailty models with neural networks and provide results on learning rates for these models. Experiments demonstrate the efficacy of the proposed models.

Strengths

1. The theoretical analysis is interesting and cool.

Weaknesses

1. The paper states that: "[56] used a neural network to approximate the conditional survival function and could be thus viewed as another trivial extension of NHR." How is this not exactly the same idea but for frailty models, you're proposing a neural parametrization very much akin to [56].

Questions

1. Everything is evaluated using IBS and IBLL, non-proper scores when censoring is not independent of event-times. Can you prove that the datasets you use indeed exhibit this independence. Can you provide evaluations using the likelihood, a proper score? 2. The objective seems computationally intensive to compute, can you provide computational comparisons with other methods?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Reviewer 82in7/10 · confidence 3/52023-07-04

Summary

The authors propose a neural architecture to estimate the survival function for observations of survival times and censored survival times. The authors describe a methodology to include heterogeneity among the population by including the frailty component. The authors then theoretically and empirically illustrate properties of their approach. The theoretical results are used to provide additional justification for why a neural network architecture is appropriate for this kind of problem. Empirical results are added to place into context the performance of the proposed methods with respect to existing methods.

Strengths

The paper provides a thorough analysis on using neural networks for survival estimation beyond the analyses present in previous works. The authors also consider frailty which they naturally include within their modelling framework, removing some of the homogeneity assumptions of previous works. The empirical results suggest that the method performs well in the considered scenarios. I did not check the details of the proofs, but the theoretical component seems useful and original for predicting convergence rates of the estimator and applicable to other similar estimators. Finally, the writing is clear and easy to follow.

Weaknesses

The main weakness involves the empirical evaluation of the paper. None of the empirical gains are major when considering the empirical experiments. This is expected with real world data since it's difficult to estimate the counterfactual (i.e. maybe the survival time could have been much longer/shorter). I would suggest to add an empirical result that compares the methods in idealized settings where the true survival distribution is known and that the metrics can be appropriately compared. Another potential weakness involves the theoretical guarantees. If I'm understanding correctly, the theory applies in the case where the networks are well-learned, which is still subject to the issues associated with neural network optimization. It may be good to emphasize this in the text.

Questions

How does the empirical convergence compare with the theoretical? The authors mention using Clenshaw-Curtis as the integrator, how does the parameters of the integrator (e.g. step size) affect the performance of the method? Is there some numerical bottleneck related to this integration? In the theoretical results, the MLPs scale with the number of samples, is there a rigorous connection to the number of parameters in the network and the number of samples? Additionally, was this enforced in the experimental setup, I could not tell if this was the case in the hyper parameter section of the appendix? Does the method extend to representing the joint distribution of multiple survival times? E.g. something like $P(T_1 > t_1, \ldots, T_d > t_d \mid Z)$? Was a nonparametric model for the frailty component examined, i.e. could $f_\theta$ be represented by some generative model constrained to be positive?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did mention limitations in the appendix. Alongside their notes, it may be good to get a review of where the authors think that their method is appropriate and where other methods would be more appropriate. Additionally, it would be nice to see conditions where the method fails to get a better idea of the properties of the method. This could possibly come from some of the theoretical assumptions being violated (e.g. it’s unclear which ones are really necessary in practice versus for proving the theorems).

Reviewer PAZm4/10 · confidence 3/52023-07-06

Summary

The frailty model is one of the popular models in survival analysis, and it is an extension of the classical Cox proportional hazard model. This paper extends the frailty model by using neural networks, and this paper provides the theoretical analysis of the proposed models.

Strengths

+ This paper proposes two new models NFM-PF and NFM-PM by combining the frailty model and neural networks. + This paper provides the theoretical analysis of the proposed models to show its correctness.

Weaknesses

I think that the (original) frailty model is interesting, simply because it uses slightly weaker assumptions than the classical Cox model. Even though their difference is small, the frailty model can be significantly better than the Cox model in practical prediction performance. This is because the Cox model is essentially a linear model and therefore a slight change (i.e., using frailty) can be a huge differentiator. However, the proposed extensions of frailty models by using neural networks are not so interesting. We already know many neural network models for survival analysis such as DeepSurv [42] and DeepEH [75], which are extensions of the Cox and AFT models. All of the neural network models for survival analysis are flexible enough, and the advantage of using the concept of frailty in neural network models should be minimal. The experiments are not good enough with respect to showing the practical advantage of using frailty in neural network models. Although the proposed models were compared with many existing models, the performance differences seem coming from the differences of the neural network architectures (e.g., the network structures, the number of layers, the number of neurons in each layer, and many other implementation details) rather than the frailty. Since the goal of the experiments is to show the advantage of using frailty in the neural network models, the authors should compare the two neural network models with and without frailty by using (almost) the same neural network architecture. Moreover, the authors should have used the right-censored log-likelihood, which is shown to be a strictly proper scoring rule in [56], as the evaluation metric instead of IBS and IBNLL in the experiments. The statement “Both IBS and INBLL are proper scoring rules” in Line 301 is not true, and it is shown that both IBS and INBLL are not proper in [56].

Questions

I have no question.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

I couldn’t find any description on limitations.

Reviewer U6h74/10 · confidence 3/52023-07-10

Summary

The authors propose a framework for survival regression called neural frailty machine. They have shown that most of the existing methods are a special case of the proposed framework. Also, they have drawn statistical convergence guarantees for the proposed model. The experiments show marginal improvement compared to existing models in terms of some metrics.

Strengths

The paper is well-written and has clear motivation. The overall approach is intuitive and makes sense (although I haven't checked the proofs). The main contributions are introducing frailty variable and the the statistical guarantees drawn for the proposed approaches.

Weaknesses

The main weakness is lack of discussion/explanation around the results and the two proposed approaches (PF and FN). More on this in the next section.

Questions

- What is the intuition behind introducing PF? It is mentioned (in third page, line 127) that "Proportional-style assumption over hazard functions has been shown to be a useful inductive bias in survival analysis". If this type of function approximation has been already shown deployed in survival analysis, then it wouldn't be very novel. - Checking results, sometimes FN performs better, while PF works better other times. Why? Are there any explanations for each dataset? - Is it possible to report numerical results for Figure 1, so we can compare PF and FN? - Addition of frailty variable (w) has marginal effect (<~1%) on most of the dataset (Table 5 in Appendix) while having huge effect on one the datasets. Any explanations for this discrepancy? Overall, I think more discussion around numerical results is needed.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Authors discussed some the limitations in the appendix.

Area Chair SB7u2023-08-15

please engage in discussion

Hello reviewers. Please read the authors' response to your reviews *and to the other reviewers' reviews*, and discuss. The authors have provided reasonably detailed rebuttals so it would be great if you engage in discussion as soon as possible, especially to indicate if your opinion of the paper has changed or if you would like any sort of additional comments or clarifications. Thanks, Area chair

Reviewer nR172023-08-16

Re: Rebuttal

Thanks for the response, raising to 5.

Reviewer 82in2023-08-18

I thank the authors for their response. After seeing the other reviews and the response, I would like to keep my score. However, it would be good to make a statement regarding how the proposed method is more than just a straightforward extension of existing methods (e.g. highlighting the major contributions that make the frailty model more challenging than a simple extension) as well as including additional numerical results demonstrating when the method should perform well. Of course, only so much is possible during the response period, so it would be good to augment the final version of the paper with some of these additional experiments.

Reviewer PAZm2023-08-19

Thank you for the authors' comments. I acknowledge that I have read all of the four reviews and the authors' comments. Having read the comments, I keep my score (i.e., I'm on borderline and I'm inclined to rejection), because I'm not convinced why the proposed methods *significantly* outperform the prior methods in the experiments even though the technical difference between the proposed methods and the prior methods is so small (i.e., using frailty). I would suggest clarifying all the hyper parameters (both for the proposed methods and the prior methods) and other details (e.g., python version) used in your experiments so that researchers can reproduce your experimental results. Note: I'm *not* requesting additional comments/experiments in this rebuttal phase, but I would suggest adding the information in the camera-ready version (if this paper is accepted) or the future manuscript (if this paper is rejected).

Authorsrebuttal2023-08-19

Some further clarifications

Thank you for the response. We will incorporate your suggestions in revised versions of the paper. Additionally, we provided experiment details in appendix C.3 of the paper regarding the tuning of hyper-parameters. Moreover, we would like to point out that **we did not make the conclusion that NFM is significantly better than previous SOTA, but mostly comparable and sometimes slightly surpassing** regarding the benchmark datasets which we used. So far as we have noticed, the empirical results on the chosen datasets rarely exhibit large performance gaps: Even compared to the vanilla CoxPH, the current SOTA methods (considering the best performing one) only shows statistically significant advantage (at level 0.05) on 4 out of 6 datasets. Finally, frailty is a flexible framework has many possible parameterizations which we elaborate in appendix A of the paper. Could you tell us a little bit more on why you insist that the difference between frailty formulations and previous approaches is "so small"?

Reviewer U6h72023-08-20

Thanks authors for the response. Regarding Q2 results, it doesn't seem that the proposed score shows the level frailty (maybe only presence of frailty?). Because, It is not aligned with the table 5 in appendix. MIMIC-III shows significantly higher improvement while having a comparable level of frailty estimate to metabric (or gbsg).

Authorsrebuttal2023-08-21

Some further clarifications

Thanks for the response. Perhaps there are some misunderstandings and misinterpretations regarding the results in table 1 in the rebuttal thread. We would like to make a few clarifications: - Speaking in a statistical context, the meaning in the frailty parameter is usually related to the variance of the random effect. In the additional experiments related to those results in table 1, we are primarily interested in assessing the presence of frailty (as you have also pointed out in the response), which is mostly reflected through the surrogate Z-score (third row of table 1). According to table 1, the presence of frailty is mostly significant in metabric and mimic-iii, which is coherent with table 5 in the appendix. Here we would like to re-emphasize that this is merely empirical observations. - As far as we can tell, the mean value of the frailty parameter has no explicit connection with how NFM performs regarding the predictive metrics like Cindex, IBS and INBLL. It might be somewhat misleading of using the term **strength of frailty effects**, which we want to correct here.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC