Physics-Regularized Multi-Modal Image Assimilation for Brain Tumor Localization

Physical models in the form of partial differential equations serve as important priors for many under-constrained problems. One such application is tumor treatment planning, which relies on accurately estimating the spatial distribution of tumor cells within a patient's anatomy. While medical imaging can detect the bulk of a tumor, it cannot capture the full extent of its spread, as low-concentration tumor cells often remain undetectable, particularly in glioblastoma, the most common primary brain tumor. Machine learning approaches struggle to estimate the complete tumor cell distribution due to a lack of appropriate training data. Consequently, most existing methods rely on physics-based simulations to generate anatomically and physiologically plausible estimations. However, these approaches face challenges with complex and unknown initial conditions and are constrained by overly rigid physical models. In this work, we introduce a novel method that integrates data-driven and physics-based cost functions, akin to Physics-Informed Neural Networks (PINNs). However, our approach parametrizes the solution directly on a dynamic discrete mesh, allowing for the effective modeling of complex biomechanical behaviors. Specifically, we propose a unique discretization scheme that quantifies how well the learned spatiotemporal distributions of tumor and brain tissues adhere to their respective growth and elasticity equations. This quantification acts as a regularization term, offering greater flexibility and improved integration of patient data compared to existing models. We demonstrate enhanced coverage of tumor recurrence areas using real-world data from a patient cohort, highlighting the potential of our method to improve model-driven treatment planning for glioblastoma in clinical practice.

Paper

Similar papers

Peer review

Reviewer S9Sg6/10 · confidence 4/52024-07-11

Summary

The authors proposed a discrete physic-based loss function which can be minimized to infer the spatial distribution of tumor cell distribution, which can not be assessed by imaging data only. The loss maps imaging data to a tumor tissue biomechanical model to solve inverse problems and to better understand individual tumor progression processes. One specific promise of the model is that it can learn the unknown initial condition of the brain tissue before the pathology. Their model is validated by achieving results comparable to state-of-the-art results on the task of tumor recurrence prediction from preoperative brain MRI.

Strengths

The proposed method incorporates brain tissue deformation seen on medical images to include a tissue displacement term in their biomechanical to more accurately simulate individual tumor growth. Thanks to this contribution, the model can predict an estimation of the brain tissue before the apparition of the tumor thanks to a loss term ensuring symmetry of brain tissue structures to simulate a healthy anatomy. The explanation about the different loss terms are clear and well written.

Weaknesses

In the reference [16], presented as the previous state-of-the-art method for recurrence coverage prediction, it seems that they obtained better results than the results displayed in table 1. More precisely, they obtained a recurrence coverage on all tumor areas of 76.09% which is better than the method presented in the submitted paper. It appears to me that the models were validated on the same dataset, but perhaps I missed something. Could the authors clarify this point? Following the previous point, it seems the paper lacks discussion about the use of the “Healthy anatomy” loss term. Compared to the results displayed in [16], it seems that there are no significant improvements using this new loss. How can we assert the correctness of the initial healthy brain tissue reconstruction? The authors claim that this initial state could be used to better identify post-surgical complications, but no clear method is described.

Questions

- Could the authors highlight the differences between [16] and their paper? Does addition of the tissue deformation in the physical modeling really improve the results? - Could the authors better explain the interest of the initial condition reconstruction? - Is the data-driven neural network trained on only 20% of the 58 data from radiotherapy planning dataset? Is it enough to train such a model? Could we obtain better results by increasing the training dataset? - How are inferred the parameters theta_up and theta_down? - In the appendix, in section B, it seems that c_i should be written instead of u_i (equation 14). - While it seems that visually the brain tissues exhibit symmetry, it should not be completely symmetric as we can observe on Figure 3 on the average brain template. It seems that “Healthy anatomy” term could lead to overly symmetric structures (see Figure 3 on the learned initial condition between the right and left hemisphere). It appears to me that this could be a limitation of the reconstruction method. Could the authors elaborate on this? The value of the paper could be increased with some analysis/discussion of this reconstruction.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors incorporated a “limitations” paragraph to address the convergence seed of their method and their tissue deformation model. However, the paper lacks some discussion in regard to their contribution compared to reference [16]. To which extent does the tissue deformation model improve the results? How can we assert the correctness of the initial condition reconstruction?

Authorsrebuttal2024-08-11

Dear Reviewer S9Sg, We thank you again for taking the time to read our manuscript and providing valuable feedback. We hope the rebuttal and additional experiments we provided were helpful. We hope the situation regarding [16] and the initial condition importance is now clear. Please let us know if there are still pressing concerns. Thank you, The Authors

Reviewer S9Sg2024-08-12

Thanks for the detailed answer. I updated my rating accordingly. I believe the paper is stronger with those changes.

Authorsrebuttal2024-08-13

We appreciate the reviewer's valuable feedback and are grateful for updating the score. Please feel free to reach out if any further questions arise.

Reviewer ZDBA6/10 · confidence 4/52024-07-12

Summary

The authors propose a method using soft physics-based regularization to predict the distribution of brain tumor. The main contribution is a novel discretization scheme of physics equations to model brain tumor.

Strengths

1. This work is an improvement for physics-based approaches for medical imaging 2. The proposed method of discretizing the physics residuals is novel and potentially applicable to related problems and of interest to the research community 3. The paper is clearly written and the technical details are sound and sufficient

Weaknesses

1. The main weakness is the lack of ground truth data to train and evaluate the model performance. The evaluation is performed not on the original task of brain tumor detection but rather on a downstream task of “recurrence coverage”, which measures the percentage of the tumor detected in follow-up MRIs (rather than the original). 2. Related to above, the authors only use one metric to evaluate their model, which may be insufficient, especially since there is no ground truth labels

Questions

1. Is there any bias in the evaluation metrics since there are no ground truth labels and authors use recurrence coverage?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

1. The authors should discuss the limitations of their methods in more details, especially related to the lack of ground truth labels to train the model

Reviewer ZDBA2024-08-08

I thank the authors for carefully addressing the reviewers' comments and improving the paper accordingly. I do not have further questions or suggestions on the paper. I updated my recommendation from "borderline accept" to "weak accept" as I believe the paper is stronger after the revisions.

Authorsrebuttal2024-08-11

We appreciate the reviewer's valuable feedback and are grateful for raising the score from 5 to 6. Please feel free to reach out if any further questions arise.

Reviewer YNED4/10 · confidence 4/52024-07-12

Summary

This paper presents a new approach to integrate physics-based tumor growth constraint with multi-modal imaging data to predict tumor cell distribution and thus enhance tumor treatment planning for glioblastomas. The core of the methods includes a discrete physics residual and initial assumptions encoding initial tumor distribution and symmetrical pattern between hemispheres. Experiments were performed on a small real dataset, and the success was measured by the “recurrence coverage” defined as the percentage of tumor (based on follow-up MRI) within the model-predicted volume. Experimental results demonstrate improved recurrence coverage of the presented method compared to fully data-driven or fully physics-based methods.

Strengths

Ability to integrate physics-based constraints and multi-modal imaging data is important and a difficult problem to address. The presented work is thus of important potential in bringing physics-informed learning into biomedical tasks. The dataset and task considered are quite complex involving multi-modal images and pre-operative and follow-up comparisons. I applaud the authors for the efforts devoted in this type of challenging tasks.

Weaknesses

The experimental evaluation of the presented work was performed on a very small dataset with unclear training-test split. This raises some question about the general conclusion that can be drawn from the results obtained on limited samples. Given the reported standard deviation (eg. Table 1), the obtained margin of improvement appears to be marginal without statistical significance. The writing of the paper lacks clarify at places. For instance, the results presented in Figure 5 was unclear. What are the “greater”, “less”, and “equal” categories, and why a bigger gap between the “greater” and “less” categories indicate a favorable performance? The objective of the presented model includes a large number of terms. The sensitivity of the model performance to the include/exclusion of these different terms and their hyperparameters deserves substantial analyses that are missing in the current paper.

Questions

Clarifications on the data split used to derive the reported results will be appreciated. The statistical significance of the reported performance (Table 1) will be appreciated. Clarifications on the results related to Fig 5 (see questions above) will be appreciated. How sensitive are the presented methods to the different terms in the loss functions, and which are the most important terms? How is asymmetry measured as an optimization objective?

Rating

4

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors presented some discussion about the limitations of the current work. Adding discussion about limitations related to the limited sample size for the experiments would be appreciated.

Authorsrebuttal2024-08-11

Dear Reviewer YNED, We thank you again for taking the time to read our manuscript and providing valuable feedback. We hope our rebuttal and the additional experiments were helpful in clarifying the data splits, demonstrating the statistical significance of our findings, and highlighting the importance of each loss term. Please let us know if there are still pressing concerns. Thank you, The Authors

Reviewer YNED2024-08-12

I'd like to thank the authors for the effort put in the rebuttals, which has helped addressed some of my previous concerns especially in adding significance test to the experimental results. I do acknowledge the value of the study especially the use of physiological terms to enable optimization given small data. I do however believe that the sensitivity of the method performance to the various hyperparameters used in the loss function deserves a much more comprehensive study -- the rebuttal did a good job demonstrating how the absence/presence of these individual terms would affect the model performance, as well as their rough range of values (large or small). What need to be demonstrate however are the best practice for tuning these individual terms, and how their values affect the margins of improvement the method is currently demonstrating, i.e, how sensitive is the demonstrate margin of improvement to the tuning of the many hyperparameters in the loss, and what would be the strategy to obtain such optimal hyperparameters in real world settings when given a new dataset. I did raise my rating although I think more work is needed to clarify this issue considering the many hyperparameters and their effect on the margin of improvements demonstrated.

Authorsrebuttal2024-08-13

HP search, a general scheme

Thank you for your valuable feedback and for updating the score. The balance between the importance of PDEs and data is indeed a broader challenge in the physics-informed ML field. The reviewer rightly highlights a crucial issue in this area. For such an ML problem, it is essential to use a diverse set of complementary assumptions and regularizations because, without them, the problem becomes ill-posed. All the terms we included are well-grounded with clear physical motivation. Regarding sensitivity, the solution should not be overly sensitive to individual changes. We refer to Fig. 2a (and to some extent, Fig. 2e) in [16] (Static Grid Discretization method), where the authors observed a 20% difference in DICE scores by varying $\lambda_{PDE}$ across five orders of magnitude (which corresponds to $\alpha_1$, in our case), which can be seen as low sensitivity to small changes. When it comes to actual values for hyperparameter selection, in our work, we opted for a general approach using calibration on synthetic data. We began by determining the order of magnitude for the relevant hyperparameters (e.g., ensuring no tissue folding and preventing the initial tumor cells from vanishing) through a grid search, followed by further fine-tuning using the same grid search method and the RMSE as the metric (as in Table 2 of the one-page PDF). While we do not claim that the parameters we identified are necessarily optimal, they are effective enough to outperform all other baselines without raising concerns about an overly complex hyperparameter optimization scheme that might not generalize to other datasets. For datasets involving pre-operative tumor images, our method should work out-of-the-box, as it relies solely on adherence to the pre-determined growth equations and tissue elasticity, without needing specific information from our dataset. We will extend Appendix D to provide more detailed information on the hyperparameter selection procedure involving grid search and the individual physical motivations behind each parameter to assist researchers in applying our framework to new domains/equations. We are always happy to consult further on this. As a final point, we would like to emphasize that almost all physics-informed ML frameworks rely on implicitly optimizing the unknown fields using CNNs or MLPs, which introduces numerous additional hyperparameters for these architectures. In contrast, our approach only requires setting the resolution aligned with the data's resolution and the stability conditions of the corresponding growth PDEs, thereby eliminating the need to include these parameters in the hyperparameter search.

Reviewer smuQ5/10 · confidence 1/52024-07-13

Summary

This paper presents a method for brain tumor modeling through a joint data-driven and physics-based approach. The proposed approach leverages a multi-task loss that incorporates physical assumptions and prior healthy anatomy to more faithfully model the tumor cell distribution. This method outperforms existing computational approaches and the clinical standard with respect to recurrence coverage

Strengths

- The presentation quality is extremely high. Care was taken to logically organize sections of the paper with helpful illustrations and definitions throughout. - The treatment of related work is thorough, and domain-specific details are effectively communicated throughout. - Each component of the proposed method is well-motivated and specifically catered to the application domain. - Experiments appear to be sound, and the proposed method outperforms relevant baselines.

Weaknesses

- I am slightly confused by the recurrence coverage metric. Has this metric been used in prior work? Why not a well-established overlap metric such as intersection over union, for example?

Questions

- According to line 207, recurrence coverage is the “percentage of the tumor segmentation… that is encompassed within the radiotherapy target volume defined by each model.” If this is true, then wouldn’t a predicted radiotherapy target volume consisting of the entire scan obtain 100% recurrence coverage? Unless I am misunderstanding, an existing overlap metric (which penalizes such “overprediction”) might be more appropriate. - Line 5: “Deep-learning based” -> “Deep learning-based” - Line 234: Missing parenthetical “(Physics-Constrained:”

Rating

5

Confidence

1

Soundness

3

Presentation

4

Contribution

3

Limitations

Limitations are briefly addressed in Section 7.

Authorsrebuttal2024-08-11

We want to thank the reviewer again for the valuable suggestions. If we have now addressed all your concerns and questions, we would appreciate if you could consider updating your scores to reflect your post-rebuttal opinion. Please kindly let us know if you have any follow-up questions or areas needing further clarification.

Reviewer smuQ2024-08-07

I acknowledge that I have read the authors' rebuttal and appreciate their hard work. In particular, thank you for clarifying the recurrence coverage metric and including other overlap metrics.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC