Summary
### Motivation
- The authors propose to adapt 3D Gaussian Splatting (3DGS) to sparse-view tomographic reconstruction, i.e., to recover a radiodensity 3D volume from a small set of X-ray images and corresponding sensor information. This is relevant for various clinical and industrial applications.
### Contributions
- Their _R2-Gaussian_ model iterates over existing 3DGS solutions tuned for XR/CT imaging, proposing a 3DGS initialization scheme better suited for tomographic reconstruction.
- The authors also correct an integration bias in 3DGS (meant for faster image inference but causing ambiguities in volumetric reconstruction) and provide the corresponding CUDA patch.
- The proposed _R2-Gaussian_ includes other adaptations (custom densification parameters, voxel-based regularization, simplified isotropic kernels), resulting in an end-to-end CT reconstruction system.
### Results
- The authors provide extensive qualitative evaluation and quantitatively compare to other NeRF-based and traditional CT reconstruction methods, showing that their method provides a better trade-off between volume accuracy and reconstruction time.
### Relevance
- Effort in applying 3DGS to XR/CT imaging has grown the past year [6, 27, 39], as 3DGS appears to be a well-suited representation for such applications (due to its compactness, fast convergence, etc.). This work is a meaningful iteration and could benefit the community.
Strengths
_(somewhat ordered from most to least important)_
### S1. Convincing Comparative Evaluation and Qualitative Results
- The quantitative comparison to state-of-the-art CT reconstruction methods appear convincing, with the proposed solution demonstrating a better trade-off between volume accuracy and reconstruction time.
- The authors provide a lot of meaningful qualitative results, to illustrate theoretical contributions, to highlight the benefits of their method, but also to showcase its limitations. This makes reading this paper the more interesting.
- An ablation study w.r.t. some of the key contributions and w.r.t. some hyperparameters is also provided.
- Though limited in number (15 volumes), the authors evaluate on different categories (animal, vegatal, and synthetic targets).
### S2. Iterative yet Meaningful Contributions Towards 3DGS for XR/CT
- The discussion w.r.t. the integration bias in vanilla 3DGS is interesting, and the technical solution brought by the authors appear valuable to the community. As mentioned in the paper, their corrected CUDA implementation could benefit other 3DGS works targeting volumetric reconstruction.
- The authors propose an initialization scheme dedicated to volumetric tomography, as usual 3DGS initialization techniques (e.g., SfM) are not applicable here. This is a relevant contribution, properly described and evaluated (qualitative + quantitative evaluation).
- The proposed system, tackling xrays-to-CT reconstruction in an end-to-end differentiable manner, is indeed novel. Existing 3DGS models [6, 27, 39] for XR/CT imaging rather focus on digitally-reconstructed-radiograph (DRR) novel-view synthesis (NVS) rather than CT reconstruction.
### S3. Sound Theory and Reproducibility
- Background theory is well described by the authors, and the scientific/technical insight of the authors w.r.t. the identified integration bias could benefit the community.
- The authors provide their model implementation, which appears sound and well-structured (note that I did not try to run the code, but had a look at key files).
### S4. Detailed Discussion of Limitations and Contributions
- The authors put significant effort in discussing and illustrating some of their method's limitations (needle-like artifacts inherent to 3DGS, varying convergence time, limited extrapolation ability, etc.), as well as summarizing its impact (possible clinical/industrial applications, benefit of their CUDA code to the CV community, etc.) in Appendices G and H.
### S5. Well-Written and Illustrated Paper
- Overall, the paper is nicely structured, written, and illustrated. E.g., Figures 2-4 are helpful to understand the methodology at a glance.
Weaknesses
_(somewhat ordered from most to least important)_
### W1. Lack of Consideration for Real-World Noise and Anisotropic Effects
- The authors claim that "X-ray attenuation depends only on isotropic density" [L140] to justify their radiodensity-based model, but this is not entirely correct. While most models generating digitally reconstructed radiographs (DRRs) from CT volumes indeed consider x-ray attenuation as an isotropic phenomenon, this is a approximation. Some x-ray transport effects, such as Compton scattering, are actually anisotropic (but because CT volumes do not inherently contain the material information necessary to accurately simulate Compton scattering, DRR models ignore the anisotropy part) [a].
However, the authors do claim that they trained their model on DRRs generated using TIGRE [5] configured to simulate Compton scattering [L217-224]. Does it mean that the authors preprocessed the CT volumes to replace the attenuation values by material information (e.g., mapping HU values to a set of predefined materials)? More information on the data generation process would be helpful here.
If indeed the model was trained on DRRs containing anisotropic residual noise (c.f. Compton effect's residual impact on XR imaging), but the proposed algorithm itself only consider isotropic attenuation, how does it impact the results? E.g., it could be interesting to generate 2 sets of input DRRs (one generated with the approximated/simplified attenuation model and one more realistic) and compare the final accuracy of the reconstructed CT volumes.
- The fact that the method is only applied to synthetic inputs (DRRs generated by TIGRE) rather than real, usually noisier, X-ray images is also problematic. The paper would benefit from a real-world evaluation, or at least a discussion on why it was not performed.
### W2. Lack of References/Comparisons to SOTA on XR-3DGS
- The authors mention some of the existing 3DGS solutions applied to CT/XR imaging (e.g., X-Gaussian [6], GaSpCT [39], Li et al.'s model [27]) [L89-94] but do not perform any form of comparison with those. The lack of qualitative/quantitative comparison is fair (the authors argue that these models "cannot generate CT models" [L91-92], which is somewhat true ; though a comparison on the XR-NVS task could have made this paper stronger). But I would argue that the authors should have better referenced some of these works in the Methodology. E.g., even if less formalized, radiative Gaussians are already presented in X-Gaussian [6] ; and even though it is performed in the pixel domain rather than voxel one, GaSpCT [39] already proposes a total variation loss to regularize their XR-3DGS.
While I still believe that the proposed work is a valuable iteration over these works (by better formalizing and adapting the Gaussian properties and rasterization to CT data), I think the authors should be more transparent w.r.t. the SOTA.
- References w.r.t. total variation (TV) theory are also missing, making it hard to contextualize the scope of the authors' contribution w.r.t. the objective function.
### W3. Somewhat Overstated Contributions
- The CUDA implementation of the radiodensity voxelization seems to be more a technical feat rather than a scientific contribution. While possibly valuable to the community, I do not see any novelty in this module (maybe not as GPU-optimized, but differentiable point-cloud-to-voxel-grid tools already exist, e.g., in PyToch3D).
- Changes to the adaptive control are unclear/minor, according to the authors' descriptions [L210-214] (i.e., changing the size threshold w.r.t. pruning large Gaussians, editing the density of cloned/split Gaussians).
- The positive impact of the TV regularization does not appear that statistically significant (+0.32dB for PSNR, +0.009 for SSIM, +1m33s for convergence). Maybe some qualitative results could help grasp its contribution?
### W4. Limited Dataset Size
- The quantitative evaluation is performed on only 15 samples, even though varied. Larger CT datasets are available, e.g. CTPelvic1K [b].
### W5. Limited Impact of Integration Bias Correction (?)
- The authors claim that "this integration bias, though having a negligible impact on imaging rendering, leads to significant inconsistency in density retrieval" [L184-186], but they only provide qualitative imaging results (Fig. 6) to justify their un-biasing contribution. Additional results could better contextualize the corresponding claims.
----
### Additional Reference:
[a] Gao, Zhongpai, et al. "DDGS-CT: Direction-Disentangled Gaussian Splatting for Realistic Volume Rendering." arXiv preprint arXiv:2406.02518 (2024).
[b] Liu, Pengbo, et al. "Deep learning to segment pelvic bones: large-scale CT datasets and baseline models." International Journal of Computer Assisted Radiology and Surgery 16 (2021): 749-756.
Questions
_see **Weaknesses** for key questions/suggestions._
### Q1. Typo?
- [L223] Do the authors mean "Compton scatter" rather than "ponton scatter"?
Limitations
Some limitations and societal impacts are discussed in detail (see **S4** above).