DDGS-CT: Direction-Disentangled Gaussian Splatting for Realistic Volume Rendering

Summary

1. The proposed direction-disentangled 3DGS (DDGS) method decomposes the radiosity contribution into isotropic and direction-dependent components, able to approximate complex anisotropic interactions without complex runtime simulations. specifically, it modeling isotropic and anisotropic contributions via distinct 3D Gaussians. 2. The paper clearly mentioned that 3DGS-for-DRR solutions [30, 11] do not account for noise-inducing photon interactions (e.g., scattering) when applying the analytical methods [29, 5] to render their training data.

Strengths

1. This paper has a clear definition of the problem to be solved and a good visualization. 2. Experiments combining 2D/3D CT Image registration were also presented.

Weaknesses

1. Although Gaussian Splatting is widely used in natural color scenes, this paper does not provide a detailed mathematical description of the "migrated version" of Gaussian Splatting in combination with the physical scene of CT. Instead, it directly use the mathematical description of Gaussian Splatting in natural color scenes (Equation (1)). In CT imaging, the contribution of a voxel to the final projection does not suffer from attenuation due to occlusion by the following voxels, but is instead Beer-Lambert law. Therefore, equation (1) no longer holds. This makes it difficult to highlight the motivation of this paper in terms of algorithm details. 2. Theoretical contributions appear to be minor. (Formula (3) is the theoretical contributions of this paper). This makes this paper more suitable for delivery to places that are more concerned with simulation applications. 3. The paper needs to compare with Physics-based Monte-Carlo simulations [2, 15, 3 , 4 ]. Although these methods are time-consuming, they guarantee the quality of reprojection. The paper needs to clearly point out the difference in image quality with Physics-based Monte-Carlo simulations. The ground truth(GT) needs to select the real projection obtained by the CT machine. So the comparison table will be： ______________________________________________________________________________________ | Physics-based Monte-Carlo simulations | 3DGS | X-Gaussian | DDGS _____________________________________________________________________________________ PSNR with GT | ______________________________________________________________________________________ SSIM with GT | ______________________________________________________________________________________

Questions

1. what is the means of high-dimensional residual contribution？Is there a formal description? This concept is not explained in the full text. 2. Before use the method, it need first get the 3D CT volume. The reconstruction is based on the projection obtained by the CT machine. However, there are errors and noise in the 3D CT volume (from the reconstruction of the projection noise) and also from the reconstruction errors caused by the polychromatic X-rays in medical CT machine. The paper is equivalent to reprojecting the 3D CT volume using gaussian splatting. How do the work ensure the reprojection process does not amplify this error from the 3D CT volume?

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

1. If this method is used in clinical practice, what problems will it cause without considering the time cost? For example, the appearance of artifacts, because once artifacts appear, it will seriously interfere with the doctor's clinical operation.

Summary

Digitally reconstructed radiographs (DRRs) are simulated 2D X-ray images generated from 3D CT volumes, widely used in preoperative settings but limited in intraoperative applications due to computational bottlenecks. In this paper, the author proposes an approach that combines realistic physics-inspired X-ray simulation with efficient, differentiable DRR generation using 3D Gaussian splatting (3DGS). This method excels in both image accuracy and inference speed, demonstrating its potential for intraoperative applications and inverse problems like pose registration.

Strengths

Digitally reconstructed radiographs (DRRs) are simulated 2D X-ray images generated from 3D CT volumes, widely used in preoperative settings but limited in intraoperative applications due to computational bottlenecks. In this paper, the author proposes an approach that combines realistic physics-inspired X-ray simulation with efficient, differentiable DRR generation using 3D Gaussian splatting (3DGS). This method excels in both image accuracy and inference speed, demonstrating its potential for intraoperative applications and inverse problems like pose registration.

Weaknesses

1. The methods used for comparison are limited and do not reflect the current state of the field. The state-of-the-art methods claimed by the author are not widely accepted as state-of-the-art by researchers, and some have not been peer-reviewed, like reference [7,11]. Therefore, I believe the experimental results lack persuasiveness. 2. The structure of the paper is not well organized. Despite the author clearly stating the motivation and contributions, the intentions of the author remain unclear. I have a rough idea of what the related work section is trying to convey, but its organization is also poor. It is recommended to include the 'Application' from section 3.1 within the related work. 3. In the ablation experiments, only a portion of the data was tested. Results from testing the entire dataset would be more convincing.

Questions

1. The methods section lacks explanations for some symbols. Please review and provide explanations where necessary. Additionally, are g_i^{iso} and c_i^{iso}, as well as g_i^{dir} and c_i^{dir}, referring to the same concepts? Please review carefully. For the methods section, it is recommended to include an overview diagram for clarification. 2. Add more true sota methods for comparison.

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

1. The experiments are insufficient. The comparison method lacks representativeness and fails to demonstrate the true level of the field. 2. The method description is confusing and the contributions are not clear.

Summary

This paper proposes DDGS, a Gaussian splatting (GS) based method for rendering realistic 2D X-ray images from 3D CT volumes. Taking advantage of GS, the proposed method operates in real-time. Moreover, the DDGS model employs isotropic Gaussians and anisotropic, direction-dependent Gaussians to model complex X-ray physical interactions, such as Compton scattering. The proposed method is well-motivated and technically sound. The effectiveness of the DDGS model is confirmed on multiple datasets.

Strengths

**Motivation** Conventional Monte-Carlo methods can produce realistic X-ray images, but they are time-consuming. Existing real-time GS-based methods ignore complex X-ray interactions, limiting reconstruction performance. The proposed DDGS model integrates complex X-ray physical interactions into GS to achieve real-time, high-quality X-ray image rendering. This is well-motivated. **Technical novelty** The proposed DDGS model introduces novel anisotropic, direction-dependent Gaussians to account for the Compton scattering effect. This is a good idea and technically sound. Moreover, the authors propose a novel Radiodensity-Aware Dual Sampling method for CT volumes. **Clarity and organization** This paper is well-written and easy to follow. **Experimental evaluation** The authors performed extensive experiments on three different datasets, including comparisons with state-of-the-art (SOTA) methods, ablation studies, and downstream tasks. The results confirm the effectiveness of the DDGS model.

Weaknesses

**Effectiveness of the Radiodensity-Aware Dual Sampling** From Table 4, the improvements made by the Radiodensity-Aware Dual Sampling are marginal (<0.3 dB), so its effectiveness needs further investigation. **Lack of CT volumes** Although the purpose of this work is digitally reconstructed radiographs, showing the CT volume represented by GS is necessary for evaluating the proposed model. **Visualization results** The qualitative results shown in Figure 3 are hard to distinguish visually. Please consider showing the error maps between the reconstructions and the ground truth. **Some typos** For example, in line 90, “] − 800, −200], “fat” for ] − 200, −100]” should be “[ − 800, −200], “fat” for [ − 200, −100]”.

Questions

See the section of Strengths and Weaknesses, please.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors mention enough limitations of their work.

Summary

This manuscript presents a novel method called Direction-Disentangled Gaussian Splatting (DDGS-CT), tailored for balancing realistic X-ray simulation and efficient DRR generation using 3D Gaussian Splatting (3DGS). It addresses the challenges posed by intricate physics computation, which often harms the application of Monte Carlo simulations. The method decomposes the radiosity into isotropic and direction-dependent components to approximate complex anisotropic interactions without runtime simulation. Notably, it achieves significant improvements in PSNR and SSIM on DRR renderings of multiple organs, demonstrating superior performance.

Strengths

1. Innovative Approach: This manuscript introduces an innovative method of X-ray simulation techniques to balance the efficiency and realism in DRR rendering, which is a significant step forward in the field of medical imaging. 2. Significant Empirical Improvements: The method substantially improves PSNR and SSIM scores, demonstrating its effectiveness over existing methods. 3. Detailed Methodological Framework: This manuscript presents a well-structured and comprehensive methodological framework, introducing the disentanglement of isotropic and direction-dependent components and initialization strategy that accounts for the nature of CT data.

Weaknesses

1. “… we equally split P^dw to initialize both isotropic and direction-dependent Gaussians.” I’m confused about how to split the points set P^dw into disentangled isotropic and anisotropic point sets. The authors should provide a more detailed explanation. 2. How do the disentangled isotropic and anisotropic 3DGS be rendered? Based on Figure 1, it seems that the two disentangled 3DGSs are integrated to render the output. If so, why should they be disentangled? The authors should explain the motivation for this design. 3. In the Experiments, the 3D Gaussian-based methods are evaluated, yet the DRR-based methods mentioned above are not validated both in qualitative and quantitative comparison. Can they be added?

Questions

1. I wonder if it is necessary to disentangle the isotropic and anisotropic point sets since they are both splatted to render the output images. I think if they are disentangled, it would be more reasonable that they are also rendered using different algorithms. 2. As far as I know, 3DGS is actually an explicit representation of 3D data. I wonder if it is necessary to transform 3D data from an explicit representation (volume) into another explicit representation (3DGS).

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors have adequately discussed their limitations.