3D Gaussian Splatting has shown fast and high-quality rendering results in static scenes by leveraging dense 3D prior and explicit representations. Unfortunately, the benefits of the prior and representation do not involve novel view synthesis for dynamic motions. Ironically, this is because the main barrier is the reliance on them, which requires increasing training and rendering times to account for dynamic motions. In this paper, we design a Explicit 4D Gaussian Splatting(Ex4DGS). Our key idea is to firstly separate static and dynamic Gaussians during training, and to explicitly sample positions and rotations of the dynamic Gaussians at sparse timestamps. The sampled positions and rotations are then interpolated to represent both spatially and temporally continuous motions of objects in dynamic scenes as well as reducing computational cost. Additionally, we introduce a progressive training scheme and a point-backtracking technique that improves Ex4DGS's convergence. We initially train Ex4DGS using short timestamps and progressively extend timestamps, which makes it work well with a few point clouds. The point-backtracking is used to quantify the cumulative error of each Gaussian over time, enabling the detection and removal of erroneous Gaussians in dynamic scenes. Comprehensive experiments on various scenes demonstrate the state-of-the-art rendering quality from our method, achieving fast rendering of 62 fps on a single 2080Ti GPU.
Paper
References (91)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The author proposed a Fully Explicit Dynamic Gaussian Splatting method to decrease the training time and memory cost w/o losing the high fidelity. The core of the proposed approach based on the decomposition of the static and dynamic Gaussians during training is to sample dynamic Gaussians at sparse timestamps and then do interpolation to estimate the dynamic Gaussians within each time interval. Also, the proposed backtracking technique is used to prune the useless dynamic Gaussian points. However, there are still some unclear questions. Will consider increasing the rating if the authors can provide reasonable response.
Strengths
1. unlike the existing works, the authors attempt to select keyframes and utilize different interpolation techniques on keyframes to model the dynamic Gaussians. 2. The results on the N3V dataset show that the proposed method achieves comparable or higher performance than the existing methods, while largely reducing the memory and training time cost. 3. The novel techniques to extract the dynamic points and model the temporal opacity seem very interesting.
Weaknesses
1. From the results shown in Fig.6 and the supplementary materials, the extracted points from the 'dynamic points extraction' technique seem actually to be the points with color change. I think the results are not very concrete to support the contribution. More experiments in the scenes including objects with large movements might be helpful. 2. From the proposed opacity modeling technique and the limitations claimed in Sec.6, it seems the proposed method can only work well in such scenes that the objects within the scenes are stable w/o new objects coming in. I would like to know the performance of the proposed method when some new objects come in while progressively processing. It seems that the results do not contain such scenarios.
Questions
1. In lines 238 and 239, why the top 2% points are converted? why consider the distant points as the potential bias? 2. Can the authors provide more details on how the parameters of keyframes are optimized? 3. In line 258, what does it mean by 'use only COLMAP point clouds from the first frame'? 4. What if the movements between key frames are not regular motions?
Rating
5
Confidence
4
Soundness
2
Presentation
3
Contribution
2
Limitations
Please refer to 'Weaknesses'.
Reference tables
### Table A. Quantitative results of the experiment on handling color changes without dynamic points. | Model | PSNR | SSIM | LPIPS | |-|-|-|-| | 3DGS | 21.69 | 0.851 | 0.126 | | 3DGS + ours dynamic | 26.07 | 0.891 | 0.089 | | Ours | 29.03 | 0.922 | 0.068 | $~$ $~$ ### Table B. Quantitative results on the Birthday scene in the Technicolor dataset. | Technicolor Birthday longer | PSNR | SSIM | LPIPS | |-|-|-|-| | Ours | 29.12 | 0.900 | 0.094 | $~$ $~$ ### Table C. Experimental results on the conversion rates to dynamic points. | Percent | PSNR | SSIM | LPIPS | |-|-|-|-| | 0.5 | 32.55 | 0.956 | 0.043 | | 1 | 32.86 | 0.957 | 0.042 | | 2 | 33.04 | 0.956 | 0.041 | | 4 | 32.90 | 0.956 | 0.042 | | 8 | 31.72 | 0.955 | 0.042 | $~$ $~$ ### Table D. Experimental results on different keyframe intervals and skipped frames. | Skipped frames | 1 | | | | 2 | | | | 3 | | | | |-|-|-|-|-|-|-|-|-|-|-|-|-| | Keyframe interval | PSNR | SSIM | LPIPS | Size (MB) | PSNR | SSIM | LPIPS | Size (MB) | PSNR | SSIM | LPIPS | Size (MB) | | 1 | 31.17 | 0.948 | 0.057 | 595 | 31.47 | 0.948 | 0.056 | 415 | 31.81 | 0.946 | 0.051 | 142 | | 2 | 32.06 | 0.952 | 0.051 | 314 | 32.33 | 0.954 | 0.049 | 322 | 31.81 | 0.953 | 0.044 | 101 | | 5 | 31.70 | 0.953 | 0.047 | 206 | 32.29 | 0.954 | 0.043 | 126 | 32.53 | 0.954 | 0.045 | 80 | | 10 | 33.04 | 0.956 | 0.041 | 119 | 32.65 | 0.956 | 0.043 | 93 | 31.79 | 0.953 | 0.046 | 74 | | 20 | 32.78 | 0.955 | 0.043 | 90 | 32.07 | 0.952 | 0.047 | 78 | 32.08 | 0.953 | 0.048 | 73 | | 50 | 32.14 | 0.955 | 0.046 | 79 | 31.93 | 0.951 | 0.052 | 72 | 30.91 | 0.949 | 0.056 | 73 |
Summary
The paper proposes a new method in the field of novel view synthesis for video input. The authors propose a Gaussian Splatting-based algorithm that introduces a fully explicit representation at keyframes and models interpolation of gaussians (position, rotation, opacity) between the frames. Additionally, the paper proposes to learn scene separation into static and dynamic parts in the progression of training. The method is evaluated on popular video novel view synthesis literature datasets.
Strengths
- The idea of modelling the scene explicitly in keyframes is certainly an interesting choice - it has the potential to offer a solution to modelling long-range dependencies. In combination with the use of Gaussian Splatting, it is definitely a novel concept. - The choice of interpolators seems to be good and well supported through experiments, specifically two-gaussian opacity modelling is a neat way of dealing with objects appearing and disappearing. - Separation into static and dynamic Gaussians is a good idea and based on the results and visualisations seems to work well in the proposed methodology. The premise of the majority of the scene being modelled as static was explored in NeRF-based solutions. However, this work provides a novel solution of how to implicitly divide the scene into dynamic and static gaussians. - The description of the method is easy to follow. - The authors provide a large number of ablation experiments with many components of the methods focused on. - The proposed method reports very convincing training times, especially for training on videos. - The supplementary video presents the qualitative results in a very clear manner.
Weaknesses
- I would like to see more detailed descriptions of the approach throughout the paper. Mainly: - How does the temporal opacity modelling relate to the densification of Gaussians? - More details on keyframe selection would be useful - why fixed step, how is the step chosen. - What is the motivation of static gaussians having linear position interpolation? - It would be good to see the same detailed breakdown per scene for the Technicolor dataset. Similarly, the Technicolor experiment shows fewer comparisons to other methods (I would try to include more, at least Gaussian Splatting based ones). - The keyframe interpolation naturally draws attention to the method potentially being better at handling big movements or longer videos. Therefore, I think such a comparison is missing in this work. It could be done on the same datasets. Other works compare only short portions of the Technicolor dataset which hides the potential issues with longer videos and bigger movements, however, full scenes are available. Similarly, Neural 3D Video offers a very long sequence for the *Flame Salmon* scene. - Even though the ablation is extensive, there could be more insights on the results provided. - Firstly, we are missing the information on what is the ablation performed on - which dataset, one or more scenes. - Is w/o Dynamic point extraction treating everything as static, just with position linear interpolation? - It seems that all the components are highly relevant (removal of one degrades the quality by almost $0.5dB$ at least). Is that a consistent result between multiple runs? Also, is this scene-specific or an average? - It would be great if the authors provided some statistical analysis of the results.
Questions
- HumanRF [1] pays particular attention to how to manage bigger and smaller movements in the video via non-uniform segmentation into manageable chunks - did you notice the high sensitivity of your method to keyframe interval? - Relating to mentioned more detailed descriptions: - It seems that only the initial set of gaussians is being modified throughout the training. Therefore, how does the appearance of a new object happen? For a big object, there have to be a lot of new gaussians. Also, all appearances in the video have to be modelled with two gaussians. Are there any issues when an object appears and disappears repetitively (e.g. the rotating decoration in the Technicolor Birthday scene)? - Point backtracking for pruning is not fully clear to me. To make sure, does $\mathcal{D}$ include all training views in all timesteps? Further, what is the pruning rule (is it carried over from the original Gaussian Splatting)? - Regarding linear position change for static gaussians - why is this necessary given the gaussians should be static? Is it to correct tiny movements? Is there no need to allow linear rotation interpolation as well? - I would encourage authors to check [2] - it seems that this method also models the gaussian representation explicitly in keyframes, however, takes a different approach for motion modelling. Note that this has only just been published at ECCV and does not affect my opinion on this paper, it may just be a worthwhile mention in related work of how the proposed method stands out with respect to others. [1] Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, Matthias Nießner, *HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion*, ACM Transactions on Graphics (TOG), 2023 [2] Richard Shaw, Jifei Song, Arthur Moreau, Michal Nazarczuk, Sibi Catley-Chandar, Helisa Dhamo, Eduardo Perez-Pellitero, *SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting*, European Conference on Computer Vision, 2024
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors provided a brief limitations paragraph. I believe the issue with new object appearance is a good mention.
Answers to [Q1, 2]
[Q1] The sensitivity of motions depends on keyframe intervals. However, instead of directly measuring motions, we decide to determine the keyframe intervals based on rendering quality. As explained in W1-2, the keyframe interval is based on both storage usage and rendering quality. Here, we assume that sufficiently small keyframe interval (in our cases, 10) can handle most of complex motion. We report the experimental evidence in Table J. To simulate and measure the motion speed, we intentionally skip frames in videos. [Q2] Each answer is as follows: 1. As described in W1-1, we adopt the densification algorithm of the original 3DGS model, and this densification is applied to both static and dynamic points. In addition, static points are periodically converted to dynamic points so that even if there is a new object, it can be treated as a dynamic point by splitting the Gaussian around it. In our experiments, the initial Gaussians start at about 7000, but by the end of training, the scene usually includes more than 200,000 Gaussians. Although densification can handle new objects as long as there are Gaussians around the dynamic object that can be split (e.g., static Gaussians can be converted to dynamic after splitting, or dynamic Gaussians can be split and optimized), unfortunately, when there is a sudden appearance of an object as described in Section 6, it is difficult to learn because there are no Gaussians around it to split. This is the same problem as the original 3DGS, which does not learn well unless there is a Gaussian to split around it (e.g., random initialization). Also, for objects that appear repeatedly, there is no way to determine if the disappearing object and the reappearing object are the same because its temporal consistency is required to determine this. Our model represents these as separate objects when an object is temporally discontinuous. In our experiment, the rotating decoration that is occluded and reappears, our model learns that the dynamic points disappear and new dynamic points appear in Figure C of the uploaded PDF file. In this figure, you can see that a number of point clouds changes before and after the object flips. This shows that FEDGS creates different Gaussians when the object reappears. Nevertheless, you can see that the rendering results look good. We further carry out an experiment with the presence of occlusion. We have added comparison evaluation in Table K. Figure A of the uploaded PDF file shows the comparison results when occlusion is present. We select 100 frames of the train scene in the Technicolor dataset and compare FEDGS with the other models. We use the first frame as input and do not use the point cloud information of other frames. The results show that FEDGS renders dynamic objects well, even when they disappear and reappear. 4DGS learns dynamic parts well, but struggles to render static parts, while STG struggles to render dynamic parts. 4D Gaussians fails to render dynamic objects. 2. $\mathcal{D}$ represents the images of all timestamps in all training sets.The original 3DGS does not have it, and we add this because the opacity-based pruning method of the original 3DGS does not completely remove floaters caused by dynamic objects. This is because some dynamic objects are trained to move to an invisible area, instead of vanishing. This method measures how much each Gaussian contributes to the error in the rendered image, much like accumulating a gradient in densification. More specifically, using this operation on a rendered image, we can get the sum of the error (L1 or SSIM error) and the sum of the alpha values of all the Gaussians in the image. Dividing this cumulative error by the sum of the alpha values can normalize it. We threshold this error to remove Gaussians that cause large errors. 3. The reason, why we assume that static points have linear motion, is to distinguish dynamic points, as explained in W1-3. This does not aim to model the motion of static parts exactly. We agree with your idea of modeling the rotation of the static points, but in our method, the magnitude of the position change is enough to distinguish dynamic points. Different from the original 3DGS model learning positional information of static points, all static points in FEDGS are approximated to have linear movement. And we measure a magnitude of the motion between frames to reassign static point to dynamic point if the magnitude is more than a predefined threshold value. Interestingly, although static points are represented as linear motions over time, they can explain all temporal changes, including color, rotation, and opacity. This is because the static points are considered as moving points during optimization if one of the temporal changes exists. Therefore, instead of using a more complex model to represent static points, we use a simple yet effective way that can further reduce the computational complexity. Answer to [Q3] is followed by the next comment.
Answer to [Q3]
[Q3] Thanks for introducing a relevant paper. We have looked at this paper and will state the differences between it and this paper. SWinGS disentangles static and dynamic scenes using MLP. For dynamic regions, the model slices the timestamp using a sliding window strategy and trains a deformation MLP of the canonical space by window. SWinGS has similarities with FEDGS in that it divides static and dynamic parts and slices timestamps, but the differences between them are as follows: * __Detecting Dynamic part__: SWinGS uses MLP to classify static and dynamic in a binary manner. It performs classification by thresholding L1 error and uses spatial location as input. FEDGS, on the other hand, determines dynamic points based on the magnitude by which the static gaussian moves. This can distinguish dynamic points even if two points are in the same location. * __Dynamic representation__: SWinGS uses an implicit function defined for each window to find the deformation in the canonical space of the dynamic grid. On the other hand, we store the dynamic motion of each point independently in an explicit way. Therefore, we do not need the coordinates of the canonical space. We only need the position information of the neighboring keyframes. * __Keyframe selection__: SWinGS uses the magnitude of the optical flow to measure the magnitude of the motion and divides the keyframes based on this. FEDGS, on the other hand, uses a fixed keyframe to take advantage of indexing, as described earlier. This interval is a hyperparameter. * __Optimization__: SWinGS adds a loss of pixel difference to achieve temporal consistency. In FEDGS, point backtracking for dynamic point pruning, conversion of static points to dynamic points, and progressive learning scheme are added.
Reference tables
### Table A. Experimental results on different keyframe intervals and skipped frames. | Keyframe interval | PSNR | SSIM | LPIPS | Size (MB) | |-|-|-|-|-| | 1 | 31.17 | 0.948 | 0.057 | 595 | | 2 | 32.06 | 0.952 | 0.051 | 314 | | 5 | 31.70 | 0.953 | 0.047 | 206 | | 10 | 33.04 | 0.956 | 0.041 | 119 | | 20 | 32.78 | 0.955 | 0.043 | 90 | | 50 | 32.14 | 0.955 | 0.046 | 79 | $~$ $~$ ### Table B. Quantitative evaluation on the Technicolor dataset using PSNR. | PSNR | Birthday | Fabien | Painter | Theater | Train | Average | |-|-|-|-|-|-|-| | DyNeRF | 29.20 | 32.76 | 35.95 | 29.53 | 31.58 | 31.80 | | HyperReel | 29.99 | 34.70 | 35.91 | 33.32 | 29.74 | 32.73 | | STG(Sparse) | 31.97 | 34.54 | 36.50 | 30.55 | 32.66 | 33.25 | | 4DGS | 28.04 | 26.22 | 33.80 | 31.50 | 27.93 | 29.50 | | 4D Gaussians | 30.87 | 33.56 | 34.36 | 29.81 | 25.35 | 30.79 | | Ours | 32.55 | 34.54 | 36.56 | 31.11 | 31.72 | 33.30 | $~$ $~$ ### Table C. Quantitative evaluation on the Technicolor dataset using SSIM. | SSIM | Birthday | Fabien | Painter | Theater | Train | Average | |-|-|-|-|-|-|-| | DyNeRF† | 0.952 | 0.965 | 0.972 | 0.939 | 0.962 | 0.958 | | HyperReel | 0.922 | 0.895 | 0.923 | 0.895 | 0.895 | 0.906 | | STG(Sparse) | 0.942 | 0.886 | 0.925 | 0.877 | 0.942 | 0.915 | | 4DGS | 0.905 | 0.867 | 0.900 | 0.874 | 0.839 | 0.877 | | 4D Gaussians | 0.906 | 0.867 | 0.886 | 0.848 | 0.729 | 0.847 | | Ours | 0.943 | 0.884 | 0.929 | 0.875 | 0.921 | 0.910 | †: Use structural similarity function from scikit-image library $~$ $~$ ### Table D. Quantitative evaluation on the Technicolor dataset using LPIPS. | LPIPS | Birthday | Fabien | Painter | Theater | Train | Average | |-|-|-|-|-|-|-| | DyNeRF | 0.067 | 0.242 | 0.146 | 0.188 | 0.067 | 0.142 | | HyperReel | 0.053 | 0.186 | 0.117 | 0.115 | 0.072 | 0.109 | | STG(Sparse) | 0.039 | 0.135 | 0.098 | 0.122 | 0.033 | 0.086 | | 4DGS | 0.089 | 0.199 | 0.138 | 0.157 | 0.166 | 0.150 | | 4D Gaussians | 0.088 | 0.188 | 0.162 | 0.189 | 0.272 | 0.180 | | Ours | 0.043 | 0.152 | 0.089 | 0.143 | 0.071 | 0.100 | $~$ $~$ ### Table E. Quantitative results on the extremely long duration video sequences (Flame Salmon) in the N3V dataset. | | PSNR | SSIM | LPIPS | Size (MB) | |-|-|-|-|-| | Ours | 28.77 | 0.919 | 0.076 | 392 |
Reference tables
### Table F. Statistical analysis (Mean ± STD) of ablation studies measured by PSNR. |PSNR|Coffee Martini|Cook Spinach|Cut Roasted Beef|Flame Salmon|Flame Steak|Sear Steak|Average| |-|-|-|-|-|-|-|-| |w/ Linear position|28.04 ± 0.3174|32.26 ± 0.3547|33.02 ± 0.1712|28.93 ± 0.2574|32.92 ± 0.3782|33.46 ± 0.1081|31.44 ± 0.2645| |w/ Linear rotation|28.25 ± 0.1379|32.21 ± 0.1264|32.78 ± 0.4129|28.68 ± 0.3950|33.08 ± 0.2415|33.49 ± 0.2382|31.41 ± 0.2587| |w/ Linear position&rotation|28.17 ± 0.2974|32.48 ± 0.2138|32.61 ± 0.9545|28.99 ± 0.4750|32.92 ± 0.1230|33.27 ± 0.3726|31.41 ± 0.4061| |w/o Dynamic point extraction|27.59 ± 0.3643|28.95 ± 1.0637|29.44 ± 1.3829|27.43 ± 0.2988|29.18 ± 1.4691|31.39 ± 0.9176|29.00 ± 0.9161| |w/o Temporal opacity|28.18 ± 0.3947|32.23 ± 0.2807|32.18 ± 1.1642|28.81 ± 0.4199|32.84 ± 0.1935|33.58 ± 0.0855|31.30 ± 0.4231| |w/o Progressive growing|28.26 ± 0.2780|32.36 ± 0.1803|32.92 ± 0.3817|29.49 ± 0.2209|32.70 ± 0.6805|32.99 ± 0.3908|31.45 ± 0.3554| |w/o Regularization|28.06 ± 0.4647|32.39 ± 0.1820|32.70 ± 0.7968|28.72 ± 0.2011|33.10 ± 0.3345|33.37 ± 0.1111|31.39 ± 0.3484| |w/o Point backtracking|28.02 ± 0.3477|32.38 ± 0.3267|32.93 ± 0.2917|28.63 ± 0.6187|32.99 ± 0.1653|33.48 ± 0.1177|31.40 ± 0.3113| |Ours|28.43 ± 0.1691|32.77 ± 0.2060|33.31 ± 0.1972|29.11 ± 0.1202|33.41 ± 0.2379|33.23 ± 0.2406|31.71 ± 0.1952| $~$ $~$ ### Table G. Statistical analysis (Mean ± STD) of ablation studies measured by SSIM. |SSIM|Coffee Martini|Cook Spinach|Cut Roasted Beef|Flame Salmon|Flame Steak|Sear Steak|Average| |-|-|-|-|-|-|-|-| |w/ Linear position|0.9180 ± 0.0019|0.9518 ± 0.0012|0.9549 ± 0.0008|0.9234 ± 0.0014|0.9621 ± 0.0010|0.9622 ± 0.0002|0.9454 ± 0.0011| |w/ Linear rotation|0.9178 ± 0.0010|0.9507 ± 0.0005|0.9550 ± 0.0008|0.9213 ± 0.0016|0.9621 ± 0.0005|0.9624 ± 0.0008|0.9449 ± 0.0009| |w/ Linear position&rotation|0.9174 ± 0.0020|0.9526 ± 0.0010|0.9551 ± 0.0005|0.9233 ± 0.0017|0.9620 ± 0.0006|0.9616 ± 0.0006|0.9453 ± 0.0011| |w/o Dynamic point extraction|0.9102 ± 0.0014|0.9345 ± 0.0048|0.9402 ± 0.0047|0.9166 ± 0.0015|0.9501 ± 0.0039|0.9519 ± 0.0026|0.9339 ± 0.0032| |w/o Temporal opacity|0.9164 ± 0.0016|0.9505 ± 0.0013|0.9530 ± 0.0004|0.9213 ± 0.0024|0.9612 ± 0.0006|0.9622 ± 0.0004|0.9441 ± 0.0011| |w/o Progressive growing|0.9166 ± 0.0025|0.9503 ± 0.0015|0.9556 ± 0.0008|0.9266 ± 0.0010|0.9611 ± 0.0018|0.9613 ± 0.0008|0.9453 ± 0.0014| |w/o Regularization|0.9179 ± 0.0013|0.9518 ± 0.0006|0.9545 ± 0.0017|0.9231 ± 0.0008|0.9628 ± 0.0011|0.9625 ± 0.0004|0.9454 ± 0.0010| |w/o Point backtracking|0.9176 ± 0.0008|0.9521 ± 0.0011|0.9547 ± 0.0015|0.9208 ± 0.0036|0.9622 ± 0.0007|0.9624 ± 0.0005|0.9450 ± 0.0014| |Ours|0.9154 ± 0.0013|0.9562 ± 0.0004|0.9572 ± 0.0004|0.9253 ± 0.0005|0.9621 ± 0.0007|0.9603 ± 0.0009|0.9461 ± 0.0007| $~$ $~$ ### Table H. Statistical analysis (Mean ± STD) of ablation studies measured by LPIPS. |LPIPS|Coffee Martini|Cook Spinach|Cut Roasted Beef|Flame Salmon|Flame Steak|Sear Steak|Average| |-|-|-|-|-|-|-|-| |w/ Linear position|0.0723 ± 0.0011|0.0490 ± 0.0009|0.0428 ± 0.0003|0.0759 ± 0.0011|0.0350 ± 0.0006|0.0365 ± 0.0003|0.0519 ± 0.0007| |w/ Linear rotation|0.0723 ± 0.0014|0.0495 ± 0.0008|0.0441 ± 0.0011|0.0777 ± 0.0019|0.0351 ± 0.0002|0.0363 ± 0.0006|0.0525 ± 0.0010| |w/ Linear position&rotation|0.0720 ± 0.0029|0.0486 ± 0.0003|0.0437 ± 0.0011|0.0762 ± 0.0012|0.0354 ± 0.0002|0.0372 ± 0.0007|0.0522 ± 0.0011| |w/o Dynamic point extraction|0.0786 ± 0.0010|0.0749 ± 0.0051|0.0716 ± 0.0096|0.0836 ± 0.0014|0.0642 ± 0.0065|0.0617 ± 0.0070|0.0724 ± 0.0051| |w/o Temporal opacity|0.0725 ± 0.0023|0.0490 ± 0.0004|0.0452 ± 0.0010|0.0772 ± 0.0026|0.0354 ± 0.0007|0.0368 ± 0.0007|0.0527 ± 0.0013| |w/o Progressive growing|0.0699 ± 0.0021|0.0511 ± 0.0007|0.0443 ± 0.0005|0.0734 ± 0.0008|0.0384 ± 0.0036|0.0403 ± 0.0034|0.0529 ± 0.0019| |w/o Regularization|0.0713 ± 0.0030|0.0495 ± 0.0008|0.0437 ± 0.0012|0.0756 ± 0.0016|0.0343 ± 0.0004|0.0369 ± 0.0006|0.0519 ± 0.0013| |w/o Point backtracking|0.0716 ± 0.0019|0.0491 ± 0.0003|0.0452 ± 0.0038|0.0777 ± 0.0027|0.0350 ± 0.0007|0.0364 ± 0.0004|0.0525 ± 0.0016| |Ours|0.0721 ± 0.0019|0.0417 ± 0.0004|0.0416 ± 0.0005|0.0659 ± 0.0005|0.0346 ± 0.0010|0.0357 ± 0.0012|0.0486 ± 0.0009| $~$ $~$ ### Table I. Statistical analysis (Mean ± STD) on the N3V dataset. |Scene|PSNR|SSIM|LPIPS| |-|-|-|-| |Coffee Martini|28.43±0.169|0.915±0.0013|0.0720±0.0009| |Cook Spinach|32.77±0.206|0.956±0.0004|0.0420±0.0004| |Cut Roasted Beef|33.31±0.197|0.957±0.0004|0.0416±0.0005| |Flame Salmon|29.11±0.120|0.925±0.0005|0.0659±0.0005| |Flame Steak|33.41±0.238|0.962±0.0007|0.0346±0.0010| |Sear Steak|33.23±0.241|0.960±0.0009|0.0357±0.0012| |Average|31.71±0.195|0.946±0.0007|0.0490±0.0009|
Reference tables
### Table J. Experimental results on different keyframe intervals and skipped frames. | Skipped frames | 1 | | | | 2 | | | | 3 | | | | |-|-|-|-|-|-|-|-|-|-|-|-|-| | Keyframe interval | PSNR | SSIM | LPIPS | size (MB) | PSNR | SSIM | LPIPS | size (MB) | PSNR | SSIM | LPIPS | size (MB) | | 1 | 31.17 | 0.948 | 0.057 | 595 | 31.47 | 0.948 | 0.056 | 415 | 31.81 | 0.946 | 0.051 | 142 | | 2 | 32.06 | 0.952 | 0.051 | 314 | 32.33 | 0.954 | 0.049 | 322 | 31.81 | 0.953 | 0.044 | 101 | | 5 | 31.70 | 0.953 | 0.047 | 206 | 32.29 | 0.954 | 0.043 | 126 | 32.53 | 0.954 | 0.045 | 80 | | 10 | 33.04 | 0.956 | 0.041 | 119 | 32.65 | 0.956 | 0.043 | 93 | 31.79 | 0.953 | 0.046 | 74 | | 20 | 32.78 | 0.955 | 0.043 | 90 | 32.07 | 0.952 | 0.047 | 78 | 32.08 | 0.953 | 0.048 | 73 | | 50 | 32.14 | 0.955 | 0.046 | 79 | 31.93 | 0.951 | 0.052 | 72 | 30.91 | 0.949 | 0.056 | 73 | $~$ $~$ ### Table K. Quantitative comparisons on the Train scene in the Technicolor dataset. | Technicolor Train longer | PSNR | SSIM | LPIPS | |-|-|-|-| | STG | 32.17 | 0.940 | 0.035 | | 4DGS | 29.11 | 0.877 | 0.119 | | 4D Gaussians | 23.31 | 0.657 | 0.385 | | Ours | 32.18 | 0.938 | 0.044 |
Thanks for the response
I'd like to thank the authors for their detailed response to my comments. I think that with respect to questions regarding explanations, I am fully clear in what authors did in their method (and I think this should be reflected in the final version of the manuscript). The breakdown per scene for Technicolor looks interesting as well (it invites some analysis/comments on why the authors believe their methods improve in particular scenes etc.). As for the long scene experiment , it would be great to see at some point a comparison with other methods (I am not asking for this within rebuttal, just believe that it can increase the value of the experimental section in the future).
We are pleased that our responses help to clarify the understanding of our work. Your insights and suggestions are valuable for us. We are also interested in experimenting with long durations and see results when we use different hyperparameters/iterations. Additionally, we report the experimental result on the long duration video over the comparison methods below: | | PSNR | SSIM | LPIPS | |-|-|-|-| | 4DGS | 26.73 | 0.899 | 0.112 | | 4D Gaussians | 28.48 | 0.905 | 0.095 | | **Ours** | **28.77** | **0.919** | **0.076** | We will add the additional results and analysis to the revised version.
Summary
This paper models dynamic scenes using 3DGS, unlike other methods that model dynamic scenes with both implicit and explicit representations. It proposes Fully Explicit Dynamic Gaussian Splatting (FEDGS), a method that models 4D scenes using a purely explicit approach. FEDGS employs a Cubic Hermite Interpolator to predict positions at different times and a Spherical Linear Interpolation for predicting rotations at different times. FEDGS can fit the dynamic scenes with sparse initialization in the N3V dataset without any auxiliary modules or tricks to encode lengthy temporal information.
Strengths
1. Modeling 4D scenes with a purely explicit method is rare, and FEDGS can render at 120fps under 157MB, and the training speed is fast. 2. FEDGS, as a purely explicit method, achieves better results in the case of sparse COLMAP point clouds. Many methods based on 3DGS for dynamic scene reconstruction face challenges with sparse initial point clouds.
Weaknesses
1. In the supplementary video, compared to other methods, many positions that should be static are moving, causing flickering in objects like hats and tables, and even some Gaussian spheres can be seen rotating. Is this a drawback of the method described in section 4.2.2? This is completely different from the results of previous methods like NeRFPlayer [69] that separate dynamic and static elements, and this part lacks further explanation and clarification. 2. The most attractive aspect of FEDGS is its dynamic scene reconstruction results under sparse COLMAP point cloud conditions, which may be an advantage of the purely explicit approach. However, the paper does not clearly explain this, and providing more experiments could highlight the advantages of the purely explicit method, enhancing the paper's contribution. 3. Compared to 4DGaussians, the rendering speed is slower. Is this due to slower predictions of positions and rotations for 3DGS? However, 4DGaussians use hexplane predictions, which should have a higher computational cost than formulas 6 and 9.
Questions
Why the purely explicit approach can bring improvements under sparse COLMAP point cloud conditions is not clearly explained in the paper. I would be pleased to see the author address the issues mentioned in the weakness section, improving the quality and contribution of the paper.
Rating
5
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
After separating dynamic and static Gaussians, the rendering result of the static parts may experience jitter, which is the main limitation, and it's worth more exploration here.
Reference tables
### Table A. Quantitative results of the experiment on handling color changes without dynamic points. | Model | PSNR | SSIM | LPIPS | |-|-|-|-| | 3DGS | 21.69 | 0.851 | 0.126 | | 3DGS + ours dynamic | 26.07 | 0.891 | 0.089 | | Ours | 29.03 | 0.922 | 0.068 | $~$ $~$ ### Table B. Quantitative comparison on the Train scene in the Technicolor dataset. | Technicolor Train longer | PSNR | SSIM | LPIPS | |-|-|-|-| | STG | 32.17 | 0.940 | 0.035 | | 4DGS | 29.11 | 0.877 | 0.119 | | 4D Gaussians | 23.31 | 0.657 | 0.385 | | Ours | 32.18 | 0.938 | 0.044 | $~$ $~$ ### Table C. Quantitative results on the Birthday scene in the Technicolor dataset. | Technicolor Birthday longer | PSNR | SSIM | LPIPS | |-|-|-|-| | Ours | 29.12 | 0.900 | 0.094 | $~$ $~$ ### Table D. Quantitative results on the extremely long duration video sequences (Flame Salmon) in the N3V dataset. | Flame Salmon longer | PSNR | SSIM | LPIPS | Size (MB) | |-|-|-|-|-| | Ours | 28.77 | 0.919 | 0.076 | 392 | $~$ $~$ ### Table E. Quantitative evaluation on the Technicolor dataset. | Method | PSNR | SSIM | LPIPS | FPS | |-|-|-|-|-| | 4D Gaussians | 30.79 | 0.847 | 0.180 | 45 | | Ours | 33.30 | 0.910 | 0.100 | 62 |
Summary
The authors propose a fully explicit dynamic Gaussian splatting method, based on keyframe interpolation. The authors separate a dynamic scene into static Gaussians and dynamic Gaussians during training and apply interpolation techniques under temporal explicit representation, including a polynomial basis interpolator for position, a spherical interpolator for rotation, and a simplified Gaussian mixture model for opacity. Additionally, the authors introduce a progressive training scheme and a point-backtracking technique to improve the final convergence. The proposed method was validated on the Neural 3D Video dataset and Technicolor dataset, and exceeds the baselines.
Strengths
1. The paper is well-written and easy to understand. 2. The proposed designs are well-validated. The authors have done a lot of ablation experiments to validate their designs. 3. As a fully explicit method, the paper shows impressive results regarding to rendering quality, model size, and rendering efficiency.
Weaknesses
1. Some important baselines are missing, including [1], [2], and [3]. All of them had released their code before the NeuriPS deadline. They have shown better quality than most baselines selected by this paper. 2. Some metrics are missing. I am not sure why in Table 1, the authors only report PSNR. The average LPIPS and SSIM comparison should be included. 3. The quality improvement is minor compared with some baselines. According to Table 1, the average PSNR is just about 0.2dB higher than some baselines. [1]3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos [2]4K4D: Real-Time 4D View Synthesis at 4K Resolution [3]Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes
Questions
1. I am curious about the comparison results between this method and the baselines I mentioned in the weaknesses section. 2. I am curious about the choice of keyframe interval. According to L259, the time interval is set to be 10. I am curious why the authors chose 10 to be the final hyperparameters. Will a different interval number like 5 or 20 affect the final quality?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The limitations have been well discussed by the authors.
Reference tables
### Table A. Quantitative evaluation on the N3V Dataset using PSNR. | PSNR | Coffe Martini | Cook Spinach | Cut Roasted Beef | Flame Salmon | Flame Steak | Sear Steak | Average | |-|-|-|-|-|-|-|-| | NeRFPlayer | 31.53 | 30.56 | 29.35 | 31.65 | 31.93 | 29.13 | 30.69 | | HyperReel | 28.37 | 32.30 | 32.92 | 28.26 | 32.20 | 32.57 | 31.10 | | Neural Volumes | N/A | N/A | N/A | 22.80 | N/A | N/A | 22.80 | | LLFF | N/A | N/A | N/A | 23.24 | N/A | N/A | 23.24 | | DyNeRF | N/A | N/A | N/A | 29.58 | N/A | N/A | 29.58 | | HexPlane | N/A | 32.04 | 32.55 | 29.47 | 32.08 | 32.39 | 31.71 | | K-Planes | 29.99 | 32.60 | 31.82 | 30.44 | 32.38 | 32.52 | 31.63 | | MixVoxels-L | 29.63 | 32.25 | 32.40 | 29.80 | 31.83 | 32.10 | 31.34 | | MixVoxels-X | 30.39 | 32.31 | 32.63 | 30.60 | 32.10 | 32.33 | 31.73 | | Im4D | N/A | N/A | 32.58 | N/A | N/A | N/A | 32.58 | | 4K4D | N/A | N/A | 32.86 | N/A | N/A | N/A | 32.86 | | Dense COLMAP point cloud input | | | | | | | | | STG | 28.41 | 32.62 | 32.53 | 28.61 | 33.30 | 33.40 | 31.48 | | 4DGS | 28.33 | 32.93 | 33.85 | 29.38 | 34.03 | 33.51 | 32.01 | | 4DGaussians | 27.34 | 32.46 | 32.90 | 29.20 | 32.51 | 32.49 | 31.15 | | Sparse COLMAP point cloud input | | | | | | | | | STG | 27.71 | 31.83 | 31.41 | 28.06 | 32.17 | 32.67 | 30.64 | | 4DGS | 26.51 | 32.11 | 31.74 | 26.93 | 31.44 | 32.42 | 30.19 | | 4DGaussians | 26.69 | 31.89 | 25.88 | 27.54 | 28.07 | 31.73 | 28.63 | | 3DGStream | 27.75 | 33.31 | 33.21 | 28.42 | 34.30 | 33.01 | 31.67 | | Ours | 29.04 | 32.46 | 33.21 | 29.56 | 33.23 | 33.84 | 31.89 | $~$ $~$ ### Table B. Quantitative evaluation on the N3V dataset using SSIM$_{1}$. | SSIM$_{1}$ | Coffe Martini | Cook Spinach | Cut Roasted Beef | Flame Salmon | Flame Steak | Sear Steak | Average | |-|-|-|-|-|-|-|-| | NeRFPlayer | 0.951 | 0.929 | 0.908 | 0.940 | 0.950 | 0.908 | 0.931 | | HyperReel | 0.892 | 0.941 | 0.945 | 0.882 | 0.949 | 0.952 | 0.927 | | Dense COLMAP point cloud input | | | | | | | | | STG | 0.916 | 0.952 | 0.954 | 0.918 | 0.960 | 0.961 | 0.944 | | 4DGS | N/A | N/A | 0.980 | 0.960 | N/A | N/A | 0.970 | | 4DGaussians | 0.905 | 0.949 | 0.957 | 0.917 | 0.954 | 0.957 | 0.940 | | Sparse COLMAP point cloud input | | | | | | | | | STG | 0.904 | 0.946 | 0.946 | 0.913 | 0.954 | 0.955 | 0.936 | | 4DGS | 0.902 | 0.948 | 0.947 | 0.904 | 0.954 | 0.955 | 0.935 | | 4DGaussians | 0.893 | 0.944 | 0.913 | 0.896 | 0.946 | 0.946 | 0.923 | | Ours | 0.915 | 0.947 | 0.948 | 0.917 | 0.956 | 0.959 | 0.940 | $~$ $~$ ### Table C. Quantitative evaluation on the N3V dataset using SSIM$_{2}$. | SSIM$_{2}$ | Coffe Martini | Cook Spinach | Cut Roasted Beef | Flame Salmon | Flame Steak | Sear Steak | Average | |-|-|-|-|-|-|-|-| | Neural Volumes | N/A | N/A | N/A | 0.876 | N/A | N/A | 0.876 | | LLFF | N/A | N/A | N/A | 0.848 | N/A | N/A | 0.848 | | DyNeRF | N/A | N/A | N/A | 0.960 | N/A | N/A | 0.960 | | HexPlane | N/A | 0.983 | 0.985 | 0.980 | 0.988 | 0.986 | 0.984 | | K-Planes | 0.953 | 0.966 | 0.966 | 0.953 | 0.970 | 0.974 | 0.964 | | MixVoxels-L | 0.951 | 0.968 | 0.966 | 0.949 | 0.971 | 0.976 | 0.964 | | MixVoxels-X | 0.954 | 0.968 | 0.971 | 0.953 | 0.973 | 0.976 | 0.966 | | Im4D | N/A | N/A | 0.970 | N/A | N/A | N/A | 0.970 | | 4K4D | N/A | N/A | 0.972 | N/A | N/A | N/A | 0.972 | | Dense COLMAP point cloud input | | | | | | | | | STG | 0.910 | 0.947 | 0.950 | 0.913 | 0.956 | 0.958 | 0.939 | | 4DGS | N/A | N/A | N/A | N/A | N/A | N/A | 0.972 | | Sparse COLMAP point cloud input | | | | | | | | | STG | 0.898 | 0.940 | 0.939 | 0.907 | 0.949 | 0.950 | 0.931 | | 4DGS | 0.894 | 0.944 | 0.943 | 0.896 | 0.951 | 0.951 | 0.930 | | 4DGaussians | 0.886 | 0.939 | 0.907 | 0.889 | 0.942 | 0.942 | 0.917 | | Ours | 0.922 | 0.951 | 0.953 | 0.923 | 0.960 | 0.963 | 0.945 | $~$ $~$ ### Table D. Quantitative evaluation on the N3V dataset using LPIPS. | LPIPS | Coffe Martini | Cook Spinach | Cut Roasted Beef | Flame Salmon | Flame Steak | Sear Steak | Average | |-|-|-|-|-|-|-|-| | NeRFPlayer | 0.085 | 0.113 | 0.144 | 0.098 | 0.088 | 0.138 | 0.111 | | HyperReel | 0.127 | 0.089 | 0.084 | 0.136 | 0.078 | 0.077 | 0.096 | | Neural Volumes | N/A | N/A | N/A | 0.295 | N/A | N/A | 0.295 | | LLFF | N/A | N/A | N/A | 0.235 | N/A | N/A | 0.235 | | DyNeRF | N/A | N/A | N/A | 0.083 | N/A | N/A | 0.083 | | HexPlane | N/A | | | 0.098 | | | 0.098 | | K-Planes | 0.024 | 0.017 | 0.017 | 0.024 | 0.015 | 0.013 | 0.018 | | MixVoxels-L | 0.106 | 0.099 | 0.088 | 0.116 | 0.088 | 0.080 | 0.096 | | MixVoxels-X | 0.081 | 0.062 | 0.057 | 0.078 | 0.051 | 0.053 | 0.064 | | Dense COLMAP point cloud input | | | | | | | | | STG | 0.069 | 0.043 | 0.042 | 0.063 | 0.034 | 0.033 | 0.047 | | 4DGS | N/A | N/A | 0.041 | N/A | N/A | N/A | 0.055 | | Sparse COLMAP point cloud input | | | | | | | | | STG | 0.087 | 0.056 | 0.060 | 0.074 | 0.046 | 0.046 | 0.062 | | 4DGS | 0.079 | 0.041 | 0.041 | 0.078 | 0.036 | 0.037 | 0.052 | | 4DGaussians | 0.095 | 0.056 | 0.104 | 0.095 | 0.050 | 0.046 | 0.074 | | Ours | 0.068 | 0.049 | 0.048 | 0.075 | 0.035 | 0.036 | 0.052 |
Thanks to the authors for their explanations. However, there are still some left questions. Thanks to the authors for their explanations. However, there are still some left questions. 1. In your answer to [W2], I think you want to claim that your proposed method can work well in scenarios with new objects appearing. However, you also claimed in limitations that that is one of the limitations. I am confused with it. It seems the qualitative result presented is good, but the quantitative result is not as good as the performance of your method on the other dataset. 2. Please clarify which data does the experiments on the dynamic point ratio in Tab C are conducted on. 3. I am still concerning the empirically selected dynamic points ratio and the intervals between keyframes might limit the generalization of the proposed methods. 4. I am still confused about the answers to [Q3], what do you mean by 'the COLMAP result is provided for the first frame (about 20 images) out of a total of 300 frames'? Does it mean these 20 images are multi-view images at the same time?
Thank you for your thoughtful feedback, your comments are valuable to us. Here are the answers to your questions: 1. We mean that the newly appeared object mentioned in the Section 6 has **"no relevant 3D Gaussians in neighboring frames"**. Here "relevant 3D Gaussians" refers to **3D Gaussians that are close to be densified from the object**. These Gaussains are not only Gaussians corresponding to the same object, but also Gaussians in neighboring objects (in Figure D of the uploaded pdf, neighboring Gaussains of the person could be a table and decorations). So, it is true that FEDGS can handle new objects that appear. However, it is still difficult to represent suddenly appeared objects with no neighboring objects. Although we wanted to show this example, unfortunately, there is no scene where objects suddenly appear in the public datasets. Instead, as shown in Figure D of the uploaded pdf, we display the result on slowly appearing objects. \ In this rebuttal, we chose the video with a different time range for the additional experiment. The frame range includes newly appearing objects **(Frame #50~#169)**. Therefore, the performance can be different between the additional experiments and the main paper **(Frame #151~#200)**. 2. We experiment Table C on the Cooking Spinach scene from the N3V dataset. 3. First, for the static point extraction, **the percentage does not determine the ratio of dynamic points**. In our model, pruning and densification are applied to both static and dynamic points, automatically reducing or increasing the number of both. Even if we convert more static points to dynamic points, and if it is not excessive, the static points will be split and the dynamic points will be pruned to correct the erroneous conversion, and vice versa. Therefore, the extraction percentile is varied during optimization, not fixed value to determine the ratio of dynamic points. Actually, when we used the same extraction percentile, 56.4% of the points were finally considered as dynamic points on the Birthday scene from the Technicolor dataset, while 32.3% of the points were dynamic points on the Cook Spinach scene from the N3V dataset. Unlike the extraction percentile, we do not control the ratio of dynamic points; it depends on the dataset or scene. \ For more generality of the keyframe interval, as the SWinGS (ECCV 24), mentioned by reviewer Vpue, an adaptive keyframe interval selection based on a magnitude of optical flow between frames can be one possible solution. By using this, we can expect that dynamic keyframe allocation is available, varying keyframe intervals even in the same scene. In total, we agree with your concern on the generality and will mention it as one of our future works in the revised version of this paper. 4. Exactly. In the case of the N3V dataset, it is a multi-view video dataset with 300 frames captured by approximately 20 time-synchronized cameras (the number of cameras varies from scene to scene). We only use the 20 images taken in the first frame. We hope our answers will help you better understand our work. And if you have any other questions, don't hesitate to ask us.
Thanks for the authors' response! I have carefully read the rebuttal, and most of my concerns have been addressed. Although Table E shows good rendering speed, it does not show the storage cost, which is 392MB in the complex scene shown in Table D. There is still a concern about the balance between speed and storage in the explicit method proposed in this paper.
Thanks for your comments that our rebuttal addresses almost all your concerns. Here is the answer to your additional question: We report the results on both the Technicolor dataset and the N3V dataset. As shown in the tables below, the 4D Gaussians model has a larger variation in FPS, despite having a similar storage size in both datasets. This validates that rendering speed is more dependent on the scene configuration than on storage size. Our method, on the other hand, shows consistent performance regardless of scenes. We note that regarding the storage size, around 300MB should be manageable for most devices in these days (note that the storage size of public datasets used is about GB). As another aspect, we measure shared GPU memory of ours and the 4D Gaussian on a machine with one NVIDIA 2080Ti. The difference between them is marginal, which means both models can be worked on the same GPU machine. Nevertheless, we recognize that our model takes more storage than 4D Gaussians. We will include an introduction of compression techniques [1, 2] into our model to save storage sizes as one of our future works. We expect that it will be feasible because our model preemptively separates a static part from a scene which is easily compressed. $~$ ### Table F. Averaged results from Technicolor dataset | Method | PSNR | SSIM | LPIPS | FPS | Size (MB) | GPU memory (GB) | |-|-|-|-|-|-|-| | 4D Gaussians | 30.79 | 0.847 | 0.180 | 45 | **44** |**1.3**| | **Ours** | **33.30** | **0.910** | **0.100** | **62** | 307 |1.6| $~$ ### Table G. Averaged results from N3V dataset | Method | PSNR | SSIM | LPIPS | FPS | Size (MB) | GPU memory (GB) | |-|-|-|-|-|-|-| | 4D Gaussians | 26.69 | 0.923 | 0.074 | **147** | **34** |1.4| | **Ours** | **29.04** | **0.940** | **0.052** | 121 | 157 |**1.1**| $~$ [1] Simon N, et. al., _Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis_, Conference on Computer Vision and Pattern Recognition, 2024 [2] Joo Chan L, et. al., _Compact 3D Gaussian Representation for Radiance Field_, Conference on Computer Vision and Pattern Recognition, 2024
Thanks for your replies. Although the papers have shown some strengths and the rebuttal has answered some of my concerns, it still misses some important baselines as I mentioned, and its overall performances on different metrics (PSNR, LPIPS, and SSIM) are not SOTA. Therefore, I am inclined to maintain my score. I strongly suggest the authors include more baselines and metrics in the revised paper.
We are sorry for the unsatisfactory response for your concern. However, it is notable that we have tried to reproduce the comparison baselines that you suggest. To be specific, both 4K4D and Im4D require trained backgrounds of scenes, but they are not provided in their official source codes, which makes it impossible to run them. For 3DGStream, we also have used the authors’ provided code, but were unable to see the same performance as reported in the original paper: we just see PSNR 32.55dB on Cook Spinach scene, which is lower performance than reported 33.31dB. We are now contacting the authors to investigate these issues, and will make it clear in the revised version.
Thanks for your reply. I can understand why some baselines are missing for now. Another important issue is that even without comparing with the baselines I suggested, the proposed methods can't always be SOTA on every metric. For example, even though K-Planes has a slightly lower PSNR, its performance is much better than the proposed methods regarding LPIPS and SSIM_2.
We sincerely apologize that we made a mistake while directly referring to the quantitative metrics of LPIPS from STG paper (the scene-specific LIPIS for K-Planes is not reported). We ask you to check the Table.11 in STG paper. The original metrics that we want to directly refer is below: | | PSNR | SSIM | LPIPS | |-|-|-|-| | K-Planes | 31.63 | 0.964 | N/A | | STG | 31.48 | 0.939 | 0.047 | | Ours | 31.89 | 0.945 | 0.052 | As shown in the table above, we would like to highlight that K-Planes does not show SoTA results. That’s, with respect to PSNR, our method achieves SoTA results. In contrast, STG and K-Plane show the best results in terms of LPIPS and SSIM, respectively. At this point, we agree with your opinion that our method does not produce the best result in every evaluation metric. In addition, we would like to describe the root of this problem. 4D Gaussians and K-Plane only report per scene breakdown for both PSNR and SSIM values, and 3DGStream and 4DGS only show PSNR. As discussed in Appendix C.3. of HyperReel, there is no consistent standard implementation of SSIM used. As an example, a work in [1] uses their own implementation of SSIM, not library function. Therefore, to reliably collect all the PSNR, SSIM and LPIPS values of the comparison methods, we had to refer to them from multiple papers, leading to the mistake. Lastly, we clearly indicate references of the PSNR, SSIM and LPIPS values used in this rebuttal below (Method - Reference): * PSNR \ NeRFPlayer, HyperReel, Neural Volumes, LLFF, DyNeRF, HexPlane, K-Planes, MixVoxels-L, MixVoxels-X - Table 11 of STG paper \ Im4D, 4K4D - Table 6 of STG paper \ 4DGS - Table 4 of 4DGS paper \ 4DGaussians - Table 6 of 4DGaussians paper \ 3DGStream- Table 5 of 3DGStream paper * SSIM1 \ NeRFPlayer, HyperReel - Table 11 of STG paper \ 4DGS - Table 3 of 4DGS paper \ 4DGaussians - Table 6 of 4DGaussians paper * SSIM2 \ Neural Volumes, LLFF, DyNeRF, HexPlane, K-Planes, MixVoxels-L, MixVoxels-X - Table 11 of STG paper \ Im4D, 4K4D - Table 6 of STG paper * LPIPS \ NeRFPlayer, HyperReel, Neural Volumes, LLFF, DyNeRF, HexPlane, MixVoxels-L, MixVoxels-X - Table 11 of STG paper \ Im4D, 4K4D - Table 6 of STG paper \ 4DGS - A rebuttal experiment for “Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting” in OpenReview of ICLR’24 [1] Wang, L, et al. "Fourier plenoctrees for dynamic radiance field rendering in real-time". _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2022.
Decision
Accept (poster)