Reply to Reviewer oLUf
We sincerely appreciate your time and efforts in reviewing our responses. We hope that our responses below adequately address your concerns.
***Q1: Clarification of our novelty***
***A1:*** We strongly disagree with the reviewer's statement about novelty. This assertion is not only inaccurate but also a misrepresentation of our previous responses. Our method aims to address the overfitting and maintain the multi-view consistency, as detailed in Section 3.2. While SPARF also aims to mitigate overfitting, a challenge common to sparse view synthesis, the similarity with our method ends there.
Our method utilizes the correspondence priors to guide multi-view projection, filtering out outliers and generating reliable pseudo-labels, which have been verified to be crucial. Subsequently, we compute the appearance/geometry/semantic losses with gradient weighting for the novel view. This not only broadens the available view information but also maintains multi-view consistency. Notably, during loss computation, we again utilize correspondence priors to select the minimal loss among each matched pair, which can reduce the effect of large noises. Table 4ix and 4x prove the effectiveness of the minimal operation.
Our method diverges significantly from SPARF in both approach and outcomes, delivering superior performance on 3DGS. The depth consistency loss (DCons) employed in SPARF is fundamentally limited by its reliance on single-view projections, where depth inaccuracies lead to flawed projections. The DCons in SPARF's original paper improves PSNR by 0.20 (from 20.81 to 21.01) on NeRF—a result that aligns with our own reproduction on 3DGS. This proves that the single-view projection on SPARF is insufficient to address the overfitting issue on both NeRF and 3DGS. In contrast, our depth supervision for novel views in Table 4iii improves the PNSR by 3.13.
***Q2: Analysis for semantic loss***
***A2:*** As explained in Lines 271-277 and Lines 441-443, DINOv2 and CLIP down-sample features early in the encoder with a large stride, e.g. 16. It is widely recognized that features extracted with a large stride often result in a loss of detail, which is the reason why the semantic loss with DINOv2 and CLIP obtains poorer performance. Additionally, Figure 3 further illustrates that using DINOv2 and CLIP as feature extractors generates worse results in some boundary and detailed regions. In contrast, our semantic loss using VGG can reduce artifacts and maintain details.
***Q3: Details of SFM initialization***
***A3:*** We have already explained in our previous response that we initialize the 3D Gaussians with dense reconstruction in COLMAP. This process involves two parts, SfM followed by MVS, and does estimate camera extrinsics during SfM, therefore, the statement that "this process does not estimate any camera poses" is simply incorrect. Training and test views are both served as input for SfM to obtain their poses, then only the training views are used during MVS to get the fused point clouds. The technically sound initialization method name for the shorthand "SfM initialization" would be "Initialization based on the poses estimated by SfM for all views, and the fused point clouds from MVS on training views only", which will need to be shortened when referring to. Thus the initialization method we used is actually SfM+MVS, rather than just MVS as the reviewer suggested.
The integrity of our initialization is by no means affected by the name, we ensure the dense point clouds are only from the training views, making it fair and comparable with the baselines.
***Q4: Comparison of training time and results***
***A4:*** Our method, which incorporates RoMa and VGG, achieves significantly reduced training time while delivering superior performance in sparse view synthesis compared to networks relying on pre-trained depth/diffusion models, such as FSGS and SparseGS. As noted in our previous responses, only the VGG model introduces additional training time. However, as detailed in Table S7 of the uploaded file, our method requires just 5.82 minutes of training—substantially less than the 51.78 minutes needed by SparseGS (which uses a pre-trained diffusion model) and the 10.90 minutes required by FSGS (which relies on a pre-trained depth network).
Furthermore, our approach not only enhances efficiency but also surpasses SparseGS and FSGS in performance, as demonstrated in Table 1. While incorporating VGG does involve some additional training time compared to raw 3DGS, the efficiency and speed trade-off is outweighed by the significant improvements in performance. Our method offers considerable gains over both raw 3DGS and methods reliant on pre-trained depth or diffusion models. For example, it improves PSNR by 4.09 dB on DTU and 3.32 dB on Blender compared to 3DGS.
In summary, the integration of RoMa and VGG enhances the overall speed and effectiveness of our method, making it a notable advancement over previous approaches.