We thank the reviewer for the follow-up.
> Coarse splatting
We elaborate on the coarse splatting in further detail below:
Consider an optimized 3DGS model for a scene, $\mathcal{G}$. For $n$ viewpoints, we obtain masks $\mathcal{M}$ := {$M^j$}$_{j=1}^n$
from a video segmentation model corresponding to $n$ viewpoints $\mathcal{I}$ := {$I^j$}$_{j=1}^n$ . $M^j$ indicates the set of foreground pixels in the viewpoint $I^j$.
For each Gaussian $g \in \mathcal{G}$, we maintain a weight, $w_g$, that indicates the likelihood of this Gaussian belonging to the foreground. To obtain the likelihood term $w_{g}^j$ pertaining to mask $j$ for Gaussian $g$, we unproject the posed image $I^j$ back to the Gaussians using inverse rendering and utilizing the mask information,
$$
w_g^j = \frac{\sum_{\textbf{p} \in M^j } \sigma_g(\textbf{p})T_g^j(\textbf{p}) }{\sum_{\textbf{p} \in {I}^j} \sigma_g(\textbf{p})T_g^j(\textbf{p})}
$$
where $\sigma_g(\textbf{p})$ and $T_g^j(\textbf{p})$ denote the opacity and transmittance from pixel $\textbf{p}$ for Gaussian $g$. If $g$ does not contribute to $\textbf{p}$, the transmittance is taken to be $0$. Combining over all the masks,
$$
w_g = \frac{\sum_{j}\sum_{\textbf{p} \in M^j } \sigma_g(\textbf{p})T_g^j(\textbf{p})}{\sum_j\sum_{\textbf{p} \in {I}^j} \sigma_g(\textbf{p})T_g^j(\textbf{p}) }
$$
As mentioned in the paper, we use the same formulation as proposed by GaussianEditor [a] and kindly refer the reviewer to their paper for further details.
For binary weights (hard assignment), we simply keep a count of the number of foreground and background pixels the Gaussian $g$ splats to in $I^j$.
$$
w_g = \frac{\sum_{j}\sum_{\textbf{p} \in M^j } \mathbb{I}(T_g^j > 0) }{\sum_j\sum_{\textbf{p} \in {I}^j} \mathbb{I}(T_g^j > 0)}
$$
This $w_g$ is used directly in Equation 3 in the paper. We show the IoU on several scenes comparing soft assignments and hard assignments. Since soft assignment has marginally better performance, it is our default implementation.
| Scene | Soft assignment | Hard assignment |
|--------------------|-----------------|-----------------|
| Fern (NVOS) | 83.06 | 82.56 |
| Fortress (NVOS) | 97.94 | 98.12 |
| Leaves (NVOS) | 95.95 | 95.60 |
| Lego (SPIn-NeRF) | 89.18 | 88.95 |
| Pinecone (SPIn-NeRF)| 91.89 | 91.99 |
> Novelty concerns
Regarding the novelty, the graph construction proposed in our method is significantly different from the point cloud literature. Our proposed method is much simpler and both the approaches mentioned by the reviewer involve training networks to obtain the segments.
- GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds: This method does not employ graph cut for segmentation. It learns per-point features, extracts superpoints, progressively grows those superpoints, and performs clustering for segmentation.
- SAM-guided Graph Cut for 3D Instance Segmentation: While this method does use graph cut on point clouds, their formulation differs significantly from our approach. They apply graph cut on superpoints which are obtained from another pre-segmentation model. Therefore, when an object to be selected is part of a superpoint, this technique is unable to segment it out. Our approach, on the other hand, provides finer user control as we do not rely on any point cloud pre-segmentation module. Their primary method involves training a graph neural network (GNN) using high-quality pseudo labels whereas our approach is training-free. Their non-GNN baseline also requires masks from multiple views to assign edge weights. Our proposed approach can work even with a single mask. Moreover, since this method stores SAM features, adapting it for 3DGS would require a much higher memory footprint than our approach (please see discussion section).
More generally, while some advancements in point cloud segmentation can be tranferrable for 3DGS, our coarse splatting (section 3.3) and terminal links (section 3.4) are heavily tailored for 3DGS.
[a] GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting, CVPR 2024