Despite alleviating the dependence on dense annotations inherent to fully supervised methods, weakly supervised point cloud semantic segmentation suffers from inadequate supervision signals. In response to this challenge, we introduce a novel perspective that imparts auxiliary constraints by regulating the feature space under weak supervision. Our initial investigation identifies which distributions accurately characterize the feature space, subsequently leveraging this priori to guide the alignment of the weakly supervised embeddings. Specifically, we analyze the superiority of the mixture of von Mises-Fisher distributions (moVMF) among several common distribution candidates. Accordingly, we develop a Distribution Guidance Network (DGNet), which comprises a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined latent space. Extensive experiments validate the rationality and effectiveness of our distribution choice and network design. Consequently, DGNet achieves state-of-the-art performance under multiple datasets and various weakly supervised settings.
Paper
Similar papers
Peer review
Summary
In this work, the authors use a selected feature space distribution as supplementary supervision signals, combining with cosine similarity as the distance metric, and implement a dual-branch weakly supervised point cloud semantic segmentation network. Experiments show that using feature space distribution as supervision information can improve the performance of existing point cloud semantic segmentation networks.
Strengths
This work proposes to use moVMF distribution to regulate the feature space for weakly supervised point cloud semantic segmentation. Experimental results show that this approach can improve the performances of backbone networks.
Weaknesses
As the feature space distribution is selected in a fully supervised manner, it is questionable whether it is suitable for weakly supervised scenarios where labeled samples are extremely sparse. Directly comparing different distributions in the distribution alignment branch of DGNet is more convincing. Eq. 1 in supplementary material, the last term should be log(a_zi) + k_zi * u^T_zi * vi, instead of log(a_zi) * k_zi * u^T_zi * vi. In Eq. 9, why use the same weight for each loss term? Considering that these losses are defined from very different perspectives, their values can be of different orders of magnitude. There are some minor flaws in the writing: some tables and figures are not referenced in the text, and some figures are not numbered in the order referenced.
Questions
Please respond to my comments written above.
Rating
4
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
Evaluation on outdoor datasets is insufficient.
Dear Reviewer fVFW: Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows: * One comparison experiment in the distribution alignment branch of DGNet for several distributions. * One weight balance experiment for tuning the weights of Eq. 9. * One detailed discussion of DGNet in outdoor scenarios. We understand your time is valuable, but your opinion updates are vital to our work.
I keep my rating as Borderline reject, due to the insufficient evaluation on large-scale outdoor dataset, and the serious error in the objective function.
Thanks for your response! We are pleased to see that three of your original five concerns have been addressed. Although you still maintain your original scores, we feel it is necessary to provide further clarification for the only two remaining. * About outdoor datasets: We sincerely disagree with your assertion that the evaluation of DGNet on outdoor scenarios is inadequate. To illustrate that our evaluation of outdoor datasets is relatively adequate in the field of weakly supervised learning, we show whether the methods are validated on these outdoor scenes by the following table: | Method | SemanticKITTI | Semantic3D | | ------------ | ------------- | ------------ | | SQN | $\checkmark$ | $\checkmark$ | | OTOC | $\times$ | $\times$ | | CPCM | $\times$ | $\times$ | | PointMatch | $\times$ | $\times$ | | PSD | $\times$ | $\checkmark$ | | HybridCR | $\checkmark$ | $\checkmark$ | | DCL | $\checkmark$ | $\times$ | | MILTrans | $\times$ | $\times$ | | ERDA | $\times$ | $\times$ | | MulPro | $\times$ | $\times$ | | **DGNet (Ours)** | $\checkmark$ | $\checkmark$ | This table reveals that **DGNet reaches an adequate level of outdoor validation compared with a bunch of methods.** * About typos: We guarantee that it is a transcription error without affecting the correctness of the method. **We would be regretful if our months of work were to be dismissed simply because of a typo that could be corrected immediately.**
Summary
A weakly-supervised approach has been studied to relieve the annotation burden for point cloud semantic segmentation. Unlike conventional works that mainly use priors (such as similarity or augmentation-based regularization) to overcome the lack of information in weak labels, this paper introduces a novel approach directly aligning the features on the latent space of a neural network. Specifically, the proposed method employs a mixture of von Mises-Fisher distribution to model the feature distribution. This brings meaningful gains on multiple distinct architectures across various benchmarks.
Strengths
1. The feature distribution is a core problem but has not yet been deeply addressed, at least in the field of weakly supervised point cloud semantic segmentation. 2. The methodology and presentation for it are pretty good. 3. The performance is impressive.
Weaknesses
I think this paper has no major flaws in methodological aspects. Therefore, I would like to ask some questions for further discussion. 1. Table 1 shows that a mixture of vMF distributions represents the distribution of normalized features better than the other choices. But why? Can you discuss the "oughtness" in more detail, at least at the level of the hypothesis? For example, why is using a mixture of vMFs better than that of Gaussians? 2. Intuitively, using a hypersphere can represent each feature as an angle, resulting in a more concise distribution. However, if the number of categories increases, this kind of approach may not effectively use the latent space, as all the points on a line map to one single point on the hypersphere (due to normalization). I wonder if the proposed method faces difficulty when handling the dataset including more categories. 3. It will be interesting if the authors check the feature distribution of the neural network trained by fully-supervised learning. Do the features still follow the mixture of vMFs well or not? Can the proposed method even enhance the performance in a fully-supervised setting? 4. Related works. As far as I know, MPRM does not use 2D images (Line 82). And it would be better if the authors could cite recent papers in CVPR 2024.
Questions
See the weaknesses.
Rating
7
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes.
Thanks for your prompt response! We are pleased that our response addresses most of your concerns. We also respect the importance you place on von Mises-Fisher distribution, and we will further elaborate on its superiority in solving the weakly supervised point cloud semantic segmentation task from multiple perspectives in our next release. We consider further analyzing the difference between the Euclidean norm (Gaussian mixture model) and cosine similarity (moVMF) from the perspective of the **Curse of Dimensionality**. * The data becomes extremely sparse in feature space as the feature dimension increases. Most feature vectors are far from each other, causing the Euclidean distance to become ineffective in distinguishing differences between feature vectors. Cosine similarity, on the other hand, is more effective in distinguishing differences between features by measuring the angle between the vectors. * Euclidean norm is very sensitive to scale. In a high-dimensional feature space, if the feature changes at different scales in different dimensions, it may result in the contribution of some dimensions to the Euclidean norm being magnified while the contribution of other dimensions is ignored. Cosine similarity is not affected by the length of the vectors, and the main measure is the angle between the features. Therefore, changes in the scale of different dimensions do not significantly affect the cosine similarity calculation.
Summary
This paper addresses the problem of weakly supervised point cloud semantic segmentation. The authors propose imparting supplementary supervision signals by regulating the feature space under weak supervision. The initial investigation identifies which distributions accurately characterize the feature space in fully supervised learning, subsequently leveraging this prior knowledge to guide the alignment of weakly supervised embeddings. The authors first investigate different feature space characterizations and find that a mixture of von Mises-Fisher (moVMF) distributions with cosine similarity best describes the fully supervised embedding space. Based on this finding, DGNet is proposed with two branches: a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined feature space. Experimental results demonstrate that DGNet achieves state-of-the-art performance on multiple datasets.
Strengths
1. The proposed framework presents a novel and intriguing approach, distinguishing itself from consistency training, self-training, and similarity metric methods. 2. The authors perform a comprehensive study to identify the most appropriate model for characterizing the feature space in fully supervised learning. 3. DGNet is designed with two complementary branches: one dedicated to weakly supervised learning and another focused on distribution alignment. This dual-branch structure facilitates the effective integration of the distribution prior into the learning process. 4. DGNet demonstrates its efficacy by achieving superior results across multiple datasets and various weakly supervised settings.
Weaknesses
1. Computational Complexity: The paper omits a crucial discussion on the computational complexity of the distribution alignment branch, particularly regarding the EM-like algorithm. This is important, given that EM-based clustering methods traditionally tend to be computationally intensive. 2. Ablation Study: The current ablation study would benefit from exploring a broader range of loss term combinations. For instance, examining the performance of the complete loss function without $L_{DIS}$ or without $L_{CON}$ would provide more comprehensive insights into the contribution of each component. 3. Discussion and Analysis of moVMF: The paper lacks a deep analysis or discussion explaining why the mixture of von Mises-Fisher (moVMF) distribution outperforms other distributions. Additionally, it fails to provide formal guarantees on convergence or optimality. A more thorough discussion or justification for the choice of moVMF over alternative distributions would be better.
Questions
See weakness
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
A main limitation is the computational complexity of the training.
Dear Reviewer KHTz: Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows: * A crucial discussion on the computational complexity. * An ablation study for removing the loss terms individually. * The discussion and analysis of the superiority of moVMF. We understand your time is valuable, but your opinion updates are vital to our work.
Summary
This paper outlines a study focusing on distance metric and distribution modeling, highlighting the effectiveness of combining mixture of von Mises-Fisher distributions (moVMF) with cosine similarity. A Distribution Guidance Network (DGNet) is introduced, featuring two main branches: weakly supervised learning and distribution alignment. The network utilizes reliable clustering initialization from the weakly supervised branch to iteratively update moVMF parameters and network parameters, ensuring alignment with moVMF-defined feature spaces.
Strengths
Originality: DGNet is a distribution guidance network that combines a weakly supervised learning branch and a distribution alignment branch. The weakly supervised learning branch makes use of reliable cluster initialization for learning. This means that the network is able to perform preliminary learning and classification of data through clustering methods without detailed labeling. This approach helps to reduce the dependence on large amounts of labeled data and improves the robustness and scalability of the model. The distribution alignment branch alternately updates the parameters of the moVMF (von Mises-Fisher distribution) and the parameters of the network to ensure that the network parameters are aligned with the feature space defined by the moVMF. This method helps to guide the network to learn the structure and characteristics of data distribution, so as to improve the generalization ability and adaptability of the model. Quality: DGNet explains the prediction probability from the perspective of Bayes' theorem and provides the confidence of each prediction, which proves the rationality of DGNet selection distribution and network design. In addition, extensive experiments confirm the soundness and efficacy of the chosen distribution and network design.
Weaknesses
1. The section of 4.2 and 4.3 can be written in more detail. In particular, the explanation of changes in model performance in ablation experiments should be added. Specifically, a thorough explanation of the changes in model performance observed during ablation experiments should be incorporated. Ablation experiments involve systematically removing components or features from the model to analyze their impact on performance. By detailing how each modification affects the overall performance metrics, readers gain insights into the critical elements contributing to the model's effectiveness. 2. In Abstract, there is no a clear problem to address from existing methods of weakly supervised learning for 3D point clouds. Similarly, in Introduction, it’s also not clear about the motivation of this work and existing problems of this topic. 3. There is no main contributions about this work in Introduction. 4. How to balance weights of each loss function in Equation (9)? 5. There is no any qualitative comparison between this work with others. 6. Some quantitative results from this work are also incremental improvements compared to others within one percentage point. 7. Compared with previous work, the performance improvement of DGNet model may not be substantial enough to justify its superiority over existing methods, especially on the ScanNet and SemanticKITTI datasets. 8. In the section of 4.3, why does the performance of the model degrade when optimizing some parameters? What is the impact of parameter optimization on resource consumption?
Questions
Please answer the questions in Weaknesses.
Rating
4
Confidence
5
Soundness
2
Presentation
2
Contribution
2
Limitations
The authors have thoroughly discussed the limitations of DGNet, and there are no potential negative societal impacts associated with their work.
A detailed explanation for R1
Next, we provide a detailed elaboration of the analysis of the experimental results in Sec. 4.2 and 4.3. **Please note that these additions of details do not detract from the main conclusions in the paper, and we believe that some details are so burdensome that may dilute the focus on key experiments and make the paper lengthy.** * **Results on S3DIS.** The gain that DGNet brings to the baseline method is more pronounced at 0.01% label rate compared to 0.1% label rate, since the guidance on feature distribution is more valuable with extremely sparse annotations. The 0.02% label rate denotes a sparse labeling form of "one-thing-one-click", and unlike methods that use this labeling form, we did not introduce super-voxel information to enrich the original sparse labeling. However, DGNet outperforms these methods at a smaller label rate. In addition, we would like to re-emphasize that DGNet demonstrates strong generalization capabilities, both in terms of different annotation scales and various baseline methods. * **Results on ScanNetV2.** Confronted with the diverse categories and versatile scenes of ScanNetV2, we follow the other methods and report on the segmentation performance with the sparse annotations processed by super-voxels. The cross-entropy loss term can generate relatively sufficient supervised information to train the network due to the introduction of pseudo-labelings, resulting in a less pronounced improvement of DGNet than that of S3DIS. However, DGNet is still slightly superior to the latest SOTA methods. * **Results on SemanticKITTI.** In the field of weakly supervised point-cloud semantic segmentation, only a few works provide segment performance on outdoor SemanticKITTI. With the same baseline method RandLa-Net, DGNet improves 1.0% and 2.1% mIoU over SQN at 0.1% label rate and 0.01% label rate, respectively. Considering the inherent density inhomogeneity of outdoor scenes, the excellent performance with such sparse annotations validates the effectiveness of DGNet with the outdoor dataset. * **Interpretability of prediction results.** The interpretation of Bayesian posterior probabilities for trained networks based on the distribution function of the moVMF is an attractive advantage of DGNet. Figure 3 visualizes the posterior probabilities for some categories. Taking the “floor” as an example, according to the Bayesian theorem, those points with relatively high posterior probabilities have high probabilities of belonging to the floor, which is also consistent with the network predictions. * **Hyperparameter selection.** In Table 5, we search the parameter space for suitable $\kappa$, $t$, and $\beta$. We observe that (1) the segmentation performance shows an increasing and then decreasing trend as the concentration constant $\kappa$ increases. Our analysis suggests that too small $\kappa$ leads to a dispersion of features within the class, which can be easily confused with other classes. And too large $\kappa$ forces overconcentration of features within the class and overfits the network. (2) As the iteration number $t$ increases, the segmentation performance gradually rises and then stabilizes. We believe that the soft-moVMF algorithm gradually converges as $t$ increases, and increasing $t$ after convergence will no longer bring further gains to the network. (3) As the truncated threshold $\beta$ decreases, the segmentation performance shows a tendency to first increase and then decrease. The conventional cross-entropy loss function is the truncated cross-entropy loss function with $\beta=1$. When $\beta$ decreases, the overfitting on sparse annotations is alleviated, but when $\beta$ is too small, it weakens the supervised signal on sparse labeling leading to performance degradation. * **Ablation study for loss terms.** Table 6 demonstrates the validity of each loss term in DGNet. It is worth noting that $\mathcal{L}\_\text{vMF}$ with hard assignment form undermines the segmentation efficiency, on the contrary, $\mathcal{L}\_\text{vMF}$ with soft assignment form has a clear positive effect. In contrast to the soft assignment, the hard assignment does not take into account the inter-cluster similarity and is mismatched with the soft-moVMF algorithm. In addition, according to suggestions from another reviewer, we supplement this ablation experiment with more ablative forms. Please refer to the response to the Reviewer KHTz. * **Ablation study for soft-moVMF.** The experimental results in the first, second, third, and last rows of Table 7 show that it is optimal to optimize both $\alpha$ and $\mathbf{u}$ with the soft-moVMF algorithm. Optimizing $\alpha$ and $\mathbf{u}$ individually impairs the segmentation performance, for which the fixed parameters affect the accurate update of the optimizable parameters. The forth, fifth and last rows of Table 7 show that the soft-moVMF outperforms hard-moVMF and KNN-moVMF due to more accurate parameter updates.
Dear Reviewer bWaK: Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows: * Further explanation of the experimental results. * One weight balance experiment for tuning the weights of Eq. 9. * Detailed clarification of the article writing. We understand your time is valuable, but your opinion updates are vital to our work.
After reading the authors' responses and comments from other reviewers, I do not change my final score due to the following: 1. As shown in Table 1, compared with the Category Prototype (CP) model plus cosine similarity, the proposed method only shows an incremental improvement. And the method does not have any theoretical certificates. 2. Further, the comparison is insufficient because the authors do not directly compare their method with CP and other distributions under different weak label ratios. 3. Regarding the Qualitative comparison, this paper does not give insufficient results (only a few in the manuscript), and they also can not be found in the Appendix. 4. Some Symbols do not have definitions and are even wrong, e.g., there is no Kc in Equation (5).
Thanks for your response. We believe that your final comments are basically from other reviewers and have already been responded to by us. Even though you did not change your final score, we would still like to clarify these points of concern you listed as a matter of courtesy and respect: * About improvement and certificates. The category prototype model with cosine similarity and the final adopted moVMF are compared under our proposed distributional alignment framework. **There is a logical problem with taking two different distributional choices under the same framework to show that the framework is not valid.** In addition, we do not agree that there is no theoretical proof for the method you claim. We devote considerable space throughout the rebuttal phase to theorizing the superiority of moVMF, please refer to R3 to reviewer KHTz and R1 to reviewer ruvS. * About comparison. We believe that your concern stems from reviewer fVFW's question 1. However, this concern has been eliminated based on our experiments and the response of reviewer fVFW. Please refer to R1 to reviewer fVFW. * About qualitative comparison. We think that some representative samples of visualization comparisons are sufficient to demonstrate the validity of the methodology under the current page limitations. What additional conclusions would you like to draw from more qualitative results? * About typos: We guarantee that it is a transcription error without affecting the correctness of the method. **We would be regretful if our months of work were to be dismissed simply because of a typo that could be corrected immediately.** Moreover, the $\kappa_c$ is ignored in Eq. 5 because the $\kappa_c$ is a constant value for different catergories.
Thank you for the rebuttal. Most of my concerns are addressed. However, I still do not think that the explanation about the oughtness of using vMF is sufficient. The main contribution of this paper is introducing vMF distribution to tackle weakly supervised learning of point cloud semantic segmentation. Hence, from an academic perspective, this part should be more justified and discussed in more detail. Although I would like to keep my rating, it would be better if the authors could elaborate on this further.
The analysis of moVMF from the perspective of the Curse of Dimensionality
Dear Review KHTz: The discussion and analysis of moVMF are further enriched during the conversation with the Review ruvS. Specifically, we consider further analyzing the difference between the Euclidean norm (Gaussian mixture model) and cosine similarity (moVMF) from the perspective of the **Curse of Dimensionality**. * The data becomes extremely sparse in feature space as the feature dimension increases. Most feature vectors are far from each other, causing the Euclidean distance to become ineffective in distinguishing differences between feature vectors. Cosine similarity, on the other hand, is more effective in distinguishing differences between features by measuring the angle between the vectors. * Euclidean norm is very sensitive to scale. In a high-dimensional feature space, if the feature changes at different scales in different dimensions, it may result in the contribution of some dimensions to the Euclidean norm being magnified while the contribution of other dimensions is ignored. Cosine similarity is not affected by the length of the vectors, and the main measure is the angle between the features. Therefore, changes in the scale of different dimensions do not significantly affect the cosine similarity calculation.
Decision
Accept (poster)