All Points Matter: Entropy-Regularized Distribution Alignment for Weakly-supervised 3D Segmentation

Summary

This submission proposes a semi-supervized pseudo-label-based method for 3D point cloud segmentation method with sparse label annotations. Instead of thresholding on the confidence of pseudo labels, the authors propose to use all unlabelled points and encourage high-confidence pseudo labels by regularizing with their entropy. The authors propose a theoretical analysis of their approach and experiments where they show that their method improves the low-label regime for several backbone and several datasets.

Strengths

- the authors propose a theoretical analysis with some insight - The experimental results are convincing - The experiments are extensive: 4 datasets and 4 baselines

Weaknesses

- The entire ER+DA analysis leads to using a loss which is the cross-entropy between the pseudo labels and the prediction, which is already sensible and does not need to be seen as a special case of a more general setting that is never explored anyway. The fact that KL(p,q)+H(q)=CE(p,q) was known and did not need two full pages of motivation and equations (some parts are interesting such as the gradient limits, but would be better suited in the appendix). Equation (6) is completely logical and can be used directly. - On the other hand, the most interesting part of the paper is the pseudo-label generation hidden in Appendix B, which, as far as the reviewer knows, is novel. The reviewer thinks that the non-backpropagable momentum update of the class prototypes used for pseudo labels makes this method work. Without this, if p could be learned along q, then equation (6) would be meaningless as p=q is a trivial solution, and the pseudo labels would not help at all. The paper would need to be rewritten to highlight this hidden mechanism, brushed in the main paper in a single sentence, and at the method's core. - Equation (6) reduces to classic cross entropy when all points are labelled, and yet the proposed methods improve the backbones in the fully supervised setting? - The writing is subpar and lacks rigour. This leads to imprecise or even outright false statements at the core of the motivation. For example, the authors state that regularizing by the entropy decreases the "noise" in pseudo-labels when it actually encourages confident distributions and does not affect noise. There are many vague and uninformative sentences, some dealing with critical aspects of the paper. Some variables are also not introduced. The article remains overall understandable with the suppmat open in another tab.

Questions

Overall, the reviewer believes a very good idea is hidden in the paper; but, it takes a lot of effort to see through the subpar writing, imprecise statements, core ideas hidden in the appendix, and confusing notations. The reviewer also believes the authors focused on the wrong part of their contribution. As of now, the paper is not publishable. But if the authors put the work into improving their clarity and rigour, and focused on the important part of their contribution, this could become an impactful paper. The amount of effort might be too much for a rebuttal. Q1) Equation (6) reduces to classic cross entropy when all points are labelled, and yet the proposed methods improve the backbones in the fully supervised setting? S1) Swap the derivation of (6) and the pseudo-label generation between the appendix and the main paper. S2) Add detailed proofs of Table 1 in the appendix S3) Remove the notion that lower entropy = less noise

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

ok

Summary

This paper proposes two losses for point cloud semantic segmentation. The first one is Entropy Regularization (ER) loss, which makes the pseudo-labels have low entropy and thereby be confident (like one-hot vectors). The other one is Distribution Alignment (DA) loss, which is a KL divergence between the pseudo-labels and the predictions. Extensive experiments show that the combination of these two losses is not only beneficial in a weakly supervised setting but also in a fully supervised setting.

Strengths

1. The proposed method is extremely simple, but theoretically sounding and effective in practice. 2. Extensive experiments strongly support the benefit of the proposed method. 3. Overall presentation is great. Especially, section 3.2 is notably helpful to understand the working logic behind the proposed method.

Weaknesses

1. Could the authors provide a visualization of the (1) entropy map, (2) pseudo-label, and (3) KD loss map of the point cloud, and how they change as training proceeds? The figure would be extremely helpful to elucidate how the proposed method works. 2. Actually, it was quite surprising for me that the proposed method brings performance gain even in a fully-supervised setting. I think more discussion is required for this. Also, it would be great if the authors could provide an ablation study of Table 7(a) in a fully-supervised setting. I expect that DA would be much more beneficial compared to ER in this case, but not sure of course. 3. I understand why the authors refrained from comparing with some super-voxel-based approaches for a fair comparison. However, using a super-voxel partition is one of the widely used settings in 3D semantic segmentation. Hence, to clearly demonstrate the superiority of the proposed method, it would be much better if the authors could provide the performance of ERDA using a super-voxel partition. 4. Some related works are missing. Joint Learning of 2D-3D Weakly Supervised Semantic Segmentation (NeurIPS 2022) Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes (ECCV 2022) Weakly Supervised 3D Segmentation via Receptive-Driven Pseudo Label Consistency and Structural Consistency (AAAI 2023)

Questions

The results of the experiments on 2D semi-supervised semantic segmentation (SSSS) are interesting. I think the less significant improvement is mainly due to the gap between the weakly-supervised 3D semantic segmentation task and the SSSS task. Maybe the proposed ERDA can make benefit from 2D WSSS (using some points or scribbles).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations are properly mentioned in the paper (section 5).

Summary

This paper proposes a novel learning strategy to regularize the generated pseudo-labels and narrow the gaps between pseudo-labels and model predictions. It introduces an Entropy Regularization loss and a Distribution Alignment loss for weakly supervised learning in 3D segmentation tasks. The approach can better leverage unlabeled data points and achieves state-of-the-art performance under different settings.

Strengths

1. The proposed losses can work with other frameworks to consistently boost their performances. It can potentially achieve better performance with future stronger approaches. 2. The approach can improve many existing approaches by a significant margin.

Weaknesses

I think the motivation and design of the proposed losses needs further analysis. 1. Entropy Regularization loss: does it filter out some high-frequency predictions? How to balance the effects of filtering high-frequency components and removing noise? 2. Distribution Alignment loss: (a) Does it always improve the performance to encourage the consistency of prediction and pseudo labels? (b) Are the pseudo labels derived from network predictions? If so, they should be the similar thing. In what cases they tend to be similar and in what cases they are different? This part seems not very clearly presented.

Questions

See above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See above.

Summary

This paper considers the task of weakly supervised 3D scene semantic segmentation, where only a limited number of points in each training scene are given labels. Assuming a baseline system that operates within a pseudo-label paradigm, the paper proposes a new set of regularizing loss terms, that aim to (1) reduce pseudo-label entropy and (2) align the distribution of the pseudo-labels and network predictions. Under a default weighting strategy, these terms simplify into cross entropy from the pseudo-labels to the network predictions. Under a wide variety of experimental settings, the paper shows that incorporating this term leads to improved performance for 3d scene semantic segmentation, regardless of the level of supervision.

Strengths

This is a well-written, clear, and compelling paper on an important topic of interest to the community. While the introduced technique is not terribly complex, its benefits are well-justified, and the paper provides substantial analysis to support its inclusion: investigating how gradients from this loss term behave under different prediction settings, and why those gradients align with desirable properties. Further, the paper provides extensive ablation experiments that experimentally support this analysis, and show that all of its components lead to improved performance on the domains under investigation. The strongest point for the paper is in its thorough and overwhelmingly positive experimental results. For multiple datasets of 3d scenes (all standard), under multiple levels of supervision, adding this loss to an array of baseline models always improves performance, and outperforms previous state-of-the-art models on competitive benchmarks. Substantial improvements are observed when labels are severely limited, and even under fully supervised settings, including this loss term is helpful. From the presented evidence, it seems likely this term should be widely useful for this task and domain in future work, as it presents robustly strong performance under a myriad of framings and settings.

Weaknesses

My biggest outstanding question is to what extent this technique can offer benefits for other domains? In its formulation, there is nothing specific to 3D scene segmentation, so ostensibly it could be generally useful for other weakly supervised domains that employ pseudo-labels. Some evidence is provided that it can transfer to image segmentation, but it would also be interesting to consider domains like 3D shape segmentation. The initial results (on domains other than 3d scenes) provided by the paper are encouraging, but a more thorough analysis would of course strengthen the paper, and likely dramatically improve the reach/impact of the contribution. Relatedly, I would like to see more analysis / discussion about under what situations this term is helpful? Is it always beneficial to include such a term (no matter the domain / task). For instance, I could imagine that when the initial pseudo-labeling mechanism is highly inaccurate, this term might actually be harmful for learning. For 3D scene segmentation, my prior is that pseudo-labeling techniques are largely successful because strong locality cues in this domain can often be used to propagate labels to nearby unlabeled points, with a relative high degree of confidence; so the quality of initial pseudo-labels for 3D scene segmentation might be higher than would be expected for other domains of interest. It would be interesting to consider the effect that the “goodness” of the pseudo-labeling mechanism has on the final model performance, which could potentially be evaluated in a synthetically designed experiment that introduces “corruption” (at varying levels) into the distributions produced by the pseudo-labeling network. A deeper understanding of how the various components of the system interact with the added loss terms would be beneficial, and may give insight into what other domains and systems may benefit from this insights this paper provides. Minor: The formatting of table 2 can be improved. The read highlights are distracting, and largely unneeded as they overstate information. Consider replacing the red text coloring with italics, or better yet, marking only the columns that the baseline does not get improved with the added loss term.

Questions

Perhaps the most surprising result in the paper is that the method improves baselines, even under full label supervision. While I don’t doubt that trend is “real”, as the experimentation seems robust and well-designed, I was not quite satisfied by the explanation given to explain the result on lines 250-254. Is this explanation claiming that the “gt” labels have noise, so using ERD, which is “noise-aware”, can help regulate and remove the noise present in them? If so, this seems like a testable hypothesis (e.g. analyzing differences between pseudo-label predictions and gt label predictions). While I don’t think its required to have a compelte explanation for this phenomenon, the paper should either clarify the explanation here, or simply say that it is unknown why ERDA offers benefits in this paradigm, and to fully understand it would require further study.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

"Limited" limitations are given, see weaknesses section as to other potential limitations that should be explored.

Summary

The paper proposes a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions, which introduces an Entropy Regularization loss and a Distribution Alignment loss for weakly supervised learning in 3D segmentation tasks, resulting in an ERDA learning strategy.

Strengths

This paper solve an interesting problem, and the experimental results support the conclusions.

Weaknesses

1.There exists some confusions in Figure 1: a)For (a), there are generally two ways of generating pseudo-labels based on self-training weakly supervised methods. One way is to input both the sample and its augmented version into two different or shared-weight networks (with gradient updating); the other way is to input them into the student network and the teacher network updated via EMA (with only the student network being updated). I think the author's general description of sparse pseudo-labels in (a) is inadequate, which makes it difficult to establish a connection with (b) and understand the essential differences between them. b)I think the author's naming of (b) is inappropriate. It only optimizes the pseudo-labels p and predictions q simultaneously, without reflecting the "dense" aspect. 2.I appreciate the author's theoretical analysis of entropy regularization and distribution alignment, as well as the evaluation of different loss combinations via formula derivation. However, when I saw Line 160 and Table 7, I wondered if when lambda=1, the best result is achieved, and at this point, the ERDA loss simplifies to a single cross-entropy-based loss that optimizes both p and q. My question is, to my knowledge, I don't think only optimizing p for gradient update at the same time can bring such a high performance gain, and I speculate that the diversity of different perturbations as input enables the model to learn the geometric invariance of features, and the ERDA loss is just the icing on the cake. 3.In Line 207, the author uses a prototypical pseudo-label generation process due to its popularity and simplicity. However, in Table 7(a), the baseline with pseudo-labels already reaches 63.3%. I would like to see the difference between the results of prototypical pseudo-labels and plainest pseudo-labels. 4.I am a little confused about the author's approach under the setting of “fully”. Generally, the PL strategy is applied to unlabeled data, and then leverage prototypes, perturbation or contrastive learning to improve the model's robustness to the unlabeled data. However, in the fully experiment, the author applies ERDA to labeled data, so how can we talk about pseudo-labeling? There is also no gradient update for p. 5.Table 1 is difficult to understand, and I think this part is the core of the method. Therefore, the author needs to provide a detailed description of it.

Questions

See section weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

There are some issues not explained clearly, see Section Weakness.

Follow-up

Thank you for the clarification. Please make sure to include the positioning of your pseudo-label genetration module with respect ot he work you mention here. The current presentation of the paper seems somewhat oblique. Specifically, the authors only explore the scenario where lambda=1, which leads the ERDA loss to merely simplify to standard cross-entropy. Given this, I recommend one of two approaches: - Streamline this section to maintain clarity and coherence. - Explore varying lambda values. This would validate the in-depth theoretical discussion and provide readers with a more comprehensive understanding of the benefit of the ERDA loss compared to more standard approaches. Overall, the paper showcases promising ideas, and the results appear commendable. My reservations primarily pertain to its presentation. The emphasis, in my opinion, seems to be on less impactful aspects of an otherwise interesting paper. I've adjusted my rating accordingly.

Thanks for the rebuttal

Thanks for the author's rebuttal. Some of my concerns are addressed and I keep my score as "borderline accept".