Unsupervised semantic segmentation is a challenging task that segments images into semantic groups without manual annotation. Prior works have primarily focused on leveraging prior knowledge of semantic consistency or priori concepts from self-supervised learning methods, which often overlook the coherence property of image segments. In this paper, we demonstrate that the smoothness prior, asserting that close features in a metric space share the same semantics, can significantly simplify segmentation by casting unsupervised semantic segmentation as an energy minimization problem. Under this paradigm, we propose a novel approach called SmooSeg that harnesses self-supervised learning methods to model the closeness relationships among observations as smoothness signals. To effectively discover coherent semantic segments, we introduce a novel smoothness loss that promotes piecewise smoothness within segments while preserving discontinuities across different segments. Additionally, to further enhance segmentation quality, we design an asymmetric teacher-student style predictor that generates smoothly updated pseudo labels, facilitating an optimal fit between observations and labeling outputs. Thanks to the rich supervision cues of the smoothness prior, our SmooSeg significantly outperforms STEGO in terms of pixel accuracy on three datasets: COCOStuff (+14.9%), Cityscapes (+13.0%), and Potsdam-3 (+5.7%).
Paper
References (44)
Scroll for more · 32 remaining
Similar papers
Peer review
Summary
The paper tackles unsupervised semantic segmentation which aims to group pixels into semantic clusters without manual annotation. This paper proposes the smoothness prior by enforcing that adjacent features in metric space share the same semantics. The approach relies on features from a self-supervised representation learning method (ie., DINO). Additionally, the architecture relies on an asymmetric teacher-student style predictor that updates pseudo labels, leading to smoother segmentation predictions. This approach outperforms STEGO (prior s-o-t-a) in terms of pixel accuracy on three datasets: COCOStuff, Cityscapes, and Potsdam with a ViT-S backbone.
Strengths
### Method: - Leverages advantages in self-supervised learning (SSL): The proposed method leverages the advancements in self-supervised representation learning in generating dense representations. Notably, it proposes to model the relationships between image patches by relying on high-level features from a pre-trained DINO model. - Kmeans is not required: The introduced smoothness prior can be used to directly optimize the generated segmentation map. The paper claims this strategy leads to more coherent and accurate segmentation results than prior work STEGO. In particular, the latter relies on minibatch Kmeans to obtain the final segmentation map. - Learnable prior: The approach does not rely on grouping priors as explored in prior works (see below for references). Instead, the approach implicitly uses a learnable grouping prior by relying on the self-supervised representation learning method (i.e., DINO). ### Experiments: - Dataset variety: The approach is applicable to different datasets. In particular, the experimental validation considers 3 different datasets: COCOstuff, Cityscapes, and Potsdam. - Quantitative results: the approach outperforms prior art (STEGO) on pixel accuracy when using DINO pre-trained weight on Citysapes and COCOstuff dataset. ### Misc. - Clarity: The method is well presented by relying on clear figures (e.g., Figure 2). The paper is also easy to follow (for someone who’s familiar with the literature at least).
Weaknesses
### 1. Originality Method - Difference with STEGO / SlotCon: - The paper can be seen as a combination of STEGO and SlotCon. The method largely follows the methodology of STEGO. The main SmooSeg loss is similar to the correlation loss in STEGO, which is also mentioned in the paper (see L158). Overall, I don’t think this is a major issue as I haven’t seen this combination being applied to the task of unsupervised semantic segmentation directly. - (Minor) I also don’t completely follow the reasoning that the proposed method is superior to Kmeans (L165). While this is indeed an advantage, it’s not clear why Kmeans would necessarily perform worse, as used in STEGO (both STEGO and SmooSeg rely on the features from DINO anyway). Kmeans also has to rely on a distance metric (e.g., cosine distance between features), resulting in similar features/patches being assigned to the same cluster. ### 2. Robustness - setting hyperparameters: There are many parameters that require supervision during training: - CRF weights: The weights of the CRF are determined using the annotated validation set. How can these be determined without annotations? It’s not clear from the supplementary if the weights are dataset-specific. (I assume that https://github.com/lucasb-eyer/pydensecrf was used.) - Number of classes: The number of classes is not always known a priori. In particular, it’s not clear how much the approach relies on this information. As a result, the introduced setup (section 3) is relatively artificial. In most practical applications, this information won’t be available. As a result, there are important experiments missing, where the sensitivity to this parameter is ablated (ideally comparing the mean and variance over multiple runs with other methods (e.g., STEGO, PiCIE). - Dataset-specific parameters: Overall, there are many training parameters that differ across datasets and require finetuning (i.e., loss parameters). Ideally, the same set of parameters is applicable to new/unseen images, especially for an ‘unsupervised’ method. Now, it’s hard to judge the robustness of the approach as it relies on finetuned parameters for each dataset. To my knowledge, this is not the case for prior works and baselines (IIC, PiCIE, DeepCluster). Also, TransFGU keeps most of its parameters constant. While this issue is somewhat tackled in the supplementary (see Section B), it’s still not clear if this strategy was actually used to set these parameters during training. ### 3. Scope of the experiments: - The experiments are limited to results on 27 classes (COCOStuff and Cityscapes). TransFGU also includes COCO-80 and COCOStuff-171. It would be interesting to see the performance for these setups as well. - A more practical setting is the semi-supervised setup, especially as the ‘unsupervised’ results are relatively poor and not immediately useful for practical applications. Does the method improve the representations by finetuning the learned representation compared to DINO? Finetuning the complete model or simply training a linear probe on top of the model are interesting experiments [f]. I would also expect that the learned representation can be efficiently fine-tuned with only a few samples [b]. ### 4. Quantitative results - different backbones: I noticed that only a ViT-S is used for COCO-stuff and Cityscapes, while a ViT-B is used for the Potsdam dataset. However, STEGO reports its final results with a ViT-B and outperforms the numbers in this paper (i.e., STEGO obtains 28mIoU on COCOstuff with a ViT-B). So, why is a ViT-B not used for COCO-stuff and Cityscapes as well, to make sure that the observations transfer to stronger backbones? This would furthermore make the main claim of the paper stronger. ### 5. Related work There are also a few related works missing. Interestingly, some of these rely on priors, such as superpixels, edge, or saliency estimators [d, e, f], which the paper under review does not require. More suggestions can be found below: [a] Wang et al, Dense Contrastive Learning for Self-Supervised Visual Pre-Training, CVPR. [b] Wang et al, FreeSOLO: Learning to Segment Objects without Annotations, CVPR. [c] Ziegler et al, Self-Supervised Learning of Object Parts for Semantic Segmentation CVPR. [d] Hwang et al, Segsort: Segmentation by discriminative sorting of segments CVPR. [e] VanGansbeke et. al, Unsupervised Semantic Segmentation by Contrasting object mask proposals, ICCV. [f] Zhang et al., Self-Supervised Visual Representation Learning from Hierarchical Grouping, NeurIPS.
Questions
My most important questions are listed above. I also have a few additional questions: - L265 claims that the proposed method outperforms STEGO at the boundaries. However, are the boundaries not primarily dependent on the CRF? How do these methods compare when we don’t use a CRF? It would also be useful to see a few examples without a CRF in the supplementary (optional). - How is the best model selected in the absence of a validation set? Can the loss function be used to select the best model during training? - The method is dependent on the pretrained DINO weights. We see that MoCov2 performs worse in Table 1. What happens if we use other weights (MoCov3, Pixpro, DenseCL etc.)? - Are the visualized predictions in the supplementary randomly selected?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
While some of the limitations are mentioned in the paper (see Section 4), I suggest including other limitations (e.g., known CRF weights, and known number of classes). I also couldn't find a paragraph on the societal impact.
Summary
The paper presents an unsupervised training method for semantic segmentation models. It trains the network on a loss with two terms: one encouraging smoothness in the segmentation labels, and a "data" term based on self-training. It assumes a neural net architecture with a) an extractor that produces per-pixel features $X$, b) a projector that produces lower-dimensional features, to be compared against "prototypes" and used in self-training, and c) the "predictor" that outputs the per-pixel segmentation labels from the prototype/feature comparisons. The backbone feature extractors are primarily based on DINO [11], though some experiments appear to use MoCo [24] in its place. This is evaluated in terms of its accuracy in semantic segmentation tasks including COCOStuff, Cityscapes, and Potsdam-3.
Strengths
i) Experiments consider variety of semantic segmentation tasks: street, general, and aerial image datasets. This is a good demonstration that the method can be applied generally, even if it is sensitive to some choice of hyperparameters. ii) Uses a backbone (DINO) also trained in an unsupervised (self-supervised) way. Uses DINO ViT-S/8 [11] or MoCo [20] as backbones in different experiments on the segmentation head & training. These were also trained without hand labelling, wtih a self- or unsupervised method; as opposed to using, for example, a standard ImageNet backbone from supervised training. This is a correct and principled way to make sure the proposed segmentation models are truly and completely unsupervised. iii) Precise presentation Uses both an algorithm, written in plain code, along with diagrams and the math to describe the method quite completely. Those parts of the method that are unclear from the definition of the loss can be deduced quite unambiguously from Algorithm 1, and vice versa.
Weaknesses
iv) Not a lot of experiments or analyses explaining *why* the method works. The experiments mostly show end results (mAP, qualitative segmentation quality). Hyperparameter studies are useful, though: providing good insight into the role each hyperparameter plays. And there is a minimal ablation study in Table 4. It still seems unclear what exactly the network is learning from. For example, to what extent is the smoothness a constraint on the extractor vs. the projector? It seems possible that $E_\mathrm{smooth}$ is also penalizing the cosine distances, as defined in Equation (2), and pushing feature vectors $X$ together, and not just using it as a weight for spatial smoothness in the labels $Y$, as in the intuitive explanation of Eq (1). I don't see `torch.no_grad` or any equivalent around the weight computations in Algorithm 1. Fig 5 visualizes feature embeddings on Potsdam-3, and this shows that the group similar classes together. Though, if the mapping between feature vectors $X$ and the labels $Y$, in the projector and predictor, is a "smooth" function (in the sense of Lipschitz smoothness) so that similar features get similar labels, might we already expect that the smoothness penalty is reduced, even if there is no meaningful spatial smoothness of the labels? The relative role/contribution of the within-image and between-image comparisons also seems to complicate the intuitive explanation of the method. There is a large accuracy drop if $E_\mathrm{smooth}$ only penalizes adjacent pixels within the image, and not across the images. The motivation for $E_\mathrm{smooth}$ that adjacent pixels are likely to have the same label, the "natural tendency towards piecewise coherence regarding semantics, texture, or color," does not apply to pixels within different images. The smoothness between pixels that are adjacent in a metric space is a property of the construction of that metric space: not a natural property of images, so this part's role in the function of the smoothness perhaps needs more explanation and experiments. Why do we expect with this unsupervised loss that the metric space will be semantically meaningful? v) Novelty is arguably more limited than is claimed. The use, in unsupervised training of semantic segmentation models, of spatial smoothness of the labels does seem to be explored. Though many of the particulars and how it's used within the overall framework is new as of this submission. Smoothness priors themselves are, of course, quite common in semantic segmentation. Spectral methods also expected to be a form of smoothness prior, but using squared differences in place of the $\delta$ in Eq (1) of the submission. This kind of prior is used in unsupervised training of semantic segmentation models in: [a] Xia & Kulis "W-Net: A Deep Model for Fully Unsupervised Image Segmentation" [b] Melas-Kyriazi et. al. "Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization" W-Net also incorporates a fully-connected CRF, in section 4.1, though this is only for preprocessing and not for unsupervised training. I however don't see prior work that uses a fully-connected CRF within the training objective in the way that this submission does. The backbone is also updated relative to previous work: this submission is perhaps in the genre of papers that applies a given vision idea/concept to ViTs instead of CNNs. vi) There may also be some gaps in the citation of prior work similar to the submission's teacher/student/self-training approach described in section 3.2. Citations on self-supervised learning in the "Related Work" in section 2 focus on contrastive learning approaches. More similar approaches might include: [c] Scudder "Probability of error of some adaptive pattern-recognition machines" [d] Xi et. al. "Self-training with Noisy Student improves ImageNet classification" Based on my read, the submission does *not* modify the inputs or consider different views between teacher and student, as in [19, 20, 21, etc.]. (If my read is incorrect, what exactly is the "pretext task" that the authors use?) Though [c.d] and similar papers differ enough from the submission's training scheme, including the use of a projector module to get the prototypes that are compared between teacher & student, that this part is not relevant to judging novelty.
Questions
Why the focus on semantic segmentation? If the labels aren't used, then I'd assume the method should equally apply to instance, panoptic, or salient-foreground segmentation. Especially given it does seem to generalize across different kinds of semantic segmentation. Is it because the prototypes $P$ are expected to somehow relate to semantic classes of "stuff?" The stated goal of the smoothness prior is that "the segmentation model is encouraged to assign similar labels to adjacent patches, thereby promoting spatial coherence within objects." What determines which pixels are "adjacent" when constructing E_smooth? I.e. is it 4-connected, 8-connected, or fully connected? The pairwise weights don't seem to include spatial distance. Toward the end of Section 3.1 it's stated that the authors "also apply the smoothness prior across images." Which pairs of pixels across pixels are included as terms in the sum in Eq (1)? Minor comments: * Typos in Algorithm 1 comments: "updata" and "prototyeps" * Citation to [20] is repeated on line 94
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
2 fair
Limitations
The sensitivity of the results to hyperparameter choices is mentioned as a limitation. This does seem likely to be the major drawback, with bad results like mode collapse being a concern. In Appendix B of the supplement, it is described in qualitative terms that one can monitor the distribution of the differences in soft label assignments (the "smoothness degree") to adjust the parameters. They don't give a fully reproducible method or algorithm for determining b1 and b2 to achieve the desired $\delta$ distribution, but it does seem clear what one would be looking for to tune this manually.
Summary
This work addresses the problem of unsupervised semantic segmentation. In contrast to STEGO (baseline), the approach learns semantic embeddings directly in a student-teacher regime without the need for the K-means. The objective is entirely unsupervised and is reminiscent of a CRF energy formulation with data and a smoothness term. The pseudo labels from the teacher embeddings provide the signal for the data term. The work achieves improved segmentation accuracy on standard benchmarks over the state of the art and offers an interesting analysis. Overall, the work is compelling in quality and contribution.
Strengths
- The application of the smoothness constraint is natural and simple. - The learning problem is well-designed and executed. The text is generally well-written (albeit not without minor typos). - The experiments have sufficient scope and provide valuable insights. The empirical results are compelling.
Weaknesses
There are technical similarities to STEGO, such as in the loss formulation in Eq. (4). There are differences, though, as discussed in 158-174. However, it does take away a bit from the novelty. Empirically, the approach appears at its best with DINO features using ViT architecture; the improvement over prior art may not translate to other SSL representations and model architectures (MoCo, ResNet, see Tab. 1). Nevertheless, the approach remains competitive. The method introduces a number of hyperparameters, which can be challenging to fine-tune in an unsupervised setup. I like that this is acknowledged in the work, however, and Appendix B provides some interesting analysis. Typos: e.g. l. 178, 307
Questions
- l. 140-141: Why is optimisation only stable with such normalisation? - Eq. 5: Why is there a need to stop the gradient flow in this fashion? - l. 212: I would be curious to understand how the predictions and the ground truth are aligned in a bit more detail. - How were the other parameters (e.g. the momentum, the temperature) chosen/fine-tuned? - I do not quite follow the sentence in l. 172: “which represents discontinuities between image patches that should be preserved.”
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The limitations are discussed in the main text. The supp. material provides further analysis.
Summary
The paper introduces a new approach called SmooSeg for unsupervised semantic segmentation, which aims to segment images into semantic groups without manual annotation. SmooSeg is based on the idea of smoothness: adjacent features in a metric space should share the same semantics. Specifically, it formulates unsupervised semantic segmentation as an energy minimization problem (Eq. 1,4). Training is performed using teacher-student style self-training.
Strengths
### The paper is well-structured. * The introduction is well-written and motivates the problem well. The methods section breaks down the contribution into provides helpful preliminaries (S3.1) in a good amount of detail. It also describes the contribution (S3.2-3.4) precisely without unnecessary complications. ### The visualizations provided are nicely done * The t-SNE plot in Figure 5 is quite illustrative, and the qualitative examples in Figure 3 are also helpful. ### Algorithm 1 does a good job of communicating the proposed method. * It is very helpful to be able to refer to the PyTorch pseudocode alongside the equations in the paper. ### The “Discussion with CRF and STEGO” section is very helpful * My natural first question upon reading the introduction and the beginning of the methods section was about the relationship between SmooSeg and CRFs. This section answered some of my questions, as it gave an analysis of their relationship.
Weaknesses
### It would help to have further analysis of why the method works. * As discussed in the paper’s introduction, the idea of smoothness is image segmentation has a long history, having been explored since well before deep learning. This is especially true of energy minimization approaches, for which many different types of smoothness losses have been proposed over the years. Empirically, it seems that the performance of the smoothness loss proposed in this paper is good. However, I do not really understand _why_ it should be better than any other approach to enforcing image smoothness, such as the approach applied in CRFs. What would make this paper really useful to the community would be if it tried to understand why this particular formulation works well, so that the community can learn some generalizable lessons/insights (which could then be applied to other problem domains as well). * Is it something particular about the combination of your smoothness term and the student-teacher approach that works well, or is it each of them individually? * What would happen if you took STEGO (exactly as it is) and added your smoothness loss to their method? Would it be good, or does your smoothness loss work particularly well when combined with your student-teacher method? ### Why do you also need to apply a CRF? * The supplement states: “We also use a CRF as the post-processing to refine the predicted semantic maps.” This confuses me. If you already have a smoothness loss (which is your contribution), why do you need a CRF? * How does your performance compare without a CRF? _Meta note on the score (because there is nowhere else to put this in the review)_: I am between 4 and 5. I gave a preliminary score of 4, but I am certainly willing to raise my score when the above questions/points are answered.
Questions
### Removing E_{data} in Table 4 * For Table 4, I’m surprised that the method still works when the data term is removed in Table 4. In that experiment, are there any losses apart from the smoothness term? Without the data term, should there not be degenerate solutions?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
2 fair
Limitations
Limitations are discussed adequately in the paper. One main limitation (how to set hyperparameters) is mentioned clearly at the end of the paper and addressed in the Appendix.
Summary
The paper tackles the difficult task of unsupervised semantic segmentation at the level of the scene (dense prediction). The authors exploit the piecewise coherence regarding the semantics, texture, or color that similar objects naturally have (termed as smoothness prior). The problem formulation raises some challenges the author solves by using a pre-trained (frozen) feature extractor to model the closeness relationships among observations and introduce a novel pairwise smoothness loss, and a teacher-student style predictor. Results are showcased on popular benchmarks: COCOSttuff, Cityscapes, and Potsdam, on which the authors report state-of-the-art performance.
Strengths
1. Originality - the problem formulation (energy minimization objective function) and the addition of the smoothness prior counts as novel work. 2. Quality - the paper is decently structured, method and experimental analysis are sound and convincing. 3. Clarity - some aspects of the paper could be improved. Some details are left out, making it be challenging to reproduce the results based on the information provided in the paper. 4. Significance - the topic is indeed relevant to the research community.
Weaknesses
* L112 - The architecture description paragraph is tough to follow and the dedicated figure (Figure 1 in the main submission) is overcomplicated. Also, the use of the term "prototypes" for the teacher-student paradigm is confusing. Please consider rephrasing this part. * L62 vs. L312 - minor contradiction. * L307 - Type "We" - no capital letter * The main submission has an appendix document but there is no reference in the main submission regarding the contents of the supplementary material. * Important details are in the supplementary material and not mentioned in the current submission - such as the use of CRF for further refining the final segmentation maps. * In the experiments section there is no mention of how the authors produced the segmentation maps from SSL feature extractors such as ResNet50, MoCoV2, DINO (Table 1), DINO (Table 2), or DINO, DINOV2 (Table 3).
Questions
Please see weaknesses above.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have not properly addressed the limitations of their proposed method. I highly encourage them to do so in a dedicated section in the main submission. Ideas for future work are a plus.
Response to authors after the rebuttal
I thank the authors for providing the rebuttal. There are several points I want to emphasize after reading the rebuttal and the other reviews: 1. The motivation why the approach outperforms STEGO could be further corroborated by also including "linear probing" results. Why is this information not provided in the paper? STEGO also includes this. 2. One of the main weaknesses is that the approach requires dataset-specific hyperparameters. This limits the scalability of the approach and furthermore contradicts the claim that the approach is unsupervised. While STEGO has shown better results by changing the hyperparameters across datasets, it’s reasonable to argue that this is far from realistic. Additional experiments are necessary to quantify the robustness of the approach as the adopted setup can be considered artificial: the exact amount of clusters, supervised CRF weights, and ideal loss weights are currently being used. As a result, including linear probing and over-clustering experiments would make the paper certainly stronger. 3. In addition, semi-supervised results (e.g., with a linear probe or a shallow head) would also make the approach more useful in practice as the current mIoU scores are relatively low (e.g., 18 mIoU on Cityscapes). 4. I also still don’t understand why different architectures are being used. As the approach heavily relies on STEGO, it makes more sense to include results with a ViT-B for Cityscapes. This is especially important since STEGO reports higher numbers than currently presented in the paper. This would confirm the claims in the paper.
Thank you for the prompt response. **1) Regarding the linear probing:** We would like to clarify that linear probing is a **supervised** approach for assessing the quality of the representations generated by self-supervised representation learning methods. Therefore, we think that linear probing is not suitable for evaluating the performance of SmooSeg as labels are not accessible in unsupervised semantic segmentation. We would like to further elaborate on the rationale as below. Existing unsupervised semantic segmentation methods can be roughly categorized into two types: - **One stage:** IIC[3], PICIE[4], HSG[10], TransFGU[6], and the proposed SmooSeg, which directly learn semantic labels, with losses typically defined on labels. - **Two stages:** STEGO[8], representation learning + Kmeans for unsupervised semantic segmentation. In the representation learning stage, losses are defined on feature maps. For one-stage methods, as they directly output the predicted semantic maps instead of intermediate features, their performance is evaluated by Acc and mIoU, which are task-specific metrics for unsupervised semantic segmentation. Linear probing is thus not directly applicable and not reported in one-stage methods [3, 4, 6, 10], as the quality of intermediate features is not the objective of these studies. For the two-stage method, STEGO, as it outputs good representations to be used by K-means for images, it is thus reasonable to assess the quality of the learnt representations by linear probing. Therefore, it is not sensible and unfair to compare the results of linear probing between a one-stage and a two-stage method due to their distinct objectives. In addition, linear probing may not be a reliable assessment even for two-stage unsupervised segmentation methods because of its sensitivity to feature dimensionality. To illustrate this point, we cite the **linear probing results** from Table 2, page 5, in [a]: | Method | COCOStuff | Cityscapes | | -------- | -------- | -------- | | | Acc / mIoU | Acc / mIoU | | DINO (ViT-B)| 75.8 / **44.4** | **91.3** / **34.9** | | STEGO | **76.1** / 41.0 | 89.6 / 28.0 | It is shown that the linear probing results of DINO (dimensionality of features: 768) are better than STEGO (dimensionality of embedding: 90 or 100). These linear probing results contrast with the superiority of STEGO over DINO for the unsupervised semantic segmentation task. *[a] Uncovering the Inner Workings of STEGO for Safe Unsupervised Semantic Segmentation. CVPR, 2023.* **2) Semi-supervised setup.** Our method aims to address the problem of unsupervised semantic segmentation rather than semi-supervised semantic segmentation [b,c]. *[b] Semi-supervised semantic segmentation with prototype-based consistency regularization, NeurIPS 2022.* *[c] Semi-supervised semantic segmentation via gentle teaching assistant, NeurIPS 2022.* **3) Problem setup and hyperparameter issue.** We strictly follow the established unsupervised semantic segmentation setup as in [3,4,6,8], where **the number of clusters is predefined**. For example, this number is used as the number of cluster centroids in PiCIE [4], the number of class embeddings in TransFGU [6], and the "K" in Kmeans in STEGO [8]. In addition, it's important to highlight that **CRF with default settings is employed only during the testing phase**. Therefore, its parameters are not relevant to the hyperparameters of our method. Finally, most existing SOTA methods, including STEGO and TransFGU, also contain dataset-specific hyperparameters. To further demonstrate the effect of the number of classes, we tune the number of prototypes in SmooSeg and show the results below. The best result of SmooSeg is obtained when the number of prototypes equals the ground-truth number of classes. Interestingly, even with more prototypes, SmooSeg consistently outperforms STEGO (Acc: 48.3, mIoU: 24.5). | Number of prototypes | 27 | 30 | 33 | 37 | |:--------------------:|:-------------------:|:-----------:|:-----------:|:-----------:| | | Acc / mIoU | Acc / mIoU | Acc / mIoU | Acc / mIoU | | SmooSeg | **63.2** / **26.7** | 58.5 / 25.6 | 61.3 / 25.3 | 59.1 / 24.2 | **4) Additional results with DINO ViT-B/8 on the Cityscapes dataset.** The results of SmooSeg with DINO ViT-B/8 as the backbone are provided below. It's evident that SmooSeg is also superior to STEGO under this backbone. | Method | Acc | mIoU | | -------- | -------- | -------- | | STEGO (ViT-B) | 73.2 | 21.0 | | SmooSeg (ViT-B) | **84.5** | **21.5** |
Response
Thank you for the discussion. I appreciate the points. - Linear probing: I partially disagree/agree. Indeed, linear probing is an excellent tool to measure the quality of the obtained representations. However, it is still extremely unlikely that the exact number of classes is known a priori. Furthermore, this single linear layer can quickly adapt when only a few labels are available. The adopted Hungarian matching algorithm is somewhat 'flawed'. In particular, this 'perfect' matching criterion (during evaluation) also assigns probability mass to classes for which the model might have predicted close to 0%. As a result, I still encourage the authors to add linear probing results to the paper as done in many prior works, such as in STEGO, HSG, and most of the related work mentioned in my initial review [C, E, F]. - Hyperparameters: The issue is not that K is predefined, but that we don't have any information on the robustness of these parameters when compared to STEGO or other prior approaches. The presented approach seems to benefit from knowing the exact classes (see table above) and furthermore relies on dataset-specific parameters. While 1 or 2 works seem to adopt this setting, most prior works don't. As a result, adding over-clustering results to the paper and also keeping the hyperparameters fixed for similar datasets/domains, is important for an 'unsupervised' method. For all we know, STEGO might be more robust to these parameters which would benefit scaling. - Semi-supervised results: I understand that semi-supervised learning is not the aim of the paper. My point is that the current mIOU scores are low. Adding semi-supervised results as in [B] would make the submission stronger. All in all, I'm questioning the usefulness of the approach as it requires the classes to be known a priori, as it relies on dataset-specific hyperparameters, and as it performs worse in the presence of (a few) labels than the baseline (DINO). Based on these points I'm still leaning towards a borderline score.
Response to Reviwer yftt
Thank you for the response. **1) Linear probing.** - To the best of our knowledge, DINO [11], STEGO [8] and the works [C, E, F] are self-supervised representation learning methods (SSL). When applied to the task of unsupervised semantic segmentation (USS), they belong to the category of two-stage methods. As elaborated in our previous discussion, it is reasonable for them to assess **their representation learning quality** by linear probing. In contrast, one-stage methods for USS such as IIC [3], PiCIE [4], TransFGU [6], and HSG [10] aim to directly predict the final semantic maps with losses imposed to the labels rather than to features. That is why these methods **did not report the quality of their intermediate features by linear probing as obtaining good feature representations is NOT their goal**. Our SmooSeg is a one-stage method for USS and not an SSL method; thus we don’t see a need to discard our predictor dedicated for USS and assess the quality of intermediate features via linear probing as what SSL methods do. **2) Hyperparameters.** - **The number of classes.** Setting the number of classes as a hyperparameter is typical for unsupervised learning, which can be traced all the way back to kmeans, a classic unsupervised learning method. **We certainly cannot conclude that kmeans is not an unsupervised method due to the use of k**. The same applies to the solutions to USS: as long as the proposed methods do not utilize any annotation (label information), they are unsupervised methods. In fact, when applying STEGO to USS, it also requires to input a value of k to kmeans and use Hungarian matching for class alignment. If one applies STEGO to USS by linear probing, it is not called “unsupervised” due to the use of labels. To clarify your concerns, we adjust the parameter "K" of Kmeans in the STEGO and present the outcomes below. We use the checkpoint of STEGO (ViT-B) on the COCOStuff dataset provided by the authors. It can be seen that the performance of STEGO also diminishes under the overclustering setting. | K | 27 | 30 | 33 | 37 | |:--------------------:|:-------------------:|:-----------:|:-----------:|:-----------:| | | Acc / mIoU | Acc / mIoU | Acc / mIoU | Acc / mIoU | | STEGO (ViT-B) | **56.9** / **28.2** | 53.5 / 23.3 | 52.7 / 23.7 | 51.9 / 20.1 | - **Dataset-specific parameters.** We do not agree that STEGO is more robust to dataset-specific parameters. Not including k needed in kmeans, **STEGO has 6 dataset-specific parameters**. It is shown in Table 6 of STEGO’s Appendix A.10, these parameters can vary significantly across different datasets (e.g., $\lambda_{rand}$ is 0.91 on Cityscapes and 0.15 on CocoStuff). In fact, we have spent a lot time tuning these hyperparameters of STEGO when obtaining its results on ViT-S as its performance is sensitive to them. On the contrary, **our SmooSeg has 2 dataset-specific parameters**, which are observed to be more stable across different datasets (Table 5 in our supplementary). We have strived to find a practical way to set their values in our Appendix B. **3) Semi-supervised results.** - We acknowledge that there is a certain gap between unsupervised and semi-supervised semantic segmentation. However, it’s important to underscore that our SmooSeg demonstrates clear superiority over existing USS methods, thus effectively establishing its efficacy. In addition, it’s essential to recognize that USS methods do not solely pursue an absolute segmentation outcome; rather, their primary objective lies in showcasing higher performance within the unsupervised setting without any annotation.
Thank you, I'm happy with the response. This work leaves an overall positive impression and I intend to keep my score. I also read the other reviews. I agree with some points, but most of them seem to be addressed well and can be included in the camera-ready. In other cases, I tend to agree with the authors that they do not seem essential (e.g. linear probing) or even feasible (e.g. adding the smoothness loss to STEGO).
Thanks for your constructive feedback
We would like to thank the reviewer for the constructive feedback which helps shape our revision. Please let us know if there are any additional concerns or suggestions that could further enhance the quality of this work.
Response
Thank you for your response and clarifications. After reading your response and the reviews of the other authors, I am satisfied that the proposed method is sufficiently different from STEGO. I agree there is no straightforward way to add the smoothness loss to STEGO; apologies for the misunderstanding. I also now get that $E_{data}$ is essentially a self-training loss; it is good to see that the method works even without $E_{data}$. I will be updating my score from 4 to 5.
Thanks for your constructive feedback
We are glad that our response addressed all your concerns. We also greatly appreciate that you awarded us a higher score. Please let us know if there are any additional concerns or suggestions that could further enhance the quality of this work.
> We apologize for not explicitly mentioning the use of torch.no_grad in Algorithm 1. We will include this missing detail in the revision. Maybe not needed! Just mentioned it as one thing I had looked for, as a hint as to what was being learned. Definitely okay if the authors can clarify this same point some other way.
> We clarify that our “adjacent” patches are defined in the high-level feature space generated by the frozen pre-trained model, rather than in the spatial coordinate space of images This is a good clarification, thanks! Definitely a few passages I'd suggest editing in the camera-ready keep this clear. For instance, the intro seems to be setting up to motivate smoothness in the image space, or conflate the two: > However, despite their effectiveness, these methods often overlook the property of spatial coherence of image segments > Observations close to each other, either in the form of neighboring pixels or adjacent features in a metric space, are expected to share similar semantic labels or, in section 3,1: > In other words, the segmentation model is encouraged to assign similar labels to adjacent patches, thereby promoting spatial coherence within objects.
Thanks for your valuable suggestions
Thanks a lot for your valuable suggestions that help improve our paper. We will certainly address this confusion in our revision. Please let us know if there are any additional concerns or suggestions that could further enhance the quality of this work.
Decision
Accept (poster)