Summary
This work primarily focuses on semantic segmentation. Specifically, a semantic and spatial adaptive (SSA) classifier is proposed to address the "feature deviation in the semantic domain and information loss in the spatial domain" issues. The proposed classifier mainly consists of
- Semantic Prototype Adaptation,
- Spatial Prototype Adaptation,
- Online Multi-Domain Distillation Learning.
The authors claim that "experimental results on three publicly available benchmarks show that the proposed SSA significantly improves the segmentation performance of the baseline models with only a minimal increase in computational cost."
Strengths
The strengths of this paper can be summarized as follows,
- Overall, the author clearly introduces the research motivation and the proposed method.
- The author validates the effectiveness of the proposed algorithm across different datasets and segmentation frameworks.
Weaknesses
The weaknesses of the paper are as follows,
- The authors claim in the abstract that the code for the proposed algorithm is included in the supplementary materials, but I did not find any related code there.
- Some expressions in the paper feel redundant, e.g., "directly classify masks by directly learning query vectors".
- In the introduction section, the authors argue that previous mask-level classification models have cumbersome decoders. Therefore, the authors should conduct a comprehensive comparison (params, fps, flops, miou, etc) between their method and these previous methods in the experiments section.
- Tables 1 to 3 only provide the FLOPs metric but do not include FPS and parameters, which are also crucial for deploying the model in resource-constrained environments.
- Can the authors provide any experiments or references to support the statement "Due to complex backgrounds and varying object distributions, pixel features in the test images tend to have a large intra-class variance with pixel features in the training set"? I think that a single image in Figure 1 can not prove this point.
- To be honest, Figure 1 is very confusing to me. For example, what do the gray dots represent? (pixel features belonging to other categories?) Can you provide the model's prediction results for this image? If your visualization is accurate, the prediction should include many pixels of other categories being classified as “door”. In other words, I believe there is a high likelihood that your visualization is unreliable, namely, in the t-SNE visualization results, being close to a semantic prototype does not necessarily mean that the model classifies it as the corresponding semantic category in the feature space.
- If the proposed SSA strategy truly facilitates “the offset of the fixed prototype towards the center of the semantic domain and the center of the spatial domain of the test image,” why don't the authors directly visualize this result using t-SNE like Fig. 1(b) instead of presenting a less credible schematic diagram?
- I disagree with the content in lines 36 to 40 of the paper. Modeling the relationship between prototypes and pixel features in the spatial domain is incorrect. This means that when classifying pixels, models need to consider which types of objects generally appear at that pixel location. This is obviously an overfitting to the current dataset and has no practical application value.
- The authors argue that "pixel features in the test images tend to have a large intra-class variance with pixel features in the training set" and their proposed method can address this issue. If this is indeed the case, the author should apply their method to cross-domain segmentation tasks to better demonstrate its effectiveness.
- The content in the Related Works section is too brief.
- In line 126, the authors mention that "Based on the above analysis", where is the analysis?
- Can the authors provide any experiments or references to support the statement between line 150-151?
- The content stated in lines 155 to 157 does not correspond with the formula.
- Section 3.2.1 essentially just integrates features similar to class prototypes and does not achieve the author's claimed “facilitating the offset of the fixed prototype towards the center of the semantic domain.”
- “utilizing the rich spatial information of the image” should be the responsibility of context modules (such as aspp and ppm) in the decoder head. The premise of section 3.2.2 is fundamentally flawed.
- Would the model's performance improve if only “randomly initialized position basis” is used? Can the authors provide related ablation experiments?
- If I understand correctly, section 3.2.2 essentially assumes that objects of the same class should all be in a specific region of the image. However, this assumption is unreasonable because, in semantic segmentation, different instances of the same class can be far apart.
- To be honest, I believe the methods in sections 3.2.1 and 3.2.2 are not highly related to the problem analyzed by the authors or the expected solutions to that problem.
- Would the model's performance improve if section 3.2.3 was replaced with self-distillation?
- Due to the existence of $\mathcal{M}_c$, the problem mentioned by the authors in the introduction section still persists in the proposed algorithm. In my opinion, the performance improvement is mainly due to the addition of more parameters and the introduction of more loss functions. While these changes have brought about performance gains, they do not align with the authors' motivation.
Due to these weaknesses, I believe this paper should be rejected.
Limitations
The author does not discuss the limitations of their algorithm in the paper. I have provided several suggestions for improvement in the weaknesses box, which I hope will help the authors enhance the quality of this work.