ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentation

Summary

This paper presented a new mechanism to train efficient panoptic segmentation frameworks, which adds relaxation to mask predictions and class prediction for panoptic segmentation. In experiments, the authors demonstrated that the relaxation techniques can consistently improve panoptic segmentation frameworks.

Strengths

1. The results are impressive, especially, with R50 backbone, ReMax-M achieves 49.1 PQ and 51.9 FPS on COCO dataset. 2. This paper is well-written and the motivation is clear.

Weaknesses

1. The generalization of the proposed training mechanism should be demonstrated. 2. Some experimental results are unconvincing. Pleas see questions and limitations for details.

Questions

1. Why can the loss function with semantic masks be a relaxation that helps training? Training with semantic masks has been demonstrated to improve the instance segmentation in many frameworks such as CondInst. Therefore, what is the main contribution or the novelty of the proposed ReMask should be carefully discussed. 2. In Table 6, the results of MaskDINO and ReMaX should be tested with same GPU. When the GPU is V100, ReMax achieves 16.3 FPS (from Table 1), but MaskDINO achieves 16.8 FPS. Typo: 1. Line 71: "... like YOSO [26] and MaskConver [26] ... "

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

1. The contributions of this paper may be limited. For the segmentation framework, this paper directly uses the kMax-DeepLab. For the ReMask technique, it has been demonstrated in instance segmentation. For the ReClass technique, the effect of applying it or not is not shown in experiments. 2. The generalization of the proposed method should be demonstrated based on more frameworks such as Mask2Former, YOSO, and Panoptic DeepLab, not only kMax-Deeplab.

Summary

This paper presents a relaxation technique for training Efficient Panoptic Segmentation models called ReMaX. Based on the observation of much higher false positive penalisation in training panoptic segmentation models, it introduces two relaxation designs, ReMask and ReClass. Results are reported on COCO, ADE20K and Cityscapes datasets.

Strengths

1. The motivation of this paper is clear, and the finding is interesting. 2. The proposed method is reasonable and fits well with the motivation. 3. Using soft semantic segmentation prediction for relaxing the training of mask transformers makes sense to me. 4. Consistent improvements are obtained with the proposed method, and ablations are thorough. According to the experiments, the proposed method can work well with multiple existing approaches.

Weaknesses

1. The design seems coupled with mask transformers. It may not be generalised to all efficient panoptic segmentation models.

Questions

About the ReClass operation. What if the overlapped objects belong to the same category? For instance, two 'persons' in Fig. 3.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Error bars are not reported.

Summary

This paper shows that the existing sota panoptic segmentation methods have an unbalanced loss ( excessively large false-positive loss due to the use of the sigmoid function). The authors designed two relaxation mechanisms to relax the supervision at the mask and class levels, thereby improving training efficiency and improve accuracy.

Strengths

1. This paper reveals the problem of excessive false-positive loss caused by the use of the sigmoid function in current transformer-based panoptic segmentation models and proves that false-positive loss is also helpful for training and proper relaxation of constraints can improve training efficiency, which is instructive for the community. 2. Both approaches to relaxation in the paper (ReMask and ReClass) design are interesting and effective. 3. The proposed ReMax is excellent in training efficiency, inference efficiency, and accuracy.

Weaknesses

1. On the one hand, excessive false-postive loss affects training efficiency, but on the other hand, false-postive loss also benefits the final result. The author provides only an empirical solution, lacking more in-depth discussions. I wonder about the qualitative analysis of the impact of different scales of false-positive loss on the final result.

Questions

see weakness

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

No applicable

Summary

The manuscript presents two novel heuristics for training efficient panoptic models based on mask-level recognition and pixel-to-mask assignment. The first heuristics affects pixel-to-mask assignment and is referred to as ReMask. ReMask has been designed to balance the overwhelming contribution of false positive mask assignments to the mask-assignment loss by leveraging an independent semantic prediction head. In particular, the authors suppress the pixel assignment towards masks that get recognized into classes that are inconsistent with local semantic predictions. However, Table 7 suggests that most of the improvement does not stem from training relaxation and that the benefits may be caused by enhanced locality of the recognition process through Lsem. The second heuristics affects mask-level recognition and is referred to as ReClass. ReClass changes the classification targets of the predicted masks from one-hot winner-takes-all to mixtures of one-hot assignments of all incident ground-truth masks. It appears that the authors conjecture that ReClass contributes to the convergence speed by reducing the penalty of inaccurate masks during early training.

Strengths

S1. Panoptic segmentation is an important computer vision task with many applications. S2. State-of-the-art performance among approaches based on mid-range backbones (RN50, MNV3). S3. Ablations and validations suggest that the proposed heuristics contribute significant performance improvements. S4. The proposed heuristics can be removed during inference; this results in competitive inference speeds.

Weaknesses

W1. the manuscript requires non-linear reading effort: * lines 161-181 start to make sense only after reading the equations * equations for x_pan and x_sem are missing (they should start from shared features) * d_pan, d_sem, N_Q and N_C should be defined before use. W2. a bird's-eye view figure is missing (example: Fig.2 in [10]). W3. many small details: * l161: it appears that x_pan should be HWxd_pan? * l148: post-processing is unclear. Suggestions S1. it may be interesting to mention the following related work: * Fully Convolutional Networks for Panoptic Segmentation with Point-Based Supervision. TPAMI 2023. * Panoptic SwiftNet: Pyramidal Fusion for Real-Time Panoptic Segmentation. Remote Sensing. 2023. * Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation. CVPR 2022.

Questions

Q1. Can you disentangle the relative contribution of loss relaxation and local enhancement (cf Table 7)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

It would be interesting to discuss whether there is any benefit in conjunction with weaker (RN-18) and stronger (SWIN, ConvNext) backbones.

Summary

This paper introduces a novel mechanism called ReMaX to enhance the training of mask transformers for efficient panoptic segmentation, making it more accessible and practical. The authors observe that the high complexity of panoptic segmentation training objectives often results in imbalanced loss, leading to challenges in training end-to-end mask-transformer based architectures, particularly for efficient models. In response to this challenge, the authors propose ReMaX, which incorporates relaxation techniques for mask predictions and class predictions during training for panoptic segmentation. Through these simple relaxation strategies, the model consistently improves its performance without incurring any additional computational cost during inference. The effectiveness of ReMaX is demonstrated by integrating it into efficient backbones like MobileNetV3-Small. The proposed method achieves a new state-of-the-art record for efficient panoptic segmentation on benchmark datasets such as COCO, ADE20K, and Cityscapes. The results showcase the significance of ReMaX in improving the performance of mask transformers and its potential for advancing the field of efficient panoptic segmentation.

Strengths

By applying such simple techniques for relaxation to the state-of-the-art kMaX-DeepLab, ReMaX can train the network stably without any gradient-clipping operation under a learning rate that is over 10× greater than the baseline. Experimental results have show that the proposed method both boosts the training speed by 3×, and also leads to much better results for panoptic segmentation. ReMaX sets a new state-of-the-art record for efficient panoptic segmentation.

Weaknesses

(1) Do you have the experimental results on test set for COCO, CityScapes and ADE20K? (2) How do you balance Lpan and Lose? Are there any weights for these two losses? (3) In Table 2, what is the result of using softmax as activation and with grad-clip? (4) In Table 5, I'm wondering why removing the auxiliary semantic head will not lead to performance drop when using identity mapping. (5) In Table 7, why does PQ drop a lot when you use ground-truth semantic masks for m_sem? What is the result of using ground-truth semantic masks with the stop-gradient operation?

Questions

I'm positive about this paper. However I still have some concerns in the Weaknesses. I'll make the final decision after I see the response from the authors for those questions in Weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I cannot find any limitations or potential negative societal impact which the authors list.

Increase my rating to weak accept

Thanks for the author's response. It has addressed all of my concerns and I'll raise my rating to weak accept. Thank you.