Segment Anything in High Quality

Summary

The authors observe that the segmentation results of SAM model are unsatisfactory in several cases such as coarse mask boundaries and increect predictions. Thus, this paper presents HQ-SAM, whose objective is to improve the segmentation capability of SAM, with the introduction of the learnable high-quality output token and the global-local fusion module. For the training process, the SAM model is freezed while the newly added components are trained on the combination of 6 datasets, i.e., DIS (train set), ThinObject-5K (train set), FSS-1000, ECSSD, MSRA177 10K and DUT-OMRON. Experiments on four datasets (DIS, COIFT, HRSOD and ThinObject) show the effectiveness of the proposed method. Nevertheless, using DIS and ThinObject as testing sets may be unfair when comparing with SAM that may not see these two datsets during training.

Strengths

1. The motivation of improving SAM model is good. SAM is a powerful foundation model in computer vision, but its performance is not always satisfactory. This paper takes a very first step towards improving SAM. 2. Good presentation and easy to read.

Weaknesses

1. From Fig.2, it seems that there is an error correction module, but I didn't see any descriptions about this module in Section 3.2. 2. The training set is composed of DIS (train set), ThinObject-5K (train set), FSS-1000, ECSSD, MSRA177 10K and DUT-OMRON, while the valuations is conducted on DIS and ThinObject, which is unfair when comparing with SAM model. From Table 2, the improvement on DIS and ThinObject is much higher that on COIFT and HRSOD. This is because the corresponding training images are used in the training. Thus, evaluatoin on such datasets cannot verity the zero-shot generation capability of the proposed approach. 3. I encourage the authors to conduct experiments without using error correction during inference on COCO, LVIS, COIFT and HRSOD .

Questions

1. Why do authors select these 6 datasets to form HQSeg-44K benchmark? What's the motivation behind this kind of selection? Please also see weaknesses box for other questions. I hope the authors could address my concerns in rebuttal.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

As authors claimed, HQ-SAM shares the heavy ViT encoder of SAM, and thus cannot achieve a real-time speed in video processing.

Summary

This paper proposes a high quality version of segments anything. It points out the issues of existing segment anything models in Figure 1 and proposes the corresponding solution. It proposes a high-quality output token and a global-local fusion module. The results are good at various open benchmarks. The visualizations are rich and impressive.

Strengths

1. This paper improves the famous SAM model to a higher grade. This contribution is great and meaningful. 2. The paper itself is also in a high quality. The writing is clear. The idea is direct and easy to understand. 3. The results are impressive. Obvious improvement upon the SAM baseline on several benchmarks show the effectiveness of the proposed method. 4. The proposed dataset is also a good contribution. This makes this area can be followed by more researchers.

Weaknesses

1. The HQ-Output token seems the key point of the proposed method. It should have some distinguished difference to common tokens. Some feature-level visualization should be provided to show these differences. This will make the design explainable. 2. The comparisons in experimental are all among the proposed and SAM. Although this work is designed improve SAM, other related woks should also be compared. For example, the work [33] is also designed for high quality segmentation. Similar to some other related works, they should be compared in at least on dataset. [33] High quality segmentation for ultra high-resolution images, CVPR 2022 3. The Global-local Fusion model fuse the early and final layers. This design seems very related to the famous feature pyramid network (FPN) in object detection. Did the author try FPN before? FPN fuses more layers and might be a better choice. It would be helpful if the author could provide some ablation on this point.

Questions

Please see the weakness. Overall, I like this work. I will further improve the rate if the weaknesses are well-addressed, but might degrade the rate if not.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

As stated by the author, speed might be a small concern. My suggestion is that the FPS gap would be smaller if TensorRT is applied. This is not a hard job, as the original SAM already contains a TensorRT conversion tool.

Summary

This paper proposes HQ-SAM that produces segmentation masks high quality while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability, by only introducing minimal additional parameters. A learnable High-Quality Output Token is injected into SAM's mask decoder. Instead of only applying it on mask-decoder features, the proposed method first fuses them with early and final ViT features for improved mask details. This paper also proposes a dataset of 44K fine-grained masks from several sources. Extensive experiments and comparisons demonstrate the strong performance of the proposed HQ-SAM.

Strengths

- Improving SAM to produce more accurate masks is an important and also interesting topic, especially with acceptable finetuning cost. - The proposed designs make sense and are justified by ablation studies. - HQ-SAM is extensively evaluated on 9 diverse segmentation datasets across different downstream tasks, where 7 out of them are evaluated in a zero-shot transfer protocol.

Weaknesses

- As shown in Table 3, HQ-SAM significantly boosts the performance on Four HQ datasets even if it has consistent settings with SAM (the first two rows). It indicates that training/finetuning on HQSeg-44K is important for HQ-SAM. So what would happen if SAM is also trained on HQSeg-44K dataset. - What will the performance be if removing the fusion conv from the last row of Table 3.

Questions

See the weakness part.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This paper proposes HQ-SAM, an improved SAM with higher-quality segmentation performance. HQ-SAM freezes the entire SAM model and appends additional learnable modules concurrently to SAM. With the HQ-output token and global-local feature fusion, HQ-SAM achieves better segmentation results and only costs marginal extra resources.

Strengths

1. Additional training cost of HQ-SAM is marginal compared to SAM, but contributes to good performance boost. 2. The idea of adding a new HQ-token specialized for HQ mask is interesting and reasonable to me.

Weaknesses

1. Although the author claim the necessity of freezing the entire SAM model, it's better to verify this by ablation study, for example, unfreezing the entire SAM, or using PEFT methods (LoRA, adapters) with performance-parameter comparison. 2. I'm curious about how HQ-SAM performs in PerSAM (personalization segmentation by SAM) ? Personalize segment anything model with one shot (https://arxiv.org/pdf/2305.03048.pdf)

Questions

None

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Reply to Authors

Many thanks for the detailed reply and experiments. My concerns have all been addressed. I will keep my rate as "7 Accept".

Sincere Request for Further Feedback

Dear Reviewers, Firstly, we would like to express our thanks for the time and effort you’ve spent in reviewing HQ-SAM. As we approach the end of the discussion period, we eagerly anticipate your feedback on our recent response. We’ve endeavored to incorporate the suggested experiments and elaborate on our methods based on your insightful comments. Should there be any ambiguities or further questions, please know that we are more than willing to provide clarity or delve deeper into any topic. Your continued guidance is really appreciated. Warm regards, The Authors