SARDet-100K: Towards Open-Source Benchmark and ToolKit for Large-Scale SAR Object Detection

Summary

The authors address the limitations of existing datasets and the inaccessibility of source codes by creating a new benchmark dataset, SARDet-100K, which is a large-scale, multi-class dataset. Additionally, the paper proposes a Multi-Stage with Filter Augmentation pretraining framework designed to overcome the domain and model gaps between pretraining on RGB datasets and finetuning on SAR datasets. The MSFA method demonstrates effectiveness and generalizability across various models.

Strengths

1. The creation of the SARDet-100K dataset is a significant contribution. It offers the research community a large-scale, diverse dataset that was previously lacking. 2. As I know, in the field of SAR object detection, open-source code is indeed rare, which has significantly hindered the progress of development. It is very exciting to see a well-documented and professional open-source code base. Making the dataset and code publicly available enhances the reproducibility of the research and facilitates further innovation by other researchers. 3. Most previous work on improving SAR object detection performance focus on designing neural network modules. It is interesting that this research tackles the problem from the perspective of pretraining and domain transition. 4. The paper provides sufficient and detailed experiments/analysis that validate the effectiveness of the proposed MSFA method.

Weaknesses

1. You mentioned that optical remote sensing datasets (like DOTA) shares similar object shapes, scales, and categories in SAR datasets, therefore the downstream SAR datasets detection can benefits from the transferred knowledge. But why not discuss joint train the DOTA and SARDet-100K datasets together? In this way, the model can also learn joint representation from both DOTA and SARDet-100K therefore potentially improves the detection performance on SARDet-100K. 2. It is recommended to add a few more recent methods for comparison in Table 5.

Questions

See above.

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Limitations are properly addressed.

Summary

The authors establish a new benchmark SAR object detection dataset (SARDet-100K) and open-source SAR detection pretrain method (MSFA). This initiative significantly addresses the limitations posed by the scarcity of public SAR datasets and the inaccessibility of source codes, fostering further research and development in SAR object detection.

Strengths

Providing a larger standardized dataset for the data-scarce field of SAR target detection addresses a critical need and significantly contributes to its development. The authors unify and standardizes ten existing datasets to create SARDet-100K, the first COCO-level large-scale dataset for SAR multi-category object detection, which represents a substantial effort. MSFA model proposed in this paper is both effective and concise. It is refreshing that the MSFA model ingeniously applies traditional handcrafted features instead of design-heavy deep learning methods. Moreover, unlike previous approaches that use handcrafted features for feature refinement in deep learning, this work employs them for model pre-training and domain transformation, representing a novel and innovative approach.

Weaknesses

The paper does not clarify if the metrics reported on the SARDet-100K dataset are for the test set or the validation set. Additionally, please clarify the training setting: is the model checkpoint used for testing from the best validation or from the last epoch? As a benchmark dataset and method, actual runtimes and memory usage should be reported. These details are currently missing from the paper. There is a lack of clarification on the abbreviations used in Table 1 and S12.

Questions

Regarding the image slicing in your dataset standardization process, how do you handle objects that lie on the slicing border? Will the slicing split the objects apart?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are well discussed in Section 6 and A.8.

Summary

This study presents the a large-scale dataset designed for SAR object detection, alongside a Multi-Stage with Filter Augmentation pretraining framework. The authors address the challenges associated with the limited availability of public SAR datasets and the lack of accessible source codes. The proposed method is effective and can be generalized to most modern backbone and detection networks.

Strengths

The establishment of the SARDet-100K dataset provides a robust foundation for large-scale, multi-class SAR object detection research. Moreover, the open sourcing of the SAR detection codebase significantly enhances research reproducibility. The introduction of the MSFA pretraining framework is a novel approach that effectively bridges the domain and model gaps between RGB and SAR datasets by leveraging traditional handcrafted features. Overall, the paper is well-written, and the experimental validation and analysis are sound and solid.

Weaknesses

Your novelty lies in incorporating handcrafted features, the introduction and related work on these features are somewhat lacking. Given the current dominance of deep learning methods, many junior researchers may not be familiar with classic handcrafted feature descriptors. Therefore, it would be beneficial to provide a more comprehensive introduction and conceptual visualization for each of the mentioned handcrafted features to enhance the paper's clarity and accessibility. The supplementary materials would benefit from including a clear, step-by-step guide on training and testing the models. While the code is provided, the lack of direct scripts and instructions makes it difficult to replicate the results. The category distribution in Figure S6(b) reveals SARDet-100K is a significantly imbalanced dataset. This imbalance may lead to the long-tail problem, potentially hindering the performance of models on tail categories. Why do not you consider balancing the dataset?

Questions

In Figure 1, are the compared image pairs spatially aligned? It seems not. Would it be an issue?

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

None.

Summary

This work introduces SARDet-100K, a new large-scale, multi-category dataset for SAR object detection. It also proposes a novel Multi-Stage with Filter Augmentation pretraining framework to mitigate domain and model gaps encountered when transferring models pretrained on RGB datasets to SAR datasets. A new benchmark dataset and open-source method in SAR object detection is established.

Strengths

The paper's most significant contribution is the creation of the SARDet-100K dataset, which is the first large-scale, multi-category benchmark for SAR object detection. This dataset addresses the long-standing issue of limited and homogeneous SAR datasets, providing a rich resource that is likely to stimulate further research and development. The introduction of the MSFA pretraining framework is another strength, as it effectively addresses the domain and model gaps between RGB and SAR imagery, demonstrating robust performance across various deep learning models. Additionally, the paper is well-written, clearly outlining the motivation, methodology, and implications of the research, making it accessible to a broad audience in the field.

Weaknesses

The concept of the 'model gap' is not clearly defined in the current manuscript. Could you provide a more detailed explanation of what you mean by 'model gap' within the context of your study? Instead of a time-consuming pretraining stage, why not directly train the SAR detection model using only the proposed dataset with sufficiently long iterations since the dataset already contains a sufficient amount of images and instances.

Questions

None

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

4

Limitations

Limitations are well discussed.