Summary
The paper proposes a highly flexible and scalable benchmark for physical adversarial attacks against detection models and evaluates physical adversarial attacks under various physical dynamics by real-world simulators. It has a complete end-to-end pipeline, including data generation, detection, evaluation, and analysis. Further, it generates comprehensive evaluations and analyses to highlight the limitations of existing algorithms and provide considerable insights.
Strengths
1. The paper shows great efforts to organize such a large benchmark and provide detailed analysis.
2. Overall, the paper is well-written and almost clear to me.
3. Analysis tools in the benchmark are helpful to many researchers, and the user feedbacks reflect its ease of use.
4. The discussion about "where are we" and "where to go" is interesting and inspiring.
Weaknesses
1. The presentation can be improved. Although comprehensive experimental results are necessary, some crowd the layout and cause a terrible experience. For instance, Figure 2 provides extensive results while each sub-figure holds only a small subset of space. It makes readers try their best to broaden the figure and keep their eyes fixed on it. Selectively displaying results may be a better choice.
2. The comparison of other benchmarks in object detection should be available. Since the benchmark is proposed to illustrate the data, detectors, and adversarial attacks in object detection, the comparison of previous works or demonstration of the applied methods is important. In fact, the corresponding information is stated in Table 1 (target objects) and Table 2 (detectors). However, a clear comparison of the previous benchmarks is missing, which can be introduced in a list like the tables above.
3. The applied objects and physical simulator might be limited. Various detectors and adversarial attacks are utilized in the paper, while the physical simulator and the applied objects are limited. For example, more objects like border trees besides vehicles, persons, and traffic signs can be considered. Furthermore, the fixed simulator is likely to retain a special pattern in the generated data, which may lead to less reliable results on the benchmark.
4. The transferability of adversarial attacks should be considered in the analysis. In practice, the detectors are unknown to attackers, usually named as a "black-box" setting in adversarial attacks, and attackers are likely to create adversarial examples based on a substitute model by the transferability of adversarial examples. The analysis of the transferability of adversarial examples in object detection seems absent, though the evaluation in white-box settings is available (e.g., Figure 3 and Figure 4).
Questions
1. Could authors publicize the code and data when the paper is under review? Since the paper is proposed as a benchmark, I suggest the authors release their code and data when it is under review. In my opinion, their extensive efforts can be better validated if these materials are accessible. However, the authors claim the abstract by saying, "The code and datasets will be publicly available." Indeed, I respect the authors' choice and promise that their choice would not reduce my ranking.
2. The evaluation in real scenarios can be taken into account. The paper introduces a benchmark on physical attacks against object detection but only contains examples synthesized by simulators. The overhead cost of time and financial resources is somewhat troublesome in producing plenty of experiments. However, small experiments can be done as in previous works. The paper is named with "physical attacks" and no special qualifier, isn't it?
3. The paper ignores the adversarial defense methods or common corruption in objection detection. Actually, adversarial defense has rapidly developed in recent years, along with the development of adversarial attacks. Nowadays, many classifiers and detectors are protected with adversarial defenses against adversarial examples. Besides, common corruption like blur or compression is present in detector inputs under real-world scenarios. Both adversarial defense and common corruption play significant roles in evaluating physical attacks. However, corresponding results suggest that the evaluation may be insufficient in the real world.
4. The primary metric worth rethinking. The primary metric used in the benchmark is mAR (mean Average Recall), explained as the ratio of TP and GT (GT = TP + FN). Nevertheless, the mAP (mean Average Precision), the ratio of TP and all predictions (predictions = TP + FP) may be more representative in evaluating physical attacks. Physical attacks can generally mislead detectors to wrong results with varying TP and FP. The mAR can only examine the influence of physical attacks on TP, while mAP shows effectiveness in evaluating TP and FP. It seems that mAP can be a better choice, as it is also frequently present in object detection.