Summary
The paper introduces a novel attack framework called Content-based Unrestricted Adversarial Attack, which aims to generate diverse and natural adversarial examples with high transferability. The authors argue that existing methods, such as lp norm-based attacks, have limitations in terms of perceptual similarity, naturalness, and robustness. To address these issues, they propose mapping images onto a low-dimensional manifold represented by a generative model trained on natural images. This manifold ensures both photorealism and content diversity. By optimizing the adversarial objective on this latent space, they generate unrestricted adversarial examples. The proposed method, called Adversarial Content Attack (ACA), utilizes Image Latent Mapping (ILM) and Adversarial Latent Optimization (ALO) techniques to optimize the latent in a diffusion model. The effectiveness of ACA is validated through experiments and visualization, demonstrating significant improvements of 13.3~50.4% in terms of adversarial transferability compared to state-of-the-art attacks.
Overall, the main contributions of the paper are the introduction of the Content-based Unrestricted Adversarial Attack framework, the development of the Adversarial Content Attack method, and the experiments demonstrating improvements in generating diverse and transferable adversarial examples.
Strengths
The authors effectively communicated the motivation, problem statement, and methodology of the proposed framework. By addressing the limitations (imperceptibility/photorealism/effectiveness) of existing methods, they proposed a novel attack framework that leverages a low-dimensional manifold represented by a generative model. By combining image mapping onto a latent space, optimizing adversarial objectives, and utilizing a diffusion model, the authors introduce a novel approach to generating diverse and natural adversarial examples. This paper might be the first to explore unrestricted adversarial examples through such a framework.
This paper offers a thorough explanation of the proposed attack framework, detailing the underlying techniques of ILM and ALO. The authors further support their claims through experimentation and visualization, providing evidence of the effectiveness of their approach and demonstrating improvements in adversarial transferability compared to state-of-the-art attacks. The improvements in adversarial transferability also shed light on the potential impact of this method in uncovering vulnerabilities in security-sensitive applications and advancing our understanding of robustness in DNNs.
Overall, this paper's strengths encompass originality in proposing a novel attack framework, quality in terms of methodology and experimental evaluation, clarity in explaining the concepts and techniques, and significance in addressing limitations and raising awareness of unrestricted but realistic adversarial examples.
Weaknesses
While the paper has several strengths, there are some weaknesses that could be addressed to further improve the work:
**Comparison with State-of-the-Art Attacks:** While the paper mentions that the proposed method achieves significant improvements in terms of adversarial transferability compared to state-of-the-art attacks, a more comprehensive comparison would strengthen the evaluation. It would be valuable to include a thorough analysis and comparison with a wider range of existing unrestricted attack methods, such as [1]. In particular, Laidlaw et al. proposed an efficient way to generate imperceptible adversarial examples. The reviewer also suggests evaluating the proposed attack on other adversarially trained models, for example, the defense method that could be generalized to unforeseen perturbations [1], or having used synthetic data during adversarial training [2].
**Defense Method:** This paper does not discuss how to defend the proposed attacks but primarily on the efficacy of the proposed method. It would be great if the authors provide potential solutions or mitigation strategies for the threats.
**Generalization to Different Datasets:** The authors only evaluate their method on a subset of the ImageNet validation set and do not say how the results generalize to other datasets. It would be beneficial to investigate the generalization of the proposed approach to different datasets.
**Unclear Claim:** The authors mentioned the Dunning-Kruger effect to emphasize that current defense methods against lp norm adversarial examples overestimate their abilities. However, this work does not provide further details and arguments to support this. Although similar arguments have also been proposed in [3], in which the authors argue that lp-based robustness evaluation might be biased, the reviewer thinks that using the Dunning-Kruger effect here is not rigorous. The reviewer suggests that the authors rethink this argument, and even consider removing it if it is not an important contribution of the paper.
**Typos:** Although this paper is well-written and easy to follow, the reviewer found some typos and grammar errors. For example, in line 161, *follw*. It would be great if the authors have proofread the paper before submitting it.
[1] Laidlaw et al. Perceptual adversarial robustness: Defense against unseen threat models. (ICLR 2021)
[2] Croce et al. Robustbench: a standardized adversarial robustness benchmark. (NeurIPS 2021)
[3] Hsiung et al. CARBEN: Composite Adversarial Robustness Benchmark. (IJCAI 2022)
Questions
Please refer to the weakness part. In brief, the reviewer would like to understand more about the following points:
- Could the authors provide more attack and defense baselines as mentioned in the weakness?
- Could the authors provide some experiments on other datasets?
- Please address the mentioned unclear claim.
- Why the authors did not provide the code for review but answered the Reproducibility as "yes"?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.