Summary
The paper proposes a new method combining Wasserstein distance, DRO, and adversarial attack. The novelty is that the method computes the adversarial attack bounded by the Wasserstein distance. The paper provides a bound on the performance of out-of-domain samples. It applies the new adversarial attack on CIFAR10 dataset. The numerical results show that the attack is effective on models that are robust to pointwise attacks, and computing attacks is much faster than other methods.
Strengths
1. The paper proposes an interesting direction by combining DRO and adversarial attacks. It is clear in the writing and presenting methods.
2. The paper gives some theoretical results on the proposed methods, like lower bound on $R_{\delta}$. The paper also shows numerical results comparing the lower bound from different methods.
3. The paper provides numerical results to show that the adversarial attack is effective at attacking and fast to compute.
Weaknesses
1. The paper misses some references and does not compare the method with the existing DRO adversarial attack methods, like [A].
[B, C] also have a similar approach by combining DRO with adversarial attacks. In the area of traditional attacks, the paper can compare with more methods, like CW attack in [D].
[A] Sinha, A., Namkoong, H., Volpi, R. and Duchi, J., 2017. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571.
[B] Volpi, R., Namkoong, H., Sener, O., Duchi, J.C., Murino, V. and Savarese, S., 2018. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31.
[C] Hua, X., Xu, H., Blanchet, J. and Nguyen, V.A., 2022, December. Human imperceptible attacks and applications to improve fairness. In 2022 Winter Simulation Conference (WSC) (pp. 2641-2652). IEEE.
[D] Carlini, N. and Wagner, D., 2017, May. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) (pp. 39-57). Ieee.
2. The paper only does experiments on CIFAR10 dataset. It would be helpful to see results on other datasets.
Questions
1. For the computation time in Table 2, do the authors compare with PGD? PGD is a simple gradient-based method, so the computation time should be very small. Other methods, like FGSM, are also very fast. Also, for Table 2, do all the algorithms stop when they successfully attack all the images?
2. In line 165, the authors mention another distributional attack, why this method is not compared to in the paper?
3. In line 39, the paper mentions the approach "is better suited for gradient-based optimization methods than duality approach adopted in most of the works to date". Can you clarify on this point?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors discuss some of the limitations of the work, including only valid on small-scale attacks. More limitations could include the stability of the attack and how it compares to more state-of-the-art attack methods. The method also should be compared with more attack methods, like DRO-based attacks. From the current experiments, it is not very convincing that the method outperforms many other adversarial attack methods.