Summary
This paper introduces two novel adversarial attacks, specifically designed to target the evasion of deep neural networks (DNNs) in classification tasks involving tabular data satisfying real-world constraints. The first attack, CAPGD, is a gradient-based method that enhances the constrained PGD (CPGD) attack proposed in [33] by incorporating improvements from the APGD definition [12], such as the addition of momentum and an optimized step size selection. The second attack, CAA, combines the efficient but less effective CAPGD with another search-based attack, MOEVA, which is more effective but less efficient regarding the computational time required to perform the attack.
The extensive experimental evaluation on four tabular datasets and five DNNs for tabular data demonstrates that CAPGD significantly improves the success rate of generating successful adversarial examples compared to CPGD. Furthermore, CAA achieves a reasonable trade-off between efficacy and computational efficiency, offering higher attack efficacy than CAPGD and requiring less computational time than MOEVA.
Strengths
I find this paper particularly compelling as it proposes a new evasion attack against deep learning models for tabular data with real constraints that clearly outperforms the other state-of-the-art proposals. The paper demonstrates two key strengths:
- Despite the two proposed attacks are improvements and combinations of existing approaches to generate realistic attacks, they are based on reasonable and novel observations. In particular, the design of CAA is motivated by the evidence that CAPGD and MOEVA are complementary. These attacks significantly outperform the state-of-the-art CPGD [33].
- The experimental analysis is comprehensive and convincing. The experiments are performed on 5 different deep learning models employed to classify tabular data on 4 different datasets with variable number of constraints. Both the accuracy and efficiency of the attacks are evaluated. Furthermore, additional results are shown varying the maximum perturbation and the value of important parameters of the attacks are shown. Finally, an ablation study shows the impact of each improvement introduced in CAPGD.
Weaknesses
Even though the proposal is good, I think that some weaknesses about the presentation of the proposal and the analysis of the results need to be addressed in the final version of the paper:
**The description of CAPGD should be improved**: the description of CAPGD can be enhanced by better explaining the role of the repair operator. Specifically, the authors mention that constraints of the form $f = \psi$ are enforced at every iteration by the repair operator. However, it is not clear whether applying this operator at every iteration guarantees the generation of an adversarial example that satisfies all dataset constraints and the maximum perturbation constraint by the end of the execution. This is a crucial property, as the attack cannot be tested if it does not satisfy the constraints, potentially wasting the time required to generate the attack.
**The depth of the analysis of the results should be improved**: the depth of the analysis of the results can be improved by discussing particular results to understand their causes and how to mitigate these cases. For instance, CAPGD and MOEVA fail to generate adversarial examples on the CTU dataset for 2 out of 5 models (Table 3 and results in the Appendix). This failure may be due to the high number of constraints (360) considered for this dataset, since the two attacks are more effective on the other datasets for which at most 30 constraints are considered. It would be interesting to evaluate the efficacy of the two attacks by varying the number of constraints considered for the CTU dataset, to better highlight the limitations of the proposed attacks.
Additionally, CAA generates less effective attacks with higher epsilon values (Figure 5b), and MOEVA generates less effective attacks with a higher number of iterations (Figure 7b). Even though these results are sporadic, they are counterintuitive, as the search space for adversarial examples increases with higher epsilon and more mutation rounds. A detailed discussion of these phenomena would provide valuable insights.
**The limitations section is missing**: the paper should include a limitations section summarizing the limitations of the presented attack algorithms and their evaluation. For example, the evaluation is performed only against an $L_2$ attacker, whereas other norm-based or synthetic attackers could have been considered. The attacks may not work well against specific datasets and models when $\epsilon$ is small. While these limitations are reported in the paper, they are scattered throughout. A dedicated section would clarify these points.
**Typos and inconsistencies**: there are some typos and inconsistencies in the notation, such as:
- $R_\Omega$ in Eq. 3 is not defined when the equation is presented.
- $x^{(k+1)}$ -> $x^{(k-1)}$ in Eq. 7.
- The classifier is represented by $H$ in Algorithm A.2, but $h$ is used in the main body of the paper. Additionally, $X$ appears twice in the function in line 3 of Algorithm A.2.
Correcting these typos and inconsistencies will improve the clarity and readability of the paper.
Finally, another weakness is more generic:
**The proposed attacks are really specific**: the paper is proposing two effective algorithms to forge realistic evasion attacks against deep learning models used for classifying tabular data, an under-explored setting. I think that the contribution is appreciable. However, the chosen setting may limit the impact of the results, since, at the best of my knowledge, tree-based models continue to be the state-of-the-art for classification tasks on tabular data [a].
[a] Ravid Shwartz-Ziv and Amitai Armon, Tabular Data: Deep Learning is Not All You Need, Information Fusion Volume 81, May 2022, Pages 84-90
**Updates after the authors' response**
The authors have provided a satisfying response addressing all the concerns that I have raised as weaknesses. The authors are willing to introduce the findings provided in the response in the next version of their paper, as far as I understood. Moreover, they will also include a dedicated limitations section and correct the typos.
Questions
- Is CAPGD guaranteed to generate an attack that satisfies the dataset and perturbation constraints at the end of execution?
- CAPGD and MOEVA may be unable to generate adversarial examples in specific settings (see results on two models on the CTU dataset in Table 3). What is the reason? Could it be due to the high number of constraints of the dataset? How does the success rate of the two attacks vary considering different numbers and types of constraints (linear and nonlinear)?
- Why may CAA and MOEVA show a smaller success rate when considering higher epsilon values and a higher number of rounds (see Figures 5b and 7b)?
Limitations
The authors should sum up the limitations of their proposals in a specific section in the paper. The negative societal impact is discussed by the authors who claim that their work may give birth to new and stronger defenses. Finally, the experimental and implementation details have been extensively documented.