The Effectiveness of Random Forgetting for Robust Generalization

Reply to Reviewer FBLx

> The description in the caption of Figure 2 is inconsistent with the content of the image. The former states that the consolidation phase is behind the forgetting phase, while the latter expresses that the consolidation phase is before the forgetting phase. We appreciate your valuable feedback on our manuscript. We acknowledge the inconsistency in the description of Figure 2, specifically regarding the positioning of the consolidation and forgetting phases. In the revised manuscript, we have ensured that the caption accurately reflects the chronological order depicted in Figure 2. > In Figure 1, FOMO is only compared with standard adversarial training, but not with methods that aim to reduce the generalization gap (such as AWP). This result may not appreciably represent the effectiveness of the proposed method. Figure 1 primarily serves as an illustrative aid to help readers grasp the concept of robust overfitting in adversarial training more effectively. To ensure a more comprehensive comparison, we have meticulously presented relevant details in Table 1. This table includes a dedicated section for comparing FOMO with methods like AWP and KD+SWA, which specifically aim to reduce the generalization gap. In light of this, we have refrained from displaying the direct comparison with AWP in Figure 1 to avoid visual clutter. We value the reviewer's perspective, and in response, we will incorporate these changes in the revised version. > The authors use two adversaial attacks to evaluate the proposed method, they can consider more adversarial attacks (such as L2-norm CW, DDN) to conduct a more comprehensive evaluation. Based on reviewer's feedback, we have evaluated our proposed method under CW attack for comprehensive evaluation. Table 10 (Appendix section A8) presents the evaluation results under CW attack on CIFAR-10/100 using the PreActResNet-18 architecture. The robust accuracy is assessed under CW attacks, and checkpoints with the best robust accuracy under PGD-20 attacks on the validation set are selected for comparison. FOMO consistently outperforms both baselines across different datasets and attack scenarios, demonstrating its effectiveness in enhancing robust accuracy under CW attacks. The best and final robust accuracies for FOMO are generally close, indicating that FOMO maintains its performance during training and does not suffer from significant overfitting under these attacks. These results emphasize the promising performance of FOMO in mitigating adversarial attacks, particularly under CW attack. | Dataset | Norm | Radius | Methods | CW Attack | | |-----------|--------|-----------------|---------|------------|-----------------| | | | | | **Best** | **Final** | |-----------|--------|-----------------|---------|------------|-----------------| | CIFAR-10 | $\ell_2$ | $\frac{128}{255}$ | PGD-AT | 67.18 | 64.29 | | | | | KD+SWA | 68.87 | 68.90 | | | | | FOMO | **70.52** | **70.35** | | CIFAR-10 | $\ell_\infty$ | $\frac{8}{255}$ | PGD-AT | 47.00 | 39.96 | | | | | KD+SWA | 49.35 | 49.44 | | | | | FOMO | **52.14** | **51.95** | | CIFAR-100 | $\ell_2$ | $\frac{128}{255}$ | PGD-AT | 37.16 | 33.43 | | | | | KD+SWA | 40.56 | 40.61 | | | | | FOMO | **42.73** | **42.35** | | CIFAR-100 | $\ell_\infty$ | $\frac{8}{255}$ | PGD-AT | 22.73 | 18.11 | | | | | KD+SWA | 25.42 | 25.35 | | | | | FOMO | **26.98** | **26.61** | > Figures can be clearer and more aesthetically pleasing. Thank you for your feedback on the figures. We'll work on improving their clarity and visual appeal in the final version of the paper.

Reply to Reviewer WNii

> The description of the method is too intuitive. We appreciate the reviewer's feedback and the opportunity to clarify the description of our method. Our FOMO approach is founded on the premise that emulating active forgetting, a phenomenon observed in the human brain, can effectively mitigate robust overfitting in Adversarial Training (AT). Our method synthesizes selective forgetting in later layers, consolidation for long-term memory, and interleaved training as mechanisms aimed at enhancing generalization. We want to emphasize that our method's development wasn't solely intuitive; it is substantiated by evidence from active forgetting studies in neuroscience, which we extensively discuss in Section 2.3 and the methodology section. Additionally, we conducted rigorous empirical evaluations across multiple datasets to validate our hypothesis. The discernible enhancements in robustness and generalization provide empirical evidence reinforcing the effectiveness of our method. We believe that while our approach may seem intuitive in its concept, its implementation and validation have been rooted in both neuroscience understanding and empirical substantiation. We remain open to further discussions or specific areas where additional clarification might be beneficial. > In Table 1, the delta, which measures the adversarial overfitting, never favors the proposed method. This cannot show that the proposed method is good at dealing with adversarial overfitting. We would like to clarify that delta alone may not be a comprehensive metric for quantifying robust overfitting, as some methods with strict regularization (such as KD+SWA) during training might exhibit lower delta values at the expense of achieving less-than-optimal the best robust accuracy. FOMO is designed to strike a balanced tradeoff between delta and best robust accuracy. Compared to KD+SWA, FOMO exhibits a notable percentage increase in the best robust accuracy: 2.08% on CIFAR-10, 2.01% on CIFAR-100, and 5.98% on SVHN, all while maintaining a minimal difference between best and final robust accuracy. These percentages highlight FOMO's effectiveness in not only mitigating adversarial overfitting but also in significantly enhancing the overall robustness of the model compared to the baseline. > In this paper, the author only shows the result under a combination of white box and black box attacks, i.e., Autoattack. However, this cannot show "the efficacy of FOMO against the black box and white box attacks". Standard Autoattack has 4 adversarial attacks: three white-box attacks and one black-box attack. It is possible that FOMO has a strong resistance against white-box attacks while being vulnerable to the black-box attack. Thank you for your question. In response to your observation, we have made revisions to the paper to explicitly address "the efficacy of FOMO against AutoAttacks." We present results on AutoAttack, a combination of white-box and black-box attacks, mirroring our considered baselines. It's crucial to note that baseline methods, such as KD+SWA and AWP, similarly lack separate results for white-box and black-box attacks. To facilitate a fair comparison, we also present the AutoAttack results. We acknowledge this and will update the paper to reflect this clarity.

Reply to Reviewer Jeha

> My main concern is that the proposed method seems to be heuristic and empirical. there is not enough discussion on its intuition or theoretical foundation. We genuinely value the reviewer's insights. Our FOMO approach stems from the hypothesis that simulating active forgetting, as observed in the human brain, can effectively address robust overfitting in Adversarial Training (AT). Our method integrates selective forgetting in later layers, consolidation for long-term memory, and interleaved training as mechanisms to improve generalization. Our rationale is supported by evidence from active forgetting studies in neuroscience, which is extensively elaborated in Section 2.3 and the methodology section. Furthermore, we have empirically validated our hypothesis by conducting comprehensive evaluations across multiple datasets. The noticeable improvements in robustness and generalization serve as empirical evidence reinforcing the effectiveness of our method. While we acknowledge the importance of theoretical analysis, our primary focus was to showcase the practical efficacy of our proposed method. However, subsequent to our empirical findings, a theoretical exploration can further deepen our understandings of the learning mechanisms employed by FOMO. This aspect is explicitly discussed in our future work section, emphasizing the necessity of further theoretical analysis to complement our empirical findings. > I don't think the running time and convergence analysis are well-studied in this paper, the authors may need to provide a table showing how many epochs are needed to converge and compare the running time with the existing methods. We appreciate your suggestion and thus, we have conducted additional experiments and present the training time (per epoch) for several methods in our revised manuscript. To ensure a fair comparison, all methods were integrated into a universal training framework, and each test was performed on a single NVIDIA GeForce 3090 GPU. Table 9 (Appendix section A7) in the revised manuscript now includes the required information. Notably, FOMO and our baselines were trained for the same number of epochs (i.e., 200 epochs for CIFAR-10/100). From the table, it is evident that FOMO imposes nearly no extra computational cost compared to vanilla PGD-AT, with specific values being 137.1s for FOMO and 132.6s for vanilla PGD-AT per epoch. This implies that FOMO is an efficient training method in practical terms. It is important to highlight that KD+SWA, a formidable method designed to counter robust overfitting, comes with an increased computational cost. This arises from its approach, which entails the pretraining of both a robust and a non-robust classifier, serving as the adversarial and standard teacher, respectively. Additionally, the method incorporates the process of distilling knowledge from these teachers. Moreover, KD+SWA employs stochastic weight averaging to smoothen the weights of the model, further contributing to its computational demands.We believe that this addition enhances the practical insights into the efficiency of FOMO and its comparison with existing methods. We would be happy to address any remaining concern or suggestion. ### Table: Training Time per Epoch on CIFAR-10 under ε∞ = 8/255 Perturbation (PreActResNet-18) | Methods | Training Time per Epoch (s) | | ------------- | ------------------------| | PGD-AT | 132.6 | | WA | 133.1 | | KD+SWA | 132.6 (pretraining of AT) +16.5 +141.7 | | AWP | 143.8 | | FOMO | 137.1 | > Minor: Please refrain from only using color to distinguish curves and bars as in Figures 3, 4, 5, and 6, as it is not friendly to readers with color blindness. We value your feedback. In the final version, we'll modify these figures to ensure they're accessible to readers with color vision deficiencies. Thank you for highlighting this important consideration. > Minor: Missing reference on robust generalization: Zhang, et al. "The limitations of adversarial training and the blind-spot attack." ICLR 2019. Thank you for your suggestions. We have added this in the related work under robust generalization.