Improving Robustness with Adaptive Weight Decay

We propose adaptive weight decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. For classification problems, we propose changing the value of the weight decay hyper-parameter on the fly based on the strength of updates from the classification loss (i.e., gradient of cross-entropy), and the regularization loss (i.e., $\ell_2$-norm of the weights). We show that this simple modification can result in large improvements in adversarial robustness -- an area which suffers from robust overfitting -- without requiring extra data across various datasets and architecture choices. For example, our reformulation results in $20\%$ relative robustness improvement for CIFAR-100, and $10\%$ relative robustness improvement on CIFAR-10 comparing to the best tuned hyper-parameters of traditional weight decay resulting in models that have comparable performance to SOTA robustness methods. In addition, this method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms, which the latter contributes to robustness to overfitting to label noise, and pruning.

cs.LG cs.CV Adversarial Robustness in Machine Learning Anomaly Detection Techniques and Applications Advanced Neural Network Applications Computer Science

Source

Paper

References (72)

01Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language ModelShaden Smith, M. Patwary, Brandon Norick et al.2022 · arXiv.org · 861 citations In Library

02Probabilistic Margins for Instance Reweighting in Adversarial TrainingQizhou Wang, Feng Liu, Bo Han et al.2021 · Neural Information Processing Systems · 83 citations In Library

03TrivialAugment: Tuning-free Yet State-of-the-Art Data AugmentationSamuel G. Müller, Frank Hutter2021 · IEEE International Conference on Computer Vision · 407 citations In Library

04Fixing Data Augmentation to Improve Adversarial RobustnessSylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian et al.2021 · arXiv.org · 320 citations In Library

05Understanding deep learning (still) requires rethinking generalizationChiyuan Zhang, Samy Bengio, Moritz Hardt et al.2021 · Communications of the ACM · 2.8k citations

06Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized LabelsSangdoo Yun, Seong Joon Oh, Byeongho Heo et al.2021 · Computer Vision and Pattern Recognition · 176 citations In Library

07RobustBench: a standardized adversarial robustness benchmarkFrancesco Croce, Maksym Andriushchenko, Vikash Sehwag et al.2020 · NeurIPS Datasets and Benchmarks · 935 citations In Library

08Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial ExamplesSven Gowal, Chongli Qin, Jonathan Uesato et al.2020 · arXiv.org · 378 citations In Library

09Geometry-aware Instance-reweighted Adversarial TrainingJingfeng Zhang, Jianing Zhu, Gang Niu et al.2020 · International Conference on Learning Representations · 319 citations In Library

10DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion ParametersJeff Rasley, Samyam Rajbhandari, Olatunji Ruwase et al.2020 · Knowledge Discovery and Data Mining · 2.2k citations

11Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder et al.2020 · Neural Information Processing Systems · 61k citations In Library

12Improving Adversarial Robustness Requires Revisiting Misclassified ExamplesYisen Wang, Difan Zou, Jinfeng Yi et al.2020 · International Conference on Learning Representations · 851 citations

Scroll for more · 38 remaining

Adaptive Weight Decay for Deep Neural NetworksKensuke Nakamura, Byung-Woo Hong2019 · IEEE Access · 55 citations In Library

Rethinking Weight Decay: Anisotropic Regularization for Improved Generalization in Deep LearningRevista, Zen, IA, 102025 · Zenodo (CERN European Organization for Nuclear Research) In Library

Rethinking Weight Decay for Robust Fine-Tuning of Foundation ModelsJunjiao Tian, Chengyue Huang, Z. Kira2024 · Neural Information Processing Systems · 11 citations In Library

Understanding Decoupled and Early Weight DecayJohan Björck, Kilian Q. Weinberger, Carla P. Gomes2020 · arXiv (Cornell University) · 40 citations In Library

Understanding Decoupled and Early Weight DecayJohan Björck, Kilian Q. Weinberger, Carla P. Gomes2021 · Proceedings of the AAAI Conference on Artificial Intelligence In Library

Weight Decay With Tailored Adam on Scale-Invariant Weights for Better GeneralizationXixi Jia, Xiangchu Feng, Hongwei Yong et al.2022 · IEEE Transactions on Neural Networks and Learning Systems · 26 citations In Library

Improve Adversarial Robustness via Weight Penalization on Classification LayerCong Xu, Dan Li, Min Yang2020 · arXiv (Cornell University) · 4 citations In Library

Dynamic Weighting Loss for Decision Boundary Adjustment based on Robust Distance in Adversarial TrainingYiqun Xu, Zhen Wei, Zhehao Li et al.2025 · IEEE International Conference on Multimedia and Expo · 1 citation In Library

Patents near this paper

Patent US 5,987,444US 5,987,444 ROBUST MODELINGUS 6,523,015 · KXEN · 2003 HIGH ACCURACY LEARNING BY BOOSTING WEAK LEARNERSUS 9,607,246 · THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK · 2017 ROBUST PRUNED NEURAL NETWORKS VIA ADVERSARIAL TRAININGUS 11,562,244 · ROYAL BANK OF CANADA · 2023 ROBUST CLASSIFIERUS 10,929,774 · KONINKLIJKE PHILIPS N.V. · 2021

Peer review

Accept (poster)NeurIPS 2023 · via OpenReview

Reviewer 3gZW6/10 · confidence 3/52023-07-03

Summary

To improve the robustness, this paper proposes a method to determine the weight decay hyper-parameter during adversarial training adaptively. The key idea is to select the proper weight decay parameter $(\lambda_t )$ to keep the decay over the gradient (DoG) as a constant. The proposed method is evaluated on image classification and label noises tasks, comparing with the adversarial training with fixed weight decay hyper-parameter $(\lambda_{wd})$. According to the experimental results, adversarial training with dynamic $\lambda_t$ performs better.

Strengths

1) The comparison results visualized by figures (Figures 2 & 3) are concise and clear.

Weaknesses

1) The writing is not good, and the research gap and motivation are vague. 2) How this method will benefit pruning, as mentioned in the abstract, is not well supported. 3) The essential part of determining the constant value of DoG is not clearly stated.

Questions

1) The way to organize the paper does not look good to me. The introduction part (lines 16 - 37) is too short, and the related work section (lines 149 - 155) appears after the proposed method. The relations between the proposed work and those existing works are blurred. I cannot see a clear clue about how the idea came up or what bottlenecks this proposed method aims to solve. 2) About the grid search strategy for $\lambda_{wd}$ and the selection for $DoG$. Taking Figure 3 as an example, I am curious about how to search for those two hyper-parameters. What is the search interval, and what is the search step? Will the search stage be very costly? Can there be a more efficient guide to deciding the constant value of DoG? 3) About the generality of this method. The proposed method can only be utilized with cross-entropy plus $l_2$-norm regularizer? If yes, I am concerned about the generality of this proposed method. 4) About the benefits that this method can bring to network pruning. This point is mentioned in the abstract (line 14) and Section 3 (line 219). However, this paper does not provide any mathematical analysis or experimental results. More important, the listed references (lines 218 - 219) are not state-of-the-art. Thus, I think the claim is a bit exaggerated. 5) About the editorial issues that make the reading difficult. The proposed method is called Adaptive Weight Decay / AWD for a while and DoG for a while. I suggest using a fixed abbreviation for the proposed method. Besides, many descriptions are inconsistent, though not big issues from the grammar aspect, they harm the reading experience significantly: - when mentioning a section, lines 31 & 35, lines 146 & 212, and lines 190 & 195. There are at least three different ways to mention the section. - when mentioning a figure, line 88 and line 99 are inconsistent. - when defining a variable, the definitions of $w$ are different in line 45 and line 68, parameter or parameters? - when mentioning the algorithm, lines 127 and line 133 are different. - …

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Reviewer 3gZW2023-08-20

Response to the Authors

Thanks for the detailed explanations about my questions about: 1) the relations between the proposed method and pruning and, 2) the hyper-parameter selection strategy. Based on that, I would like to raise my score to 6.

Reviewer pbDj5/10 · confidence 3/52023-07-04

Summary

This paper proposes a simple but efficient way to improve model robustness: Adaptive Weight Decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. Experimental results prove that this method significantly improves the robustness of the model on multiple datasets.

Strengths

1. The paper is well-written and the proposed methods are clearly formulated. 2. Experimental results prove that this method significantly improves the robustness of the model on multiple datasets.

Weaknesses

1. It would be better if more theoretical explanations about AWD could be provided. 2. Although the proposed method has a great improvement over the baseline method, it does not compare with other SOTA methods. In particular, on CIFAR10, the AWD only achieves 50.03% robustness under AA attack using the WideResNet-28-10. As far as I know, it is less robust than other methods (e.g., early stop). 3. The hyperparameter (DoG) of AWD has a great influence on the result, which makes the proposed method not so adaptive.

Questions

See Weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

See Weaknesses

Reviewer 61Q24/10 · confidence 4/52023-07-05

Summary

This paper proposed adaptive weight decay which is balancing the gradient of the loss funciton such as the coss-entropy loss and the weight decay term. Althogh the proposed method is a simple method, it empirically improves adversarial robustness and a classification with label noise.

Strengths

The strong point of this paper is the simplicity of the proposed method. The proposed method is just adaptively rebalancing the gradient of the loss and weight decay term. It empiciallly outperforms the existing studies of adversarial robustness.

Weaknesses

The main concern with this method is that we do not know what the algorithm is ultimately optimizing. We cannot know if the algorithm converges even in optimization problems such as convex optimization. We also cannot know why this method is good for adversarial robustness and label noize.

Questions

Q1. I would like to know if this algorithm is designed to minimize the original optimization problem. It is difficult to set up in general, but usually when considering adaptive optimization algorithms like this, it is important for the safety of the algorithm that convergence is guaranteed, at least for optimization of convex functions. Q2. What is inevitable about equation (7). Of course, you are scaling with the gradient of the cross entropy loss, but I would like to know the direct effect of why that is effective for adversarial robustness. Q3. Why do robust validation accuracy and loss suddenly improve around 200 epochs? Since it is possible that the performance is good when stopping early at a good 200 epochs by chance, we would like to see a discussion on convergence. Q4. What happens if we increase the number of epochs up to 300 epochs? Q5. In Table 2, Epo varies for existing methods; if each method is adjusted in terms of early stopping, this may not be a problem, but the conditions should be as consistent as possible. Q6. Does the proposed method work well in a more natural setting such as ImageNet-A? The adversarial robustness setting is a bit artificial, which is a problem of this fields itself. Q7. What is the performance in the normal problem setting instead of adversarial robustness? In particular, do we see a performance decrease with different DOG settings? If so, how do we determine the final DOG value? Even if the user makes the final decision, if the performance of the normal setting is significantly degraded as a result of considering adversarial robustness, we may not want to use it as a method.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

No convergence guarantees for adaptive algorithms. No theoretical basis for performance improvement against adversarial robustness.

Reviewer N3sV6/10 · confidence 5/52023-07-06

Summary

The paper proposes adaptive weight decay (AWD) to adaptively tune the weight decay hyperparameter during training. The AWD keeps the ratio of weight decay update and cross-entropy loss update constant for the stability of training. The experiment shows that AWD improves adversarial robustness and reduces robust overfitting in adversarial training.

Strengths

The paper is clearly written and easy to follow. The experiment shows the benefit of AWD over several baselines on CIFAR10/100 and Tiny ImageNet.

Weaknesses

As a reviewer who reviewed the submission for 3 times, my major concern before is the comparison of AWD with recent baselines like MART and MAIL. I am glad that Table 2 of this version includes such a comparison. The result looks promising to me so I would not raise any major weaknesses for this version. One minor issue is the training time difference between DoG and baselines. Table 2 shows that DoG needs 200 epochs of training while all baselines use 100 epochs. It is not clear how these baselines will perform if we increase the training time, with learning rate decay delayed of course.

Questions

Is the proposed method still better if baselines like MART and MAIL are trained for the same time as AWD?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Not discussed in the main paper.

Reviewer c9pV6/10 · confidence 4/52023-07-13

Summary

This paper studies the overfitting phenomena that is known to happen during adversarial training, with focus in image classification. The main idea hinges on the fact that weight regularization can be an effective technique to prevent such overfitting. The authors propose to augment the regular cross-entropy objective during training with a weight-decay regularization with an adaptive parameter to better utilize this toward preventing overfitting. Due to lack of theoretical analysis, the idea is tested numerically with empirical test on a number of image classification tasks and its effectiveness is shown empirically.

Strengths

The main advantage of this work is its simplicity as well as it adequate development of the idea from initial observations to intuitively explaining why it makes sense to propose this approach.

Weaknesses

The tests carried out in this paper seem somewhat limited. While the idea is based on state-of-the-art work in the literature [Rice et al., etc.], the results are not compared with these algorithms. Different choices of $\epsilon$ as well as attacks are also needed to show effectiveness of the proposition across a wide range of setups, especially given that the work is a heuristic approach without any theoretical evidence.

Questions

1- The tests should be carried out for a couple of different values of $\epsilon$ as well as other standard attacks, to demonstrate performance across a range of settings. In addition to AutoAttack, performance against PGD attacks with large enough #steps (say K= 40 or 100) and small enough step-size (say $2 \epsilon /K)$ is common practice to report. 2- Results in table 1 need to also be compared with other state of the art algorithms for comparison, and not just constant versus adaptive weight decay. 3- As mentioned in the paper, using additional data has shown to be effective in improving robustness. How effective is adaptive decay if also combined with such additional data? 4- denoting the gradient of the cross-entropy loss part of the objective with respect to $w_t$ as $\nebla w_t$ is mathematically wrong (see eq. 5 , 6 ,7). This needs to be corrected for mathematical soundness. Also, it is reflected in line 6 of the Algorithm 1. Instead, one should define a function, say $F(.)$ to denote the cross entropy, and use that for mathematical soundness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Reviewer pbDj2023-08-16

Reply to the author

Thanks for the authors' response. After reading the rebuttals, I still have some concerns. 1 Please describe the experimental setup in Figure 3 in detail. it is confusing. 2 From Figure 3 of the submitted manuscript, the hyperparameter (DoG) of AWD has a great influence on the result, If we face a new dataset or network, how should we choose DoG?

Authorsrebuttal2023-08-16

> 1 Please describe the experimental setup in Figure 3 in detail. it is confusing. Thank you, dear Reviewer pbDj, for helping us identify that the plotting function did not sort the x-axis in figure3. We believe it may have been the source of confusion? While the results remain intact, we will fix the plotting in the camera-ready/future versions. Since we could not update the figure during the discussion period, we have gathered the information of that figure and compiled it in table format (and included clean/nat accuracy) where the hyper-parameters are sorted (Table1-Table4). We would also like to use this opportunity to clarify the experiments in Figure 3. We train WRN28-10 networks using traditional weight-decay (with lambda hyperparameter) and another time using AWD (with DoG hyperparameter). In these experiments, we vary the hyper-parameters [by doing a grid search within a reasonable range] and do validation-based-early-stopping to pick the most robust checkpoint using the held-out validation set. We then report the robust accuracy on the test set by attacking the models using AutoAttack (AA). Regarding the range of the hyperparameters searched, we search for twice as many hyperparameters for WD compared to AWD and ensure that the optimal point is not at one of the two ends of the search range. |DoG|C10 - Nat|C10 - AA|C100 - Nat|C100 - AA| |:-:|:-:|:-:|:-:|:-:| |0.02|**88.05**|49.01|**62.85**|25.75| |0.02181|87.08|**50.03**|61.39|**27.15**| |0.02378|86.78|49.72|59.51|26.61| |0.02594|85.90|50.00|55.97|26.61| |0.02828|84.50|49.21|51.12|25.70| |0.03084|82.25|47.93|47.24|24.63| |0.03364|77.15|45.53|41.16|22.69| |0.03668|72.91|44.23|36.30|20.64| |0.04|67.56|41.35|31.03|18.19| Table 1: Table from Figure 3 -- AWD for CIFAR10 and CIFAR-100. |Lambda|C10 - Nat|C10 - AA|C100 - Nat|C100 - AA| |:-:|:-:|:-:|:-:|:-:| |0.00005|42.62|82.66|20.86|54.90| |0.000067|41.97|83.26|19.37|53.95| |0.000089|43.43|81.89|20.39|59.91| |0.000119|43.39|82.27|21.08|59.50| |0.000158|43.28|82.71|21.51|59.45| |0.000211|43.38|85.97|21.66|59.37| |0.000281|43.64|82.11|21.90|60.01| |0.000375|44.19|86.34|21.75|59.61| |0.0005|44.03|86.40|**22.53**|60.15| |0.00067|44.24|**86.59**|21.61|60.54| |0.00089|**45.19**|84.31|21.55|**61.33**| |0.00119|43.00|84.29|21.19|57.75| |0.00158|44.15|85.17|21.29|59.31| |0.00211|43.39|85.11|22.06|59.99| |0.00281|43.35|85.34|22.07|58.26| |0.00375|42.27|81.31|20.49|50.66| |0.005|37.23|72.96|17.76|41.96| Table 2: Table from Figure 3 -- WD for CIFAR-10 and CIFAR-100 |Lambda|Tiny - Nat|Tiny - AA| |:-:|:-:|:-:| |5E-05|45.08|10.44| |6.7E-05|45.15|10.72| |8.9E-05|45.81|10.86| |0.000119|46.51|10.81| |0.000158|43.81|10.75| |0.000211|44.89|11.25| |0.000281|47.04|11.94| |0.000375|48.08|11.58| |0.0005|49.77|11.79| |0.00067|50.24|11.79| |0.00089|50.01|12.41| |0.00119|**52.90**|15.08| |0.00158|52.05|16.42| |0.00211|47.87|**16.73**| |0.00281|38.85|14.25| |0.00375|24.46|8.41| |0.005|6.67|2.70| Table 3: Table from Figure 3 -- WD for TinyImageNet |Dog|Tiny - Nat|Tiny - AA| |:-:|:-:|:-:| |0.01361|**54.35**|17.33| |0.0147|53.11|18.79| |0.01587|51.34|19.38| |0.01714|48.46|**19.74**| |0.01852|45.23|19.74| |0.02|41.90|19.33| |0.02181|36.34|17.68| |0.02378|30.93|15.74| |0.02594|25.97|13.89| Table 4: Table from Figure 3 -- AWD for TinyImageNet > 2 From Figure 3 of the submitted manuscript, the hyperparameter (DoG) of AWD has a great influence on the result, If we face a new dataset or network, how should we choose DoG? Thank you for raising this question. Given that a similar question was also raised by reviewer 3gZW, we will include a reference to this in the appendix for the future revision. While as seen in figure3, the hyper-parameter value of 0.021 has resulted in improvements over the optimal hyper-parameter for traditional weight decay across all 3 datasets, the results can be improved even further by doing a proper grid search. The preferred method for finding the best hyper-parameter for adaptive weight decay (i.e., DoG) is to treat it as any other hyper-parameter and perform a grid-search. But having a good estimate for the range used during grid search has value. Hence, we present a method to find a good estimate which works well in practice. We suggest that if a training pipeline exists that is fully tuned for non-adaptive/traditional weight decay, we can get a reasonable estimate for DoG by taking an average (or moving average) of the DoG values from all iterations during training with traditional weight decay. This estimate works well in practice for training with AWD and comes with almost no additional computation overhead. For most experiments through out the paper and the rebuttal, we used the average(DoG) of WD experiment (e.g. 0.02) to get an estimate for the optimum DoG. Then we used a geometric progression with 16 steps between 2x smaller (e.g. 0.01) and 2x larger (e.g. 0.04) values of the estimated DoG for our grid search.

Reviewer pbDj2023-08-21

Reply to the author

Thanks for the authors' response. From the experimental results provided by the author, I am getting more and more confused. For example, in Table 2: Table from Figure 3 -- WD for CIFAR-10 and CIFAR-100, the author claimed the network is WRN28-10. the robustness of the WRN28-10 under AA is too lower. Please refer to the results in [1]. I think the experimental settings of this result may be unreasonable, resulting in the low results of Table 2. Secondly, how to select DoG still needs to go through a grid search, which makes the proposed algorithm not adaptive. Ref: [1] BAG OF TRICKS FOR ADVERSARIAL TRAINING

Authorsrebuttal2023-08-21

Thank you for sharing your feedback and allowing us to clarify the numbers in the motivational/baseline figure3 (more specifically, Table2 in our rebuttal.) Table2 uses common settings for training (w/o) any extra optimizations/modifications apart from tuning weight-decay and uses validation-based-early stopping per Rice et al. We continue using this common setting for our method in the SOTA comparison table (table P1) as well. Please note that in Table P1, where we have compared to SOTA methods, the robustness numbers for CIFAR-10 are higher than the best reported ones in [1] which use more similar architectures. For Table P1 we use a WRN32-10 which is fairly similar to WRN34-10 used in [1]. Please note that after all the tricks used in Table9 of [1], the most robust WRN34-10 model under the 8/255 adversarial attack has 53.88 AA robust accuracy and 84.24 clean accuracy, while when training a WRN32-10 using AWD — using conventional training settings and without any bag of tricks — our model has 54.04 robust accuracy and 88.55 clean accuracy. In regards to Table2 in the rebuttal and the training setting used in [1]: Training setting in [1] is an interesting setting which differs from the work of rice et al. and many others. The most notable difference is the learning rate schedule, where in [1] it is tailored to work well on CIFAR-10 such that robust overfitting is remedied to a great extent. To be more precise, in [1], they train for 110 epochs and use an initial learning rate of 0.1, which drops to .01 at the 100th epoch, and further drops to 0.001 at the 150th epoch. This is while, in all works that discuss robust overfitting, the learning rate either follows a cosine or it drops to 0.01 halfway through training, and it further drops to 0.001 during the last quarter of training. We would like to clarify that in all of our experiments we use a training setting that is a common setting used by most works in the literature that report results on various datasets. We use this common setting for Table2 in our rebuttal response as well to demonstrate the effectiveness of adaptive weight-decay over traditional weight-decay to prevent robust overfitting. While redoing Table2 for CIFAR-10, with the training setting used in [1] might still be interesting, given that only a few hours remain in the discussion period and, also that our AWD models in Table P1 (with much less bag of tricks) are both more robust and have higher clean accuracy than that reported in [1] (with many bag of tricks + uncommon training setting), we would like to believe that AWD still demonstrates its usability in settings which suffer from robust overfitting.

Reviewer N3sV2023-08-16

After Rebuttal

Thanks to the authors for consistently improving this paper along the (kind of long) journal. My concerns are addressed and I will keep my score.

Reviewer 61Q22023-08-18

After Rebuttal

Thanks for your reply to my review. I have cleared up some questions, but I still have questions about the following two points. >Q1. I would like to know if this algorithm is designed to minimize the original optimization problem. It is difficult to set up in general, but usually when considering adaptive optimization algorithms like this, it is important for the safety of the algorithm that convergence is guaranteed, at least for optimization of convex functions. What I want to know is not an intuitive explanation, but what is actually being solved as an optimization problem and does the proposed algorithm converge in a simple convex minimization. > Q2. What is inevitable about equation (7). Of course, you are scaling with the gradient of the cross entropy loss, but I would like to know the direct effect of why that is effective for adversarial robustness. You explains this reason in terms of flatter minima and generalize. However, flatness is a formulation based on perturbations in parameter space, while adversarial robustness is a formulation based on perturbations in input space. Thus, this is not a direct explanation.

Authorsrebuttal2023-08-20

We are happy that many of your concerns have been addressed and are grateful for giving us an opportunity to address/clarify the 2 remaining ones. Please find our responses below. > Q1. What I want to know is not an intuitive explanation, but what is actually being solved as an optimization problem and does the proposed algorithm converge in a simple convex minimization. Following your ask, below, we show that for a simple choice of convex functions, our proposed loss function is locally convex (at least near the minimizer), and hence SGD will be able to converge to a stable/stationary local minima. Consider the following simple regression problem where the main objective is MSE: $0.5 \cdot \| x- \beta \|^2$. In this setting, our total loss formulation that includes the adaptive weight-decay regularizer term will be: $0.5 \cdot \| x- \beta \|^2 + c \cdot \frac{|x-\beta|}{|x|}$, where $c=0.5 \cdot DoG$ and $x \neq 0$. We consider all cases for possible choices of $\beta$ and $x$ and show that for all 7 cases, as long as we chose a the hyper-parameter $0 < c < \frac{\beta^2}{2}$ which translates to $ 0 < DoG < \beta^2 $ , the problem is locally convex in those regimes by showing that the second derivative is always positive. |$\beta$|$x$|$\beta$ vs. $x$ Comparison|Minimization|Simplified Minimization|2nd Derivative|Condition for Convexity|$c$ Satisfying Convexity| |:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| |$\beta>0$|$x>0$|$x>\beta$|$0.5\cdot\|x-\beta\|^2+c\frac{ \|x-\beta\| }{ \|x\| }$|$0.5\cdot(x-\beta)^2+c\frac{x-\beta}{x}$|$1-\frac{2\beta c}{x^3}$|$1-\frac{2\beta c}{x^3}>0$|$c<\frac{\beta^2}{2}$| |$\beta>0$|$x>0$|$x<\beta$|$0.5\cdot\|x-\beta\|^2+c\frac{ \|x-\beta\| }{ \|x\| }$|$0.5\cdot(x-\beta)^2-c\frac{x-\beta}{x}$|$1+\frac{2\beta c}{x^3}$|$1+\frac{2\beta c}{x^3}>0$|$c>0$| |$\beta>0$|$x<0$|$x<\beta$|$0.5\cdot\|x-\beta\|^2+c\frac{ \|x-\beta\| }{ \|x\| }$|$0.5\cdot(x-\beta)^2+c\frac{x-\beta}{x}$|$1-\frac{2\beta c}{x^3}$|$1-\frac{2\beta c}{x^3}>0$|$c>0$| |$\beta<0$|$x>0$|$x>\beta$|$0.5\cdot\|x-\beta\|^2+c\frac{ \|x-\beta\| }{ \|x\| }$|$0.5\cdot(x-\beta)^2+c\frac{x-\beta}{x}$|$1-\frac{2\beta c}{x^3}$|$1-\frac{2\beta c}{x^3}>0$|$c>0$| |$\beta<0$|$x<0$|$x>\beta$|$0.5\cdot\|x-\beta\|^2+c\frac{ \|x-\beta\| }{ \|x\| }$|$0.5\cdot(x-\beta)^2-c\frac{x-\beta}{x}$|$1+\frac{2\beta c}{x^3}$|$1+\frac{2\beta c}{x^3}>0$|$c>0$| |$\beta<0$|$x<0$|$x<\beta$|$0.5\cdot\|x-\beta\|^2+c\frac{ \|x-\beta\| }{ \|x\| }$|$0.5\cdot(x-\beta)^2+c\frac{x-\beta}{x}$|$1-\frac{2\beta c}{x^3}$|$1-\frac{2\beta c}{x^3}>0$|$c<\frac{\beta^2}{2}$| |$\beta=0$|Any|-|$0.5\cdot\|x-\beta\|^2+c\frac{ \|x-\beta\| }{ \|x\| }$|$0.5\cdot(x-\beta)^2$|1|Always convex|Any $c$| Table R1: Conditions guaranteeing convexity and convergence of AWD for a simple regression. > Q2. flatness is a formulation based on perturbations in parameter space, while adversarial robustness is a formulation based on perturbations in input space. Thus, this is not a direct explanation. We would like to clarify this possible confusion that we did not mean that the adversarial training could yeild flatter minima, and we completely agree with the reviewer. Our response was in regards to the question about why in terms of adversarial robustness, AWD-trained models perform better compared to those trained with traditional weight decay. To elaborate further, we associate this to the model parameters/weights. As seen in Table 1 from the original paper (we have gathered the most relevant subset below in Table R2), AWD models have significantly smaller weight-norms compared to the best models trained with traditional weight decay. Models with smaller weight-norms are often simpler and could generalize better (as long as the model is still capable of fitting the data). |Dataset|Method|$\|W\|_2$| Nat | Adv| |:-:|:-:|:-:|:-:|:-:| |CIFAR-10|Weight-Decay|35.58|84.31|45.19| |CIFAR-10|AWD|**7.11**|**87.08**|**50.03**| |CIFAR-100|Weight-Decay|51.32|60.15|22.53| |CIFAR-100|AWD|**13.41**|**61.39**|**27.15**| |Tiny ImageNet|Weight-Decay|25.62|47.87|16.73| |Tiny ImageNet|AWD|**15.01**|**48.46**|**19.74**| |SVHN|Weight-Decay|102.11|92.04|44.16| |SVHN|AWD|**5.39**|**93.04**|**47.10**| |FashionMNIST|Weight-Decay|14.39|83.96|78.73| |FashionMNIST|AWD|**9.05**|**85.42**|**79.24**| |Flowers|Weight-Decay|19.94|**90.98**|32.35| |Flowers|AWD|**13.87**|90.39|**39.22**| Table R2: Relevant subset of columns from Table 1 which show that AWD trained models have considerably smaller weight-norms compared to models trained with traditional weight decay which results in simpler models which generalize better.

Reviewer c9pV2023-08-21

Response

I would like to thank the authors for their response. The above has addressed most of my concerns, and I have slightly increased my rating score.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

Source of record

arXiv

Improving Robustness with Adaptive Weight Decay

Paper

References (72)

Similar papers

Patents near this paper

Peer review

Welcome.

Patents near this paper