Wide Two-Layer Networks can Learn from Adversarial Perturbations

Adversarial examples have raised several open questions, such as why they can deceive classifiers and transfer between different models. A prevailing hypothesis to explain these phenomena suggests that adversarial perturbations appear as random noise but contain class-specific features. This hypothesis is supported by the success of perturbation learning, where classifiers trained solely on adversarial examples and the corresponding incorrect labels generalize well to correctly labeled test data. Although this hypothesis and perturbation learning are effective in explaining intriguing properties of adversarial examples, their solid theoretical foundation is limited. In this study, we theoretically explain the counterintuitive success of perturbation learning. We assume wide two-layer networks and the results hold for any data distribution. We prove that adversarial perturbations contain sufficient class-specific features for networks to generalize from them. Moreover, the predictions of classifiers trained on mislabeled adversarial examples coincide with those of classifiers trained on correctly labeled clean samples. The code is available at https://github.com/s-kumano/perturbation-learning.

Paper

Similar papers

Peer review

Reviewer fzgX5/10 · confidence 3/52024-07-04

Summary

In this paper, the authors theoretically investigate the perturbation learning phenomenon from a future hypothesis perspective. Perturbation learning means that a classifier can be learned from adversarial examples with incorrect labels (the labels used to generate adversarial examples, but seems to be incorrect in human eyes). Their theory includes two parts. First, they show that the adversarial perturbations are the linear combination of training samples, thus containing the information of the clean training dataset. This theorem provides an intuition for the feature hypothesis. Then, they show that, under some conditions, the predictions of a classifier trained on adversarial perturbations (or adversarial examples) are consistent with those of a classifier trained on a clean dataset. Some experiments on a synthetic Gaussian dataset are provided to show the effect of the input dimension and hidden dimension, validating their theorem.

Strengths

* Understanding perturbation learning and the feature hypothesis is quite important in the domain of adversarial learning. This paper provides a deeper understanding of this problem. * In this paper, the authors provide a more general theory about perturbation learning compared to prior work. The theory is based on fewer constraints in training data distribution, training time, etc.

Weaknesses

My main concern is whether the kernel regime is a suitable and extensible tool to study the feature hypothesis in adversarial training and explain other interesting phenomena such as the transferability of adversarial examples and the trade-off between robustness and accuracy. According to the feature hypothesis of adversarial training, the data contains a set of features that could used for prediction. Humans and all kinds of neural networks use different feature subsets to classify the image (the subsets have overlaps). The feature hypothesis may provide a **unified** explanation for several open problems in adversarial training. However, it seems that the theory in this paper only focuses on perturbation learning and is difficult to extend to explain other phenomena. In my opinion, the framework should have a deeper exploration of the feature subsets that are used by the trained models to make predictions.

Questions

Experimental Questions: * What is the effect of the sampling of adversarial labels on the accuracy of the model $g$? If we set the adversarial labels of all negative data to be positive and vice versa, will the accuracy of $g$ be very low? * What is the effect of the structures of $f$ and $g$? When these two networks have more divergent structures, I think the accuracy of $g$ must be lower. Is it correct in experiments? If it is, this is similar to the transferability of adversarial examples, i.e., the adversarial examples are easier to transfer to similar networks. Theoretical Questions: * What do the functions $\hat{f}$ and $\hat{g}$ mean? What are the relationships between $f$ and $\hat{f}$, $g$ and $\hat{g}$?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors state the limitation on the assumption of the width of the network in the paper.

Reviewer 5BXU5/10 · confidence 4/52024-07-17

Summary

This work aims to provide an alternative theoretical analysis to justify feature hypothesis and perturbation learning. The analysis is based on approximation theory in the kernel regime (i.e., infinite width). They show that the adversarial perturbation contains sufficient data information, which can be retrieved by the perturbation learning when y labels are uniformly sampled.

Strengths

The paper is theoretically sound and relaxed certain conditions required by prior works.

Weaknesses

What is the contribution beyond technical novelty? Are any new insights obtained via the new analysis compared to existing results? The conditions in the main theorems lack interpretability. I can understand the technical reason behind them, but how reasonable are these conditions, especially the agreement condition? The author argues that the agreement condition depends on the consistency of the correlation between z and y_nx_n. This statement is not rigorous, since z is a function argument, but not a r.v. and there is no correlation involved. I suppose a more meaningful question will be, if z follows the data distribution of x (e.g., a mixture of two Gaussian), what is the chance these conditions hold?

Questions

See weakness.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. The limitation is clearly statement in the paper.

Reviewer MQtT5/10 · confidence 3/52024-07-19

Summary

Perturbation learning, where classifiers are trained on adversarial examples with their associated incorrect labels, results in non-trivial generalization. This work theoretically tackles the perplexing former phenomenon for wide two-layer networks in the kernel regime. The authors first prove that adversarial perturbations are parallel to a meaningful feature vector (that can contain whole dataset information under some additional assumptions) up to an error term. Second, the authors provide some conditions on when the predictions with perturbation learning match those of standard learning.

Strengths

1. The problem is a very interesting one and lacks more analysis, so this paper addresses a significant problem in an original way. 2. The setting and assumptions are very well described and clear.

Weaknesses

1. I am overall confused about the choice of kernel regime to explain perturbation learning. As the authors acknowledge, there is no feature learning for the choice of width in this paper (the output of hidden units remains the same). It's not clear how such a framework can explain perturbation learning and, in particular, the "feature hypothesis," which authors claim that they do. Since there is no feature learning in this regime, there should not be any "feature hypothesis." This is not to say the analysis is uninteresting; it applies to perturbation learning with kernels. But this requires a completely different contextualization than the authors provide, one that focuses on the adversarial robustness of kernels rather than neural networks. And for the reasons above, I find the comparison with prior work a bit misleading as they tackle perturbation learning for different scenarios. 2. It is difficult to judge the validity of the assumptions in the paper. * Assumption 3.2. is very convoluted, and I don't see how one can justify this assumption in finite width settings. I equate this assumption to an infinite-width assumption, which is perfectly reasonable, but as I discussed above, I believe it changes the object of study. * The assumptions of Theorem 3.3. and Theorem 3.4. are discussed, and some intuitions are provided. But it is not discussed why they would hold for a small $\delta$. 3. The $\delta$ dependencies in Theorems 3.3, 3.4, and 3.5 are confusing. Let's focus only on Theorem 3.3. When we take $\delta \to 0$ while everything else remains fixed, the statement is trivial. So, the interesting part of the statement is when $T_f$ or $d$ scales with $\delta$. I believe it is more reasonable to consider $T_f \to \infty$. So, this verifies that Assumption 3.2. is an infinite width assumption.

Questions

1. Could you discuss again the choice of kernel regime vs. feature learning for your analysis after my comments in the Weaknesses section? 2. Could you provide references or evidence towards why $\epsilon$ should scale with $1/\sqrt{d}$?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

I think the authors overclaim by saying, "We provided a theoretical justification for perturbation learning and the feature hypothesis" (L293). Their analysis is limited to the kernel regime, which is acknowledged, but its implications for the feature hypothesis are not well explained. In addition, scaling with $\delta$ in main theorems is not discussed, and as I pointed out, it points to other limitations towards infinite width.

Reviewer MQtT2024-08-10

Thanks for your detailed explanations. I do agree with both of your claims in a) and c), and I did not claim otherwise in my review. I will detail what I meant in my original review. **Assumption 3.2.** It is true that Assumption 3.2. is satisfied by some $m$ for any $d$ and $T$ (let's grant that $\ell'$ is bounded). I do not claim otherwise. But I do not see why the finite $m$ regime satisfied by Assumption 3.2. is **any more** interesting than infinite $m$. There is nothing in the practice of ML that would make me think Assumption 3.2. is more reasonable than the infinite $m$ assumption. Again, there is nothing wrong with assuming infinite $m$: it is extremely hard to do good theory, and kernel regime is only one of the few approaches available. **Assumptions in Theorems 3.4., 3.5.** Let $\delta$ be a fixed small number. For any $z$, there are two competing conditions on $T_f$ and $T_g$ now: First, you assume them to be $\Theta(1)$. Then, you also assume the margin conditions, which depend on $T_f$ and $T_g$. There is no discussion on why these margin conditions would hold for the regime imposed on $T_f$ and $T_g$. In addition, there are remainder terms in margin conditions even if $T_f$ and $T_g$ go to infinity, which are not discussed. **Scaling with $\delta$** Theorems 3.4. and 3.5. have $\delta$ dependency only in multiplication with $T_f$ and $T_g$. For a choice of confidence level $\delta$, $T_f$ and $T_g$ will need to be larger than some quantity that depends inversely on $\delta$. This means that $m$ itself has to be larger than some quantity that depends on $\delta$. This is further evidence, along with Assumption 3.2., that authors need an infinite-width approximation, and the result holds with probability degrading depending on the validity of this approximation. This should not come across as a surprise to the authors: this is the core gist of the kernel regime. All this considered, my main point is the fact that for some confidence levels, there are large numbers of $m$ verifying the assumptions of the paper, is **not** more interesting than an infinite $m$ assumption. **Network parameters change during the training.** From now on, I will only consider the $m \to \infty$ limit. As the Kernel Regime paragraph details in the Sketch of Proof section, the outputs of hidden neurons change negligibly. This is the basis of my first objection to the kernel regime and how it can be used to explain perturbation learning. In this regime, one cannot claim feature learning in the usual sense. The final output changes, but this does not imply that the network has learned specific features. My understanding of perturbation learning was that it is equal to "learning" the features present in adversarial perturbations, the features that are claimed to exist by "feature hypothesis." And since there is no actual feature learning happening in the kernel regime, this cannot be studied in such a framework. However, the authors use a more nuanced definition, and more specifically, the word "enable," i.e., indicating that the features in adversarial perturbations can somehow help the classifiers (in contrast to the case where classifiers simply learn these features). This is exactly why I brought up the kernels, which would have no learning of features but alignment of the features such that they align with these features. **Final conclusion** I will increase my score to 4 from 3, as my initial judgment on the relationship between perturbation learning and the kernel regime was harsh. The authors do not explicitly claim features posited by "feature hypothesis" are learned by wide, two-layer networks. I am willing to increase my score to 5 from 4 if the authors agree to incorporate i) a discussion on the kernel regime and what it can model in neural networks in the related work section, ii) more nuanced discussions on the limitations of their results, including the ones I have highlighted in the text.

Authorsrebuttal2024-08-11

Official Comment by Authors (1/2)

We sincerely appreciate the reviewer's prompt response and the effort they put into engaging with the discussion. First reply: We will first address the two conditions that the reviewer has set for increasing the score. Following that, we will summarize our perspective on the generality of finite width, which may be a point of misunderstanding between us and the reviewer. Second reply: We will provide responses to the specific questions raised by the reviewer. **The reviewer's suggestion (important)** We address the two conditions that the reviewer required for raising the score as follows: (i) We will include studies related to the kernel regime, such as [14], in the related work section. Rather than focusing on studies related to training convergence, we believe it would be more effective to reference works like [8], which first discussed the invariance of hidden units, and [1] (see below), which uses the kernel regime to analyze the properties of adversarial examples. (ii) In the limitations section, we will acknowledge that the "features" we focused on are primitive, and higher-level features potentially included in perturbations and learning from them may not be theoretically clear (cf. (III)). If there are any discrepancies between the reviewer's requests and our understanding, we would be happy to address them. [1] H. Zhang et al. Adversarial Noises Are Linearly Separable for (Nearly) Random Neural Networks. AISTATS23. **Generality of finite width (important)** Furthermore, we sincerely appreciate the reviewer's thoughtful suggestion for an additional discussion, and we will certainly incorporate it into the main text (please see the elaboration at the end of this reply). However, we believe that a critical point has been missed by the reviewer: one of our main contributions is the proposal of a general theory that encompasses both finite and infinite-width cases. While the reviewer has interpreted our work with $m \to \infty$, it is important to note that the finite case is at the very core of this research. The reviewer may argue that the finite case is neither more interesting nor more practical than the infinite case, but we respectfully and strongly object to this. a) First of all, all networks that can be realized in practice are constrained by finite width. Although the assumption of infinite width is commonly used for theoretical convenience, the finiteness of width is indeed significant in practical machine learning. For instance, if we had infinitely many samples, there would be no generalization gap; if we had infinitely long training time, simulated annealing would discover a global optimum with probability one. b) Second, by deriving conditions that depend on specific variables rather than assuming infinity, we can understand how assumptions about width change. For example, in this study, we have $m > O(d^2T^2)$, which suggests that our assumptions become stricter as they are proportionate to squared $d$ and $T$. This insight is not evident under the naive assumption of $m \to \infty$.

Authorsrebuttal2024-08-11

Official Comment by Authors (2/2)

**(I) Assumption 3.2 and scaling with $\delta$.** > (scaling with $\delta$) Theorems 3.4. ... on $\delta$. > the result holds with probability degrading depending on the validity of this approximation First, these are correct. > (in "scaling with $\delta$") for some confidence levels, there are large numbers of $m$ verifying the assumptions of the paper, is not more interesting than an infinite $m$ assumption. > (in "Assumption 3.2") I do not see why the finite $m$ regime satisfied by Assumption 3.2. is **any more** interesting than infinite $m$. As the reviewer pointed out, for some confidence levels, there are infinitely many values of $m$ that satisfy the conditions, naturally including infinite width. However, this does not imply that we must choose infinite width. We can select the smallest $m$ that satisfies the conditions. Moreover, we do not claim that the finite width regime is more *interesting* than infinite width. The finite width is discussed in this paper because it is not theoretically necessary to consider infinite width. There may be some misunderstanding between us and the reviewer regarding the term "interesting." If our response does not address the reviewer's concerns, we would appreciate further clarification. > There is nothing in the practice of ML that would make me think Assumption 3.2. is more reasonable than the infinite $m$ assumption. We have interpreted this concern as follows: *In practice, the experimenter does not know in advance how large $T$ (training time) needs to be for perturbation learning to succeed. If the experimenter continues training until perturbation learning succeeds, $T$ could become very large. Therefore, the experimenter may need to select a very large $m$ in advance, which is essentially equivalent to assuming infinite width.* This concern is valid when $T$ is variable during training and unknown in advance, and the experimenter is free to choose an arbitrarily large value. However, it should be noted that our theory does not claim how the training evolves over time (i.e., $T$; variable). It characterizes the model trained with a designated setup (including the training time $T$; constant). In other words, we fix $T$ before training, as is typical in actual network training. For example, it is natural to set $T$ in advance to 100 epochs for MNIST. The experimenter selects a constant $T$ (although it does not always lead to the success of learning) and, accordingly, a finite $m$, which is more reasonable than an infinite width assumption. **(II) Assumptions in Theorems 3.4., 3.5.** > there are two competing conditions on $T_f$ and $T_g$ ... The reviewer might consider that $T$ is substantially constrained to $c < T < C$ for some $c,C$, which prevents the satisfaction of the functional margin conditions. We should note that the assumption $T = \Theta(1)$ is introduced only for notational simplicity of the functional margin conditions. Essentially, no assumption on $T$ without $T > 0$ is required. Simply speaking, this assumption is introduced to derive $O(T + (1/\sqrt{d})) = O(T)$. Although not realistic, if $T = \Theta(1/d)$, then $O(T + (1/\sqrt{d})) = O(1/\sqrt{d})$. Without any assumptions, we would need to write unnecessarily intricate conditions, which we believe should not be included in the main text. We will revise any misleading assumptions. > there are remainder terms ... which are not discussed. These time-independent terms are discussed in Lines 179--181. **(III) Network parameters change during the training.** The reviewer's understanding is generally correct. The reviewer seems view "feature learning" as the extraction of higher-level features that are potentially latent within the data. In other words, they interpret features not as raw data vectors or their parts but as more complex combinations of these elements. According to the reviewer, kernel methods and our framework (cf. Eq. (22) and (23)) perform "feature alignment" with raw data rather than "feature learning" as a higher-level feature extraction. I partially agree with this assertion. The difference in intepretation between us and the reviewer seems to be in the scope of what is referred to as "features." In addition to these high-level features, we also consider the data vectors themselves as features and regard their simple extraction as a form of "learning." For example, in the binary classification of vertical and horizontal lines or in MNIST, the raw data itself would serve as features. However, we will revise any misleading terms that may lead to misinterpretation. Furthermore, we will explicitly state in the limitations section that we prove the feature hypothesis and perturbation learning from primitive feature perspective, but do not fully explain them with higher-level features. Note that prior work shares the same limitation, and the extraction of higher-level features might require relaxing constraints related to depth rather than width.

© 2026 NYSGPT2525 LLC