Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness

Summary

This paper presents a theoretical characterization and some theorems supporting previous empirical finidings on perceptually aligned gradients (PAG) of classification neural network models. Specifically, it provides the first rigorous definitions for the previously qualitative definitions of "PAG", and provides the first attempt at a theoretical framework connecting PAG and model robustness, which has been observed empricially in previous work.

Strengths

- Very well-written and well-presented paper. Covers and taxonomizes the relevant literature very nicely. - Math and definitions are intuitive, relevant, and sound. I especially like the fact that the theorem assumes almost nothing on the model behavior or training objective. - Overall, this paper addresses a very important and relevant topic, characterizes a lot of themes in it, and provides a great theoretical explanation for it. This theoretical justification was notably missing from previous PAG literature. - Empirical evaluation metrics are carefully picked and well-thought-out. Comparing PAG to score-based models is especially relevant (this was first suggested in [6], I think a reference is due on lines 264-274).

Weaknesses

- lines 106-107: noise can also have both on- and off- manifold elements (think of a linear combination of two on- and off- manifold noise vectors). Therefore, the "otherwise" on line 107 is incorrect. - Minor issue: Defnition 1 is missing a definition of x (for formality). - While the math (theorem 1) is nice, it does not consider adversarially chosen noise. It would be nice to mention that for normal noise with $\sigma \rightarrow 0$, this covers adversarial noise as well, but it should be made clear that more work is needed to specifically consider adversarial attacks. - Line 151: Paragraph title mentions difference between data and signal manifolds, but the paragraph does not discuss "data manifold". Typo? - The definition of signal vs distractor is too simplistic. Signals and distractors are not necessarily separated by a pixel mask only. The underlying signal manifold could lie on a linear or non-linear projection of x, not necessarily on the pixel space itself. This weakens the resulting math (theorem 2), unless the authors can rewrite their proofs w.r.t. a determinstic "signal" function that is generalized beyond simple point-wise masking.

Questions

- In the abstract, is noise augmentation and randomized smoothing the same thing? If so, I would suggest listing them as one item and not two. - Lines 272-274: Is the score-based model used here class-conditional? If so, it should be made clear by defining p(x|y) and not p(x) in the score function definitions. If not, further explanation is needed. Why would we characterize perceptual alignment using a class-unaware model? "PAG" usually refers to salient class-specific features.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes

Summary

This paper studies Perceptually Aligned Gradients (PAGs), a phenomenon where the input gradients are semantically meaningful and aligned with human perception. While this trait gained research attention, we do not truly understand it. To this end, the paper proposes an explanation via off-manifold robustness. Namely, they show empirically and theoretically that models that are off-manifold robust have perceptually aligned gradients for different robustification techniques. Moreover, they propose a quantitative way to assess PAG, while prior to this work, PAG was evaluated qualitatively only. In addition, they identify three levels of robustness and analyze them in terms of PAG.

Strengths

* Perceptually Aligned Gradients is a fascinating phenomenon, and despite gaining substantial research attention, we do not understand it. Shedding light on this trait is an interesting and important research goal. The paper empirically and theoretically verifies the relationship between PAG and off-manifold robustness. To strengthen their findings, they do so for different robustification techniques. * Decomposing the gradients of a model to on and off-manifold directions is very interesting thinking. Besides being theoretically logical, they also develop a way of doing so. * Excellent and clear writing. The authors have done a good job of providing motivation for this work. In addition, while several papers use this term to describe different behaviors, this paper formulates the definitions and makes an order in the various definitions.

Weaknesses

* The connection between a conditional score function and PAG was made in [1]. Despite not being used for assessing PAG, they show that a classifier trained to mimic the score function possesses PAG. Thus, I do not find linking the two for assessing PAG very novel. * Overstatements and bit trivial claims - “In this work, we provide 7 a first explanation of PAGs” [abstract], but [4] offers an explanation of the generative capabilities of adversarial training using energy-based models. Additionally, I find some claims that the authors made efforts to explain quite straightforward. For example, robust models are off-manifold robust. We know robust models have PAG, meaning their gradients are aligned on the manifold. This means that the change of loss in off manifold direction is smaller than the on-manifold (because otherwise, the gradients weren’t aligned), making such models off-manifold robust. **Minor weaknesses and requests** * I find this important to report the LPIPS results in Figure 5 to demonstrate that the used metric indeed makes sense and aligns with the human evaluation of PAG. * Missing citations - I think that in order to motivate the significance of studying PAG for the general audience, it is important to include more works that study and utilize PAG. For instance, [2,3]. [1] Do Perceptually Aligned Gradients Imply Robustness? [2] On the benefits of models with perceptually-aligned gradients. [3] BIGRoC: Boosting Image Generation via a Robust Classifier. [4] Towards Understanding the Generative Capability of Adversarially Robust Classifiers.

Questions

* Is the score function being modeled in the measuring Perceptual Alignment paragraph for clean images or noisy images? From my understanding, diffusion networks model the noisy score function and not the clean one. * What is the rationale for measuring the LPIPS for gradients? It is usually used for images, and input-gradient distribution is significantly different. Why does it make sense to use this rather than cosine similarity or other closed-form metrics? * Figure 2 shows that there are points where models are good in off-manifold robustness before being on-manifold robust (for example, the left figure in the top row of Figure 2, around a noise level of 0.5). Can this analysis help us find robust models that have better robustness-accuracy tradeoffs? * Why are the results of Imagenet 64 so low? In general, I find this paper interesting and would like to increase my score given the authors' response.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have adequately addressed the limitations.

Summary

The paper seeks to provide a condition that leads models to have gradients that are aligned with human perception, i.e. which highlight relevant features of the image while ignoring distractor features. The paper first proposes that for any manifold, if a model is robust to perturbations which are off-manifold, then its gradients will be aligned with the manifold. The paper then defines a "signal" manifold as a hypothetical manifold of features which relevant to the classification task, defined as the multiplication of the data with a masking function. The paper then states and proves that a Bayes optimal classifier is robust to perturbations perpendicular to this manifold, and hence its gradients lie on the signal manifold. The paper further hypothesizes that gradients on the signal manifold are perceptually aligned. An argument is then provided that models robust to adversarial perturbations which have similar off-manifold robustness as the Bayes optimal classifier have perceptually aligned gradients. Finally, the paper presents empirical evidence in support of their theoretical claims and hypotheses.

Strengths

1. The paper presents an interesting hypothesis and designs a robust experimental methodology to validate it within the set of assumptions. 2. The notion of weakly robust, optimally robust and overly robust models and their connection to PAGs is novel, and interesting. Comparing such classifiers with score functions is interesting in its own right, and can be a good tool in analysing models empirically.

Weaknesses

1. It is not clear from the experiments what the signal manifold is. Line 278 talks about perturbing about 10\% of the input, but it would be good to clarify this further. 2. Hypothesis 1 is never validated empirically in the paper, but the experiments build upon this hypothesis by comparing the perceptual similarity with the gradients of the score function.

Questions

1. How are on- and off-manifold perturbations generated exactly? 2. Is there any way to provide more empirical evidence for Hypothesis 1?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors address their limitations in sections 3 and 4, where they describe how much of an approximation their empirical setup and theorems are.

Summary

This paper explores the connection between manifold alignment, perceptually-aligned gradients, and model robustness, deriving a number of novel results that explain previously observed phenomena in the explainability and robustness literatures. Crucially the paper grounds its contributions in Bayes optimal classifiers, arguing that the gradients of such classifiers are aligned with a “signal manifold”. The paper leverages denoising autoencoders to estimate the data manifold and diffusion models to estimate the gradients of a Bayes optimal classifier. Using these quantities, the paper demonstrates a number of novel results, including a strong relationship between manifold alignment and model accuracy.

Strengths

This paper presents a timely and likely impactful theoretical contribution with implications not only for robust learning but also for explainability and generalization. The paper is very well organized and well-written. The introduction provides an excellent summary of the literature and current open questions. The paper disambiguates between three closely-related notions of “perceptually-aligned gradients” and provides a powerful taxonomy through the definition of Phenomena 1-3. Drawing on prior work, the paper introduces quantitative metrics to evaluate robustness and gradient alignment with respect to a data manifold. The theoretical contributions are somewhat geometrically simplistic, but powerful. Theorem 1 has been assumed intuitively in prior work, but this paper introduces the formalisms to quantify the connection between off-manifold robustness and on-manifold gradient alignment. Argument 1-2 outline very compelling circumstantial evidence for the broader Hypotheses and will hopefully lead to future theoretical developments. The paper provides extensive experimental evidence, including useful error bars. Figure 2 provides gorgeous plots and the correlation between manifold alignment and model accuracy is striking. Finally, the paper makes a number of practical advances related to effective adversarial training. Figure 5 intuitively summarizes the paper’s crucial claims about three regimes of robust training.

Weaknesses

* It does not seem right that the mask defining distractor vs signal features is constrained to be binary (Definition 3). This means the signal and distractor distributions are not invariant to linear transformations of $x$. Shouldn’t the only requirement be the statistical independence on line 163? * All experiments evaluating perceptual alignment used the LPIPS metric. Wouldn’t another metric, such as cosine distance, be appropriate. This would allow for distractors that aren’t defined by a binary mask. * Unfortunately (probably due to computational cost) different results are presented for CIFAR-10 and ImageNet, reducing the global consistency of the paper. * The experiment shown in Figure 4 is not well explained, would it be possible to explain the setup in a little more detail using previously-established notation? Small issues: * The definition of Bayes optimal classifiers assumes an equal class distribution (i.e. uniform p(y)). This most likely will not be the case in most applications. Note that this does not invalidate any of the paper’s findings. * Type on line 200 “Bayes optimal models gradients” * Missing article on line 220: “there exists a trade-off between [the] cross-entropy loss and on-manifold robustness term” * Missing pronoun at line 351: “meaningful in that [they] lie tangent” * At line 354, “gradients is repeated, and do you mean the double negative “cannot be uninterpretable”? * Figures 4 and 5 are out of order. * The numbering of Theorems in the Supplemental material is off (the proofs continue as Theorem 3, 4 rather than 1, 2).

Questions

1. Why is Theorem 2 stated with an if and only if? Don’t you prove that both statements are true for Bayes optimal classifiers? 2. Would it be possible to add an adversarially-trained model for CIFAR-10? 3. On line 228 you say “We elaborate on this argument in the Supplementary material.” Where? 4. On line 255, does “$l_2$-adersarial robust training” mean PGD applied during training? 5. In the legend for Figure 2, is “perturbation” really the right term? Isn’t the plot essentially showing the inverse of robustness? Why not use the relative robustness metric defined in Definition 1? Why is an $l_1$ divergence used here rather than the $l_2$ divergence used in Definiton 1? Can $\rho_1$ here be connected to Brier score to provide a proper score function? 6. Are error bars standard error? 7. Do you have any comment on the point where the lines cross in Figure 2 and why there is a different trade off depending on the training objective? 8. At line 274, how were these specific $\sigma$ values chosen for each dataset to parameterize the diffusion model?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The acknowledged limitations throughout the paper and the supplemental material are superb (e.g. the statement “We provide two lines of argument in support for this hypothesis [Hypothesis 2]. Ultimately, however, our evidence is empirical”).

Summary

The paper titled "Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness" investigates the phenomenon of perceptually-aligned gradients (PAGs) in robust computer vision models. PAGs refer to the alignment of model gradients with human perception, enabling these models to exhibit generative capabilities such as image generation, denoising, and in-painting. The paper aims to explain the underlying mechanisms behind PAGs and provides insights into the relationship between off-manifold robustness, gradient alignment, and model accuracy. The paper's contributions can be summarized as follows: 1. Theoretical Explanation: The paper presents a theoretical explanation of PAGs through the concept of off-manifold robustness. It demonstrates that the alignment of gradients with the data manifold is equivalent to off-manifold robustness. It also introduces the distinction between the data manifold and the signal manifold, which helps in understanding the input gradients of robust discriminative models. 2. Connection to Bayes Optimal Models: The paper establishes a connection between Bayes optimal models and off-manifold robustness. It shows that Bayes optimal models achieve both off-manifold robustness and on-manifold gradient alignment. The input gradients of Bayes optimal classifiers lie on the signal manifold, indicating their perceptual alignment. 3. Empirical Analysis: Extensive empirical analysis is conducted to validate the theoretical findings. Robust models trained with various techniques are evaluated on different datasets. The experiments confirm the relative off-manifold robustness of robust models, the correlation between off-manifold robustness and perceptual alignment, and the presence of signal-distractor decomposition in robust models. The paper's main contribution lies in providing a theoretical framework and empirical evidence for understanding the mechanisms behind PAGs in robust computer vision models. It sheds light on the importance of off-manifold robustness and its connection to gradient alignment and model accuracy. The findings have implications for developing more explainable and generative models in computer vision tasks. By identifying different regimes of robustness, the paper also calls for rethinking standard robustness objectives and benchmarks. Overall, the paper advances our understanding of the properties and behavior of robust models, paving the way for further research in improving model interpretability and generative capabilities in the field of computer vision.

Strengths

The strength of the paper lies in its combination of theoretical analysis and empirical evaluation. It provides a comprehensive examination of the phenomenon of perceptually-aligned gradients (PAGs) in robust computer vision models. By presenting a theoretical framework based on off-manifold robustness and its connection to gradient alignment, the paper offers a solid foundation for understanding the underlying mechanisms behind PAGs. Furthermore, the empirical analysis conducted in the paper strengthens its findings. The authors perform extensive experiments using different datasets and robust training techniques, validating their theoretical explanations and hypotheses. The empirical results consistently support the theoretical claims, providing evidence for the presence of off-manifold robustness, gradient alignment, and signal-distractor decomposition in robust models. The paper's contribution also extends to the exploration of various aspects related to PAGs, such as the connection to Bayes optimal models, the trade-off between on- and off-manifold robustness, and the correlation between off-manifold robustness and perceptual alignment. These insights broaden our understanding of the behavior and properties of robust models in computer vision tasks. Overall, the strength of the paper lies in its rigorous analysis, clear explanations, and compelling empirical evidence, making a significant contribution to the field of computer vision and model interpretability.

Weaknesses

One potential weakness of the paper is that the evaluation of on- and off-manifold perturbations does not cover the entire space of perturbations. There is a significant portion of perturbations that fall in-between the two categories, which are not thoroughly explored in the current research. It would be valuable for future work to address these intermediate cases and provide a more comprehensive understanding of the model's behavior in those scenarios.

Questions

Clarification on the choice of perturbations: Can the authors explain the rationale behind the specific choice of on- and off-manifold perturbations? How representative are these perturbations of real-world scenarios? Are there other types of perturbations that could be considered in future research? Extension to non-linear models: The paper primarily focuses on linear models and their relationship to off-manifold robustness. Can the authors discuss how their findings and hypotheses can be extended to non-linear models? Are there any unique considerations or challenges in analyzing non-linear models in this context?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

- Limited exploration of perturbation space: The choice of on- and off-manifold perturbations in the experiments might not cover the entire space of possible perturbations. There could be a wide range of perturbations that lie between the two extremes. Exploring a more comprehensive set of perturbations and their effects on on- and off-manifold robustness could enhance the depth of the analysis. - Evaluation metrics: The paper primarily focuses on measuring robustness based on the change in model output. While this is a common approach, it may not capture all aspects of robustness. Considering additional evaluation metrics, such as adversarial examples or other robustness measures, could provide a more comprehensive understanding of model behavior.

Response

I thank the authors for their rebuttal and maintain my score recommending acceptance.

Reviewer respone

I thank the authors for their response which addresses my questions and raised points. As I find the paper very interesting, I am happy to raise my score.

Additional clarification

Thank you to the authors for a thorough and insightful rebuttal. I have a couple remaining questions about the experimental setup based on diffusion models. 1. You say that you “choose the parameter sigma to maximize the perceptual alignment of the score”: does this mean that this parameter was based on a human visual analysis or a metrics such as LPIPS? Do you have any anecdotal evidence pertaining to the importance of the $\sigma$ parameter, e.g. confirming the need for better diffusion models to estimate the score function [R1]? 2. Additionally in your response to Reviewer eaFT, you state that “The diffusion model is unconditional.” On line 271, you are aiming to estimate $\nabla_x \log p(x | y)$: this requires a class-conditional model [R2, eq 7]. What am I missing? 3. I believe that this concern in (2) directly connects to my question about metrics (#2 in your rebuttal). My understanding was that your experiments with LPIPS were using the gradients of a Bayes optimal classifier provided by a conditional diffusion model. However in your response to Reviewer aEc6, you say that “A potential reason for this might be that the cosine similarity is rather sensitive to the fact that the score aligns with the data manifold, whereas model gradients align with the signal manifold.” I expected cosine similarity between the gradients of a Bayes optimal classifier and the gradients of a discriminative model to estimate precisely the _projection of the discriminative model’s gradient on the signal manifold_. Is this interpretation inconsistent with your experimental setup? 4. Finally, can you confirm that this comment will be addressed in the revisions? > The definition of Bayes optimal classifiers assumes an equal class distribution (i.e. uniform p(y)). This most likely will not be the case in most applications. [R1] Wang, Zekai, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Y. A. N. Shuicheng. "Better Diffusion Models Further Improve Adversarial Training." ICML (2023). [R2] Ganz, Roy, Bahjat Kawar, and Michael Elad. "Do Perceptually Aligned Gradients Imply Robustness?." ICML (2023).