Unveiling Transformer Perception by Exploring Input Manifolds

Summary

The authors propose a mathematically rigorous methodology based on Riemannian geometry for attributing network importance of tokens in a transformer models input space (e.g. image patches, or ~words in the textual domain). The proposed methodology—whilst based on sound theory—translates into an intuitive algorithm involving what appears to be a relatively inexpensive eigendecomposition. Experiments on 3 datasets across both the image and NLP domains explore how the features correlate with ground-truth inputs in the text domain, in addition to first steps towards exploring how the features affect the networks’ output logits.

Strengths

- A major strength of the paper is the mathematically solid approach in attempting to identify regions of the **input** space that explain transformers’ model decisions. This is an important area of study: in contrast to many recent mechanistic interpretability methods finding latent network representations (that are intrinsically hard for humans to interpret by default), salient features in the pixel/text input space are much more readily interpreted by humans. - Whilst I am unfamiliar with geometric deep learning, the authors do a fantastic job of presenting the technical content in a digestible manner without sacrificing depth or rigor.

Weaknesses

# [W1] Feature importance comparisons Feature importance-based explanations are motivated on [L303] as quantifying the contribution of features `"to a model prediction"`. More concretely, around [L180], the authors motivate the eigenvalues of the pullback metric found using their method as ultimately deducing the importance of each segment (e.g. image patch) `“with respect to the the final prediction”`. Consequently, a major weakness of the paper is how there is no comparison with related work for how well the proposed method’s identified important features alter the **output** logits (e.g. upon ablation). I am slightly confused by why the authors did not adopt the established “perturbation test” experimental protocol in the baseline [1] against which they compared, to provide experimental evidence in favor of this. Currently, the only comparisons made around [L274] measure the features’ importance as they correlate to the *input’s* labels. Concretely, the authors could, for example, ablate particular patches of MNIST and observe that the resulting performance drops correlate with the pullback metric’s eigenvalues. This would provide stronger evidence of the authors’ claims about the features affecting the networks’ output, and (crucially) ground the results in contrast to those achievable by existing methods. # [W2] Limited experimental results & improvements There is a lot of interesting theory here, but ultimately this is a paper with a concrete applied goal of feature attribution in transformer models. With such a new methodology with many technical details, I believe there is an extra burden of proof on the authors to demonstrate this somehow leads to additional insights / practical gains. As such, it is a relative weakness of the paper that so few experiments are performed to justify the methodology. Beyond toy datasets, it would be interesting to see how the method performs on more complex ones (not necessarily larger ones), such as TinyImageNET. Here, we could visualize much more easily if the method helps identify salient features of animals’ body parts (for example) as being important features for classification. MNIST experiments alone in the image domain are hard to interpret given the similarity of all the input data. Furthermore, the method provides an almost insignificant increase of just `0.07` cosine similarity (over the baseline in [1]), on just a single dataset (and with just two baselines—for example, how does GradCAM perform here?). This is not sufficient evidence to convince me as a reader that the proposed methodology should be adopted. --- - [1]: Chefer, Hila et al. “Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers.” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021): 387-396.

Questions

No additional questions at this time (beyond those alluded to in the weaknesses section).

Rating

4

Confidence

2

Soundness

3

Presentation

4

Contribution

3

Limitations

Some limitations are indeed addressed throughout. However, (unless I have missed something, in which case I apologise!) I can only find the limitations of the small number of datasets used stated in the NeurIPS checklist. This needs to be stated explicitly in the main paper.

Summary

This work attempts to find the set of inputs that generate the same neural network predictions. To this end, the authors interpret the layers of the network as transformations of the input manifold. This interpretation is used to defined equivalence classes over the inputs and to define feature importance. Finally, the tools are used to identify equivalence classes for MNIST digits and for hate speech detection with BERT.

Strengths

Section 2 does a thorough job of introducing a manifold interpretation to neural networks. This introduction is then used to motivate multiple algorithms for finding equivalence classes of the inputs---or the setup of inputs that result in the same prediction---and identify features that are important.

Weaknesses

The main contribution of this work is to introduce a tool for analyzing which set of inputs produce the same output. However, this is exactly the Fisher information matrix (with respect to the inputs) and has also been introduced in prior work (https://arxiv.org/abs/2104.13289). Could the authors clarify what the differences are and what the additional novelty. If the "local data matrix" introduced in https://arxiv.org/abs/2104.13289 is identical to the tools in this work, I think it severely diminishes the contributions of this work. Furthermore the experiments are extremely similar (such as Figure 1). The second weakness is the limited number of experiments. The work does not show any quantitative results: Figure 1, Figure 2 and Figure 3 is just 1 example and is not indicative of why the tools are useful. The experiments in section 4 primarily discuss wall-clock time. It would significantly help if claims such as "(Line 266) we notice that the perturbation-based algorithm ends up producing monochrome ..." are substantiated quantitatively. Overall, the work doesn't provide novel tools and the experiments lack a novel usage of these tools and do not reveal any new insights.

Questions

1. How does this work differ from https://arxiv.org/abs/2104.13289? 2. What is algorithm 4 (exploration) used for? 3. Many experimental details are lacking. It would help to add appendices clarifying what networks were used and how they were trained. Overall, the work feels incomplete in many respects 4. For feature importance-based explanations, there are many other methods like LIME/SHAP (and many more follow up papers) which the authors do not compare to. Why were they omitted? 5. Does this method scale to larger datasets, considering we have to compute outer-product of the gradients and compute the Eigenvalues of this matrix?

Rating

3

Confidence

4

Soundness

2

Presentation

3

Contribution

1

Limitations

The authors address limitations of their work but it can be expanded upon. For example, the authors can discuss the time required to compute Eigenvalues, and other limitations such as not having any Eigenvalues to be 0 in Algorithms 3 / 4. Furthermore, their algorithm should work (in theory) for infinitesimal steps in the input manifold.

Summary

The authors present a method for exploring equivalence classes in the input space of Transformer models using a solid mathematical theory. By analyzing the Jacobian of the model, the method reconstructs and navigates these classes, offering a powerful tool for understanding Transformer interpretations and enhancing explainability. The proposed method is expected to solve problems in Computer Vision and Natural Language Processing tasks.

Strengths

I write both strength and weakness. First, I must disclose that I have no prior study or background in both Transformers and manifolds. While I am conceptually aware of them, my knowledge is limited to that extent, and I lack confidence in reviewing the technical details. Therefore, please consider my review comments as feedback from a layperson in this field, focusing on the overall mathematical consistency and readability of the paper. This paper describes mathematics in a clear and understandable manner that even a layperson like myself can grasp. Each definition and theorem is stated accurately, and I believe that the general concepts can be understood with basic knowledge. I personally feel that the objective of this paper is not clearly conveyed. While the paper claims to contribute to explainability and sensitivity analysis through the analysis of input manifolds, the logic behind this was not clear to me in the Introduction and Preliminaries. Although the concepts of explainability and sensitivity analysis become clearer in the later chapters, it might be beneficial to provide a bit more explanation in the Introduction. Additionally, it might be helpful to clearly define the equivalence class mathematically. Since I am not familiar with the existing literature, I was unable to judge the novelty of this work.

Weaknesses

See above.

Questions

See above.

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

N/A.

Summary

This paper develops a novel theoretical framework grounded in Riemannian geometry for analyzing the input space of Transformer models, and introduce two algorithms, SiMEC and SiMExp, which facilitate the exploration and interpretation of equivalence classes within this input space. These methods offer new insights the internal mechanisms of Transformers, and provide new understanding of how these models perceive and process input data which can be very useful in the field of explainable AI.

Strengths

1 novelty: This paper provide an innovative application of Riemannian geometry to analyze the input spaces of Transformer models, which is very novel in the area. 2 Theory: This paper establishes a solid mathematical theory on how Riemannian geometry is applied to Transformer models. Based on this theory, SiMEC and SiMExp are developed to explore the input spaces of Transformer models.

Weaknesses

In experiment, the MNIST dataset is a little bit trivial, as the pixels of the background is essentially zero. It is nice to see the application of the proposed algorithm on natural images like CIFAR.

Questions

See above

Rating

5

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Thanks to the authors

Thanks to the authors for their response. I appreciate the authors’ efforts to clarify the goals and claims of the paper in their first response. Unfortunately, I am still not convinced as it stands that the limited experimental results on MNIST sufficiently demonstrate the authors’ claim made in the rebuttal to `"demonstrate the potential empirical applications of our approach"` . I do not agree with the authors that adding additional results would have traded-off clarity (these can always be deferred to the supplementary material, for example). Furthermore, two additional reviewers have the same flavor of concerns—it would have been easy for the authors to conduct small experiments on any of the additional suggested datasets (such as CIFAR by **Reviewer-TeNK** or TinyImageNET as suggested in the initial review). In the absence of additional experiments, my rating currently remains the same.

Thank you for the response

Thank you for the taking the time to respond to all the questions. However many of my concerns remain unresolved. I agree with the authors that there are some differences with prior work that uses the local Fisher/data matrix. However, the differences about the activation function or loss function seem of little relevance in the context of deep networks. Unless I am mistaken, the algorithm used in (https://arxiv.org/pdf/2104.13289) is identical to algorithm 1. As mentioned in the weakness section, the experiments are limited to small datasets and 3 of the 4 results are qualitative. The purpose of algorithm 4/5 is still not clear to me; I understand the steps of the algorithm, but I don't understand what goal it achieves and there are no results that show it can be used to derive new insights. The authors have introduced new tools / algorithms but I believe that there needs to be more thorough experimental validation.