Summary
The paper introduces AtMan, a method that manipulates the attention mechanism of generative transformer models, to provide explanations. More specifically, it perturbs raw attention scores instead of the token inputs. The authors provide extensive experiments for different modalities to back their claims regarding better results to established baselines.
Strengths
1. The paper introduces a new approach towards Explainable AI, that does not require calculation of gradients through the network, thus enabling evaluation of larger models with the same hardware.
2. The idea of manipulating attention scores is novel and interesting.
3. The authors embark on an extensive evaluation of their method on different modalities. The universality of the Transformer architecture, enabled them to seamlessly switch between modalities. The methods performs well on a range of architectures (encoder, decoder).
Weaknesses
1. One of the main motivation of this work is to enable more widespread adoption of explainable AI methods, via reducing required computational resources. The issue of memory is discussed extensively. Inference speed is only mildly explored, in Figure 5. For large sequence lengths, the method requires upwards of 2 orders of magnitude more time to make the final predictions.
2. The method requires one to set new hyperpatameres, namely $f$ and $\kappa$. The effect of those with respect to e.g. sequence length or model architecture, are not discussed. For example, different sequence lengths may lead to different softmax entropy and thus different $f$ values might be required.
3. According to Eq. (4), tokens are masked by setting the corresponding $H$ values to 0. This does by no means lead to masking of the tokens, as briefly explained in Appendix A5. Setting the values to the value $0$ constitutes another hyperparameter, that needs to be set for the proposed method. In principle any real value could have been chosen (In Appendix A5 the authors explore the values $0$ and $-\infty$). The range of the possible values was not explored at all.
4. In general a lot of approximations are made for the derivations of the last method, with possible alternatives not discussed. Why was Eq. (4) chosen to mask tokens? Why is the cosine similarity of the embeddings a good measure to suppress common tokens? In [1] for example, the cosine similarity between keys is used as a more suitable measure. The cosine similarity is calculated based on the embeddings of what layer? Is there a benefit from calculating this similarity on different layers? There is a lot of work on what type of information token embeddings carry for different layers in a Vision Transformer.
[1] Bolya, Daniel, et al. "Token merging: Your vit but faster." arXiv preprint arXiv:2210.09461 (2022).
Questions
1. Line 62 states that the source code is released although I could not find it in the supplementary.
2. Line 33-34 states that memory consumption leads to uneconomical productive deployment. This is a overstatement, especially given the computational requirements of the new proposed method.
3. A lot of literature has been exploring the effect of attention scores in the context of Stable Diffusion, e.g. [2] and follow-ups.
4. Is there a minus missing before $L^{\text{target}}$ in Eq. (2)?
5. Line 245, it is not clear to me how the mean of generated scores affects the final predictions. Are some tokens in the target label more important than others as expected?
6. Is there a way to prioritize checking some tokens first instead of exhaustively trying out all tokens to find the ones that explain better the target? This might help to reduce the inference speed. You can think about having a method to rank tokens to check, and then create a plot of computational budget vs performance, where you check iteratively more tokens. You could also think of doing this in a more hierarchical way, similar to what is proposed in Appendix A8.
[2] Hertz, Amir, et al. "Prompt-to-prompt image editing with cross attention control." arXiv preprint arXiv:2208.01626 (2022).
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The paper proposes a way to manipulate attention scores to explain prediction of Transformer models. Manipulating these scores is done a token at a time, leading to inference speed that scales with the sequence length, and is orders of magnitude bigger than previous approaches. Although a lot of experiments are presented, some of the design choices are not adequately justified.