LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate

Summary

This paper proproses LookHere, a novel positional encoding method for Vision Transformers for dealing with high-resolution images. Specifically, LookHere explicitly constratin attention heads to attend certain directions via 2D masks. With comprehensive experiments. LookHere demonstrates strong performance across different benchmarks. It also achieves better performance than existing positional encoding methods on high-res images.

Strengths

1. The idea is very clear. High-res images introduce new challenges on positoinal encoding due to the mismatch between the training and inference time. By introducing 2D attention masks, we can assign different heads a direction, such that they obtain explicit bias and will know how to attend on high-res images. 2. The experiemnts are very comprehensive and strong, which well demonstrate the effectiveness of the proposed method. 3. This paper is well-written and in a very good shape.

Weaknesses

1. Actually the authors have clearly dicussed the weakness of their method, e.g. hand-designed masks, single scale of ViT. Beside that, have the authors considered a discussion with ConViT [50]? ConViT introduces convolutional bias to attention head, which also restricts the head to only attend to certain areas. 2. (Minor) It would be more promising to see if this technique can be applied into multi-modal LLMs.

Questions

See the weakness.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

Yes

Summary

This paper introduces Lookhere, a novel mask-based positional encoding designed to address the performance degradation of ViT when resolution changes. Lookhere restricts the receptive field of each head in the attention mechanism and enhances the diversity of information perceived by each head. Extensive experiments have demonstrated the effectiveness of Lookhere. Compared to other positional encodings, Lookhere has significantly better resolution scalability.

Strengths

1. The author's writing and illustrations are very clear, making it easy to understand and follow the method. 2. The authors conducted numerous experiments, thoroughly validating the effectiveness of Lookhere. 3. The issue explored by the authors, namely the resolution generalization of vision models, is a very important but rarely investigated area. The authors' work could potentially guide future research in this direction.

Weaknesses

In summary, I believe this is a very solid work with no obvious issues. I have only one personal concern. From Figure 6, it can be observed that the essence of the proposed positional encoding seems to be limiting the receptive field of each token, so that when the resolution changes, the number of tokens each token attends to does not change too drastically. However, this approach seems to reduce the model's overall receptive field. As a result, the fine-tuning results at a resolution of 384 do not show a particularly significant advantage compared to other positional encodings. I suspect that as the fine-tuning time increases, this advantage may disappear or even be worse than RoPE. Could the authors fine-tune for a longer period (for example, 30 epochs) to verify that this approach does not negatively impact the model's accuracy?

Questions

Please refer to the weakness

Rating

7

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

N/A

Summary

The authors explore an alternative to existing positional encoding methods for vision transformers. Within attention operations, 2D attention masks (subtracted from attention matrix i.e. key-query inner product, prior to softmax) limits feature mixing to fixed fields of view pointed in different directions. ViTs resulting from proposed method exhibit improved performance across diverse tasks, especially when extrapolating to input image dimensions larger than the train data. The authors also contribute a new high-resolution ImageNet dataset variant for evaluation.

Strengths

1. The paper is well written with clear motivation, thorough discussion of prior works, and detailed explanation of methodology. 2. The proposed method is simple and elegant, providing better OOD performance to ViTs. This can be highly useful to the community. 3. Rigorous experimentation including hyper-parameter tuning to all prior works on a common training setup. 4. Extensive evaluations and ablations to establish usefulness of proposed method and to highlight its various interesting aspects. 5. Useful high-res ImageNet dataset that is already released anonymously (in appendix)

Weaknesses

1. **Translation Equivariance:** Authors claim this to be a result of proposed method. However, this is not proved theoretically or demonstrated through experiments. Please explain this in method section, compare to CNNs, and provide experiments to quantify this is possible. In case I missed something related from appendix, please highlight it better in main paper. 2. **[Minor] Adversarial Attacks:** Only FGSM - maybe try newer methods like AutoAttack?

Questions

1. **Learning Parameters**: Have you tried learning the slope values? And maybe even the entire attention masks? These could be learned using some SSL setup like MAE.

Rating

8

Confidence

3

Soundness

3

Presentation

4

Contribution

4

Limitations

Discussed.

Summary

Vision transformer is known for its constrained scalability across various image resolutions and this work is designed to address the generalization ability of ViT at high-resolution images. Specifically, the authors propose LookHere, a drop-in replacement for the positional encoding of standard ViTs, which restricts attention heads to fixed fields of view oriented in different directions using 2D attention masks. It offers translation-equivariance, guarantees attention head diversity, and minimizes the distribution shift encountered by attention heads during extrapolation. Extensive experiments validate the effectiveness of the proposed method.

Strengths

1. Evaluation is extensive. The authors provide validations across different datasets, and tasks, and offer detailed analysis to demonstrate the effectiveness of the proposed approach. 2. The results seem promising. LookHere exhibits the best performance compared to other baseline methods. 3. The presentation is clear and the paper is easy to follow.

Weaknesses

1. The generalization ability of LookHere to smaller-scale images remains unverified. Although the authors particularly focus on testing at high-resolution images, it would be better if LookHere could be generalized to smaller images as well. 2. The ablation study on the design choices seems missing. Although the authors provide abundant experiments and analysis, the ablation study on the specific designs is missing and their efficacy is unverified. 3. The size of the proposed ImageNet-HR is too small (5 images per class). The limited number of images in each class may not be representative enough and could have a strong bias implicitly.

Questions

Apart from the questions in weakness, the reviewer has one additional question: Is there a particular reason that the authors did not compare LookHere with NaViT, which can also be tested at higher resolutions?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Final Review

The authors address all concerns; no changes to my original rating of 8 (strong accept). Highly encourage authors to release results on stronger adversarial attacks at a later point in time as feasible.

final review

Thanks the author for the rebuttal. My concerns are well addressed. I have raised my score to 7.

Official Comment from Reviewer

Thanks for the authors' rebuttal. They have addressed my concerns.

Post-rebuttal comment

The reviewer appreciates the detailed response provided by the authors. The concerns are addressed and the rating is upgraded to weak accept.