Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects

Summary

The paper aims to answer how Vision Transformers (ViTs) perform tasks requiring visual relational reasoning. The study focuses on two tasks: identity discrimination and relational match-to-sample (RMTS), showing that ViTs process information in two distinct stages: perceptual and relational. Further analyses using mechanistic interpretability approaches were used to validate this results. Overall, this work offered insights into same-different judgement which is intuitive to humans but proven hard for AI.

Strengths

- Similarity judgement task has been long studied in humans and animals, offering insights into perception and cognition, that's why the main question addressed in this study is of high interest for both psychology and AI community - The analyses included support the main claims - The boundaries of claims, and the questions remained to be answered were clearly stated in the discussion

Weaknesses

- Related work is poorly presented: Despite the vast literature on both the question and the methodology, only one paragraph was allocated to the related work. There are other works suggesting a two phase processing in transformers (see below for an example) which I think discussing them in the context of this work would help to put the relational reasoning in a broader context. - Maybe related to point one, but a theoretical explanation supporting the findings, that is, under what conditions these distinct phases appear would make the work stronger. @misc{cui2024, title={A phase transition between positional and semantic learning in a solvable model of dot-product attention}, author={Hugo Cui and Freya Behrens and Florent Krzakala and Lenka Zdeborová}, year={2024}, url={https://arxiv.org/abs/2402.03902}, }

Questions

I am curious about the potential impact of CLIP's text co-training, in contrast to DINO's image-only training, on their respective performances when fine-tuned for same-different tasks. Specifically, I wonder if CLIP's joint training on images and captions forces its embedding space (particularly in the later layers) to deviate from purely visual representations to accommodate textual information as well. Could this difference in training approaches be a significant factor in explaining their divergent behaviors on visual reasoning? In that case, the the perpetual/reasoning phases probably not a property of ViT (as title suggested) but a product of transformer combined with multi-modality.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

- Related work section is very limited. - Claims on ViT seem to confined to CLIP, so probably a slight revision of the title and text makes the basis of the claim stronger.

Summary

This paper examines how vision transformers (ViTs) process visual relations, focusing on same-different tasks. The paper finds that pretrained ViTs fine-tuned on these tasks often develop a two-stage processing pipeline: a perceptual stage that extracts object features into separate representations, followed by a relational stage that compares these representations. Using interpretability techniques, they show that the perceptual stage creates disentangled object representations, while the relational stage implements somewhat abstract comparisons. The paper demonstrates that failures in either stage can prevent models from learning generalizable solutions. By analyzing ViTs in terms of these processing stages, the authors suggest we can better understand and improve how models handle relational reasoning tasks.

Strengths

- Addresses an important drawback of supposedly generalist vision models - Clear demonstration and definition of tasks that the paper focus on (e.g. Figure 1) - Thorough analysis, both in terms of performance metrics and model mechanism, of results - Abundant visualization of results - Constructed dataset and tasks that can benefit future studies of this topic

Weaknesses

- Focus on more toy settings. Lack evaluation on more advanced/real world tasks. - While the central claim of dividing ViT perception into perception and relation stage is inspired by infant and animal abstract concept (line 71), there's the risk of confirmation bias from this analogy. - Plots in the paper requires clearer annotations and explanations (e.g. Figure 2 has too many acronyms, and it's difficult to understand what conclusion is for the plot) - Lacks qualitative example of more intuitively describe how model processes relational examples - While the abstract claims that understanding the relational reasoning help rectify short comings of existing and futures models (line 19), the paper doesn't discuss much what can be done to improve model relational reasoning (especially empirical experiments for improvements).

Questions

- What would the same analysis methods result when applied on real world images or more complex reasoning tasks such as ARC-AGI? - Can you compare the mechanistic interpretation result with other text-based interpretability methods such as logic lens? - How to improve relational reasoning capability in vision models? What are implications of such potential improvements? - How would success or failure in relational reasoning impact CLIP in terms of embedding's alignment with text descriptions?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The author doesn't discuss limitations (although the paper checklist claims that limitations are discussed in Section 8).

Summary

The paper uses techniques from mechanistic interpretrability to analyze the algorithms implemented by pretrained ViTs to solve abstract visual reasoning tasks. The authors use two synthetic same-different tasks: discrimination and relational match-to-sample (RMTS), to analyze CLIP pretrained ViTs, DINO pretrained and Imagenet pretrained ViTs, finetuned on these tasks, as well as a ViT trained from scratch. Some pretrained ViTs, especially CLIP pretrained demonstrated a strong perceptual stage in the early layers, disentangling object representations, followed by a relational stage in the later layers, which implements somewhat abstract same-different relations. The authors also demonstrate that the formation of only the perceptual stage is enough to solve simple relational tasks like discrimination, but not enough for RMTS.

Strengths

1. The paper is well written and easy to follow. 2. Interesting use of mechanistic interpretrability to analyze how pretrained ViTs solve same-different tasks: discrimination and relational match-to-sample (RMTS). 3. The paper shows that early layers of CLIP pretrained ViTs demonstrate local attention for within object processing (perceptual stage), whereas the later layers demonstrate global attention for between object processing (relational stage). The two stage processing doesn’t form strongly for the other pretrained models (DINO and ImageNet) especially the relational stage. 4. The perceptual stage is characterized by disentangled object representations (color and shape). Models trained from scratch can be enforced to have disentangled representations using an auxiliary loss which is enough for generalization in simple relational tasks like discrimination but not for RMTS task.

Weaknesses

1. Analysis is limited to simple tasks consisting of simple relations. It remains to be seen if the results would hold up for more tasks involving higher order relations and more complicated relations. 2. The paper doesn’t give a clear and detailed intuition behind why CLIP pretrained ViTs implement the two stage processing more strongly compared to other pretrained ViT models.

Questions

1. According to Fig 2a,c deeper layers don’t seem to be dominated by global heads as WO attention score is higher than WP? 2. Analysis of some models are missing in Fig 2 (e.g DINO Discrimination and Imagenet pretrained ViTs). 3. Can the authors also perform the same analysis for ViTs pretrained using masked autoencoding objective [1]? 4. Are the results in Fig 6 averaged for all models? [1] - He, K., Chen, X., Xie, S., Li, Y., Dollár, P. and Girshick, R., 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16000-16009).

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have discussed the limitations in the last section.

Summary

This work studies ViTs' learning behavior with relational tasks by experimenting on 2 same-different tasks: discrimination and RMTS tasks. And the authors propose a dataset to analyze. And discovers that there are 2 stages of attention processing of CLIP ViTs by attention scores from patches to other patches. They characterize 2 stages, perceptual stage and relational stage.

Strengths

This paper defined a novel avenue with inspiration from mechanistic interpretability to understand the working mechanism of CLIP ViTs. And proposed approaches to study them with attention analyses.

Weaknesses

1. I believe readers in this track would generally not be experts in concepts of the proposed area of studies, the paper should include more literature review for context upfront. 2. I find the writing and the flow of the paper hard to follow, maybe adding flow-chart would help improve readability. 3. The dataset is not well-discussed. 4. I think there is a slight mismatch between claimed study and experiments. The title suggest the study centers around ViTs but use CLIP ViTs, leaving me confused if the points made are the same on the original ViTs.

Questions

1. ln 23-24, the motivation "little breakthrough progress on complex tasks involving relations between visual entities..." is that true? What about the recent breakthroughs or datasets of vision-language models, Llava, GPT4, etc. Please give more explanations. 2. ln 41, please define what are "algorithms" learned by ViT, and are there algorithms learned by other architectures, CNNs, LSTMs, etc.? 3. Are there no existing datasets that can study the relations of visual entities? I mean in general, not the RMTS, etc. 3. The motivation of using CLIP ViTs for study is not clear. Since part of the motivation is from infant learning, why not use more human-like CNN like ConvNext? Also, CLIP ViT is different from ViT as well. 4. Vaguely remember [1] discussed about using better disentangled model for generalization and color, are there new insights from Sec 6? 5. With the 2 stage processing, what would you suggest the architecture to change to build a more "robust" version? --------- [1] Better may not be fairer? [Chiu, et. al., 2023]

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes.

Update on Point 4: How to Improve Models

We have continued to experiment with integrating attention-based losses to improve model performance in the RMTS task. We have found that combining an attention based auxiliary loss + the shape/color loss described in the manuscript improves performance dramatically for a randomly-initialized model trained on the RMTS task. Specifically, ViT B/16 accuracy improves from ~50% train, test, and compositional generalization accuracy **without auxiliary losses** to 95% train accuracy (+45%) , 91% validation accuracy (+41%) , and 77.4% compositional generalization accuracy (+27%) **with both attention and shape/color losses**. Notably, merely including the shape/color auxiliary loss results in far weaker performance improvements, at 54% train, test, and compositional generalization accuracy. These results are already present in Table 4 of the manuscript. In summary, this experiment further supports our main conclusion: that both multi-stage relational processing and well-structured object representations are required to solve the RMTS task. Inducing both of these factors is possible using straightforward auxiliary losses during training.

Official comment by Reviewer DaaT

Thank you for the detailed rebuttal, which has addressed some of my concerns. I have increased the rating to 6.