Summary
This paper presents an analysis of local patch representations generated from pre-trained Vision Transformers (ViTs). These ViTs fall into three categories: supervised (trained using ImageNet classification), self-supervised through contrastive learning (DINO), and self-supervised through reconstruction (MAE). The study involves a series of extensive experiments aimed at addressing four key questions:
Can transformers effectively classify local patches?
Why do reconstruction-based methods show suboptimal performance with K-NN?
At which granularity transformers detect objects?
How well ViTs track objects?
The findings include that DINO consistently outperforms other approaches across most of the tasks. Furthermore, it is noted that reconstruction-based methods exhibit certain high-variance features that may be detrimental to these tasks.
Strengths
The paper is well-written. The experimental settings are well-detailed and the results are clearly presented.
The authors conduct numerous experiments and I appreciate the authors' time and effort spent on them.
The findings on the relation between feature variance and K-NN clustering may lead to a new research direction. Picking up certain features from generic pre-training for specific tasks is very interesting.
Weaknesses
The paper sets ambitious goals by aiming to explore fundamental local behaviors of Vision Transformers (ViTs) and address significant questions. However, it falls short in several aspects, as outlined below:
1. Limited Novelty. While a pure experimental paper can definitely contribute to the research community, this paper's experimental settings lack innovation. The techniques employed, such as linear probing, K-NN for clustering, and the introduction of blur and noise for degradation, are rather basic. It's not that simplicity is a problem, but the experimental design should be more rigorous and well-founded, as discussed below.
2. Limited Evaluation Protocol. The paper aims to tackle four substantial questions, but it primarily relies on two experiments: clustering and retrieval. This limited evaluation protocol may not fully capture the models' true properties. For example, the clustering task involves only KNN and linear probing, which may not be suitable for the diversity of ViT features. There are alternative hypotheses, like multiple clusters in the features may correspond to the same semantic class (therefore KNN and linear probing may fail), or the features are not separable by linear methods (therefore linear probing may fail). To truly understand performance, fine-tuning the backbone and using a prediction head with better capacity should be conducted, which the paper lacks.
3. Limited Data: In addition to the limited evaluation protocol, the choice of data is also limited. For instance, the clustering experiments rely on Cityscapes data, which strongly biases towards scenes related to driving and certain classes like 'cars, trees, sky, road,' etc. Although the paper claims that findings generalize to ADE20k, the results are inconclusive with low accuracy (mostly around 5%, see Table 2 in the supplementary), making it difficult to draw meaningful conclusions. The same issue arises in the 'tracking' experiments, which are based on a subset of just four sequences from the BDD-100k dataset, another driving-related dataset.
4. Retrieval Experiment Disconnect: The retrieval experiments fail to directly address the question "At which granularity do transformers detect objects?" The experiments involve adding basic augmentations to assess embedding drift. However, the connection between these experiments and the underlying question is tenuous. The chosen augmentations are relatively simple and thus do not sufficiently represent real-world data variations. Essentially this experiment only tests the robustness of ViTs to certain handcrafted types of noise and blur.
5. Tracking Task Shortcomings: The paper's tracking tasks are not conducted using a proper tracking framework but rather rely on retrieval from different frames. This approach is suboptimal as matching between image frames may not be accurate. ViT features lack translational invariance, caused by positional embedding, so simple retrieval may fail if the object of interest moves. Consequently, this experiment only evaluates the consistency of ViT features across video frames and does not reflect the true tracking capabilities of ViTs.
In summary, while the paper's aspirations are commendable, it falls short in terms of experimental design, data choice, and providing insightful analyses, ultimately limiting its contributions to the research community.
Questions
1. It's intriguing to investigate feature variance. Have you considered normalizing the features to have the same variance and then applying KNN?
2. Clustering at the patch level in a dense-prediction dataset seems questionable. In the case of Cityscapes, a single patch can contain multiple semantic classes, and assigning it to a specific class may not align with the desired behavior of ViTs. Have you explored alternative methods for this?
3. It would be good to extend the paper to more pre-training methods, including visual-language (CLIP) and segmentation (Segment Anything).
4. The paper presents numerous experiments but more in-depth discussions may be beneficial. For example, examining MAE feature variance is interesting. It would be good to show some visual results on how these high-variance features contribute to reconstruction.
5. Is there any reason why the tracking experiments are only conducted on 4 videos in BDD-100k? Is it possible to try more generic MOT datasets?
A minor point, in the introduction, relating the paper to NLP and LLM seems like hindsight, not a motivation.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.