When Does Perceptual Alignment Benefit Vision Representations?

Summary

The paper investigates the benefits of aligning vision model representations with human perceptual judgments to improve their performance across various computer vision tasks. The study fine-tunes state-of-the-art models using a dataset of human similarity judgments and demonstrates that this alignment enhances performance in tasks like semantic segmentation, depth estimation, and instance retrieval. The results suggest that integrating human perceptual knowledge as an inductive bias can improve vision model representations without significantly sacrificing performance on other tasks, including specialized domains like medical imaging and 3D environments.

Strengths

1. The experiment evaluation is comprehensive. The paper evaluates the impact of human perceptual alignment on a wide range of computer vision tasks, including semantic segmentation, depth estimation, instance retrieval, and counting. The paper also use multiple state-of-the-art models (e.g., CLIP, DINO, DINOv2, SynCLR) and a detailed experimental setup. 2. The idea of introduces alignment for vision models with human perceptual judgments is reasonable and innovative. 3. The analysis is insightful. The paper provides a nuanced discussion on the benefits and limitations of human perceptual alignment, offering insights into how different levels of perceptual judgments (low-level, mid-level, high-level) impact model performance on various tasks.

Weaknesses

Writing Part: 1. There is a question mark in a citation on line 95. 2. The order of the references is shuffled; reference pages usually come before the supplementary files. 3. The paper lacks a conclusion section. Suggestions for Experiments: 1. More experiments on advanced vision-language models (VLMs) are expected, such as Llava, MiniGPT-4, and instructBLIP. 2. While not necessarily a weakness, it would be interesting to see the performance improvement on CLIP-blind image pairs, as introduced in the paper "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs."

Questions

How does the patch-level objective differ from the image-level objective? Does it show different performance on various downstream tasks?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

Introduced in Section 5.

Summary

This paper investigates the effects of aligning pretrained vision models to human judgments. Image-level and patch-level learning objectives are proposed for fine-tuning the pretrained models. The experimental results demonstrate that fine-tuning pretrained models (such as CLIP, DINO, and DINOv2) on the NIGHTS dataset leads to performance improvements in various downstream tasks, including semantic segmentation, depth estimation, retrieval-augmented generation (RAG), counting, and instance retrieval. Ablation experiment regarding the fine-tuning dataset is conducted to demonstrate the perceptual qualities embedded in the NIGHTS dataset.

Strengths

1. The proposed perceptual alignment process is simple yet effective, leading to performance improvements across various downstream tasks. 2. This paper discussed the effects of aligning vision models using datasets with different levels of variations, providing emprical evidence for further research.

Weaknesses

1. This paper demonstrates the effects of perceptual alignment by showing the performance improvement in downstream tasks. However, the presented evidence cannot fully explain "how" perceptual alignment affects model features as general-purpose representations. Specifically, it is unclear what the difference is between the features before and after perceptual alignment, and why such differences lead to performance improvements. 2. This paper investigates the effects of aligning vision models using dataset with mid-level variations leads to a better “general-purpose representation”. However, the reasons why using datasets with such mid-level variations (instead of datasets with high or low levels of variations) for alignment results in better representations remain unclear. 3. Experiments in this paper are conducted using ViT based models. However, the effects of perceptual alignment for CNN based models are not explored. 4. This paper compares the effects of aligning vision models using datasets with low, mid, and high levels of variations on counting and instance retrieval tasks. However, to better demonstrate the high perceptual qualities embedded in the NIGHTS dataset, comparisons should be conducted in a wider range of downstream tasks.

Questions

Please refer to the weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors discuss some of the limitations of their work in Section 5. But I would like them to consider some of my concerns above.

Summary

This paper aligns representations with human perception on mid-level semantics to improve performance on various downstream tasks. Specifically, it does this by pre-training models on additional synthetic image triplets, where the visual similarity within each triplet is annotated by human subjects.

Strengths

- The dataset this paper used for additional pre-training is rather small compared to the standard datasets, yet authors show that there is still a noticeable improvement on multiple tasks.

Weaknesses

- Lacks general instructions on when aligning with human perception would benefit representation learning. From the title, I would expect a series of studies providing insights into which levels of alignment benefit or impair different types of tasks. The empirical study in its current version is not comprehensive enough to compensate for the lack of theoretical novelty compared to previous work [1]. [1] *"Improving neural network representations using human similarity judgment,"* Oh et al., NeurIPS 2023 - It would be better if the authors could provide some empirical analysis to give insights into why mid-level alignment is better than other levels.

Questions

- Regarding dense prediction tasks, there are cases where pre-training with human alignment impairs performance, e.g. DINOv2(-HA) on VOC, etc. The authors stated that DINOv2 has seen these data, and considering that DINOv2-HA has also seen these data, how does this explain the performance drop? Since the only difference in the dataset for models with or without HA should be NIGHTS, this needs clarification. - It seems odd that the performance change from using ImageNet and THINGS for pre-training is reversed in Figure 8 for the counting task, and pre-training on THINGS looks extremely bad for retrieval tasks. Can the authors provide an explanation for this phenomenon? - Compared with previous work [1], the authors use additional patch-level alignment during pre-training. Isn't there some redundancy between the average pooling of patches and the [CLS] token? How does patch-level alignment affect performance? Additionally, since some patches from negative samples and reference images are similar, it seems counterintuitive to push them away in the embedding space. [1] *"Improving neural network representations using human similarity judgment,"* Oh et al., NeurIPS 2023

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

See above

Summary

As the title clearly indicates, this article investigates how aligning vision model representations to human perceptual judgments impacts the performance of models relying on these representations for downstream tasks such as dense prediction (semantic segmentation and depth estimation), retrieval-augmented generation, object counting and instance retrieval. Given a vision model backbone, it is finetuned with LoRA on the NIGHTS dataset, which contains human similarity judgments over synthetically-generated image triplets (NeurIPS 2023). For each considered downstream task, several backbones are compared to their "human-aligned version" with globally better results when the backbone is fine-tuned on human perception.

Strengths

* the scientific question raised by the paper is simple but interesting and the study shows it is worth investigation. The problem is clearly stated and introduced, the method and protocol are well explained and the article is well written overall. This bullet point is not just a false pretext to artificially add a "strength" to the review, it is a real pleasure to read. * the method to fine-tune the backbones relies on well-known and standard methods and tools (visual transformer, triplet loss...), which leads to a convincing methodology. The reader is not confused by an incomprehensible labyrinthine system and can therefore fully concentrate on the results of the study. * the experimental part is impressive, with an evaluation of a large variety of tasks, with various backbones as well. This is obviously a strong part of the paper: although the original idea was simple and interesting, the general quality of the study is finally supported by all these experiments. The supplementary material and the code (zip file) allow a precise investigation of the experiments and ensure reproducibility. * Section 4.5 investigates the use of alternative datasets to fine-tune the model. Some were proposed by previous works and others were built by the author to test a specific question. All of them have a size similar to NIGHTS. The resulting analysis is interesting and is a nice complement to the study.

Weaknesses

* the alignment is performed both at the image level (section 3.2) and patch level (section 3.3) and the authors claim (lines 164-166) that "Additionally, we find that local patch-level representations can be improved by tuning on image-level perceptual judgments, and show performance increases on semantic segmentation and depth estimation". However, in the following all experiments only compare a backbone with its "human aligned version" (HA) and it is not clear what is the contribution of the alignment at a global and a local level. On line 148, the text says that the model consists to "train heads for semantic segmentation and depth estimation [at the local level]". Globally, the manuscript could be more explicit on this "details": is the local patch level used for training for segmentation and depth estimation only or is it the same training protocol (with global and local levels) for all experiments? If so, what is the contribution of adding (or not) the heads that are trained at a local level? * one may regret that the negative result on VTAB is not reported in the main paper and only discussed in the Limitation without further investigation. However, one must recognize that the paper already contains significant experimental works, that allow to support the main demonstration. * the fact that the same backbones are used in several experiments/tasks is interesting but sometimes it leads to models that are very weak baselines in comparison to the state of the art. In particular, in Section 4.3, specialized class-agnostic models on few-shot counting have much better performance than those reported (less than 0.15 MAE/RMSE on FSC147 for SafeCount or CounTR) **minor** * the experimental protocol for "object counting" reports that $k$ is determined on the training set among 4 values then the best one is used on the test set. Reporting the performance for each $k$ on the training set, or at least the one that was retained on the test set, would be useful. * line 95: a reference is missing

Questions

* is the local patch level used for training for segmentation and depth estimation only or is it the same training protocol (with global and local levels) for all experiments? If so, what is the contribution of adding (or not) the heads that are trained at a local level? * will the code (zip file) be released with the paper?

Rating

8

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

An experiment on the VTAB benchmark, whose results are reported in the supplementary material, leads to lower performance with fine-tuned representations. This case is significantly discussed in Section 5, with several hypotheses proposed to explain the phenomenon.

Summary

This paper studies the question of when perceptual alignment supervision makes better vision representations. They fine-tune existing pre-trained vision backbones (eg, CLIP and DINO) with LoRA on NIGHTS, a synthetic triplet-based dataset of human similarity judgments. Models fine-tuned in this way generally perform better on vision tasks, ranging from dense prediction, RAG, counting, and object retrieval. However, the results can be the opposite when evaluating classification tasks, especially natural classes. They also considered fine-tuning on different types of human preference datasets, and found mid-level supervision to be more beneficial.

Strengths

1. The tasks considered in this paper for evaluation are comprehensive and cover many different properties of the representation. 2. The delivery of this paper is clear and easy to follow. Key motivation, setting, and conclusions of each part are easy to spot. 3. Evaluating human preference for vision representations is an emerging topic and interesting for research.

Weaknesses

1. My major concern is that although the topic is somehow interesting and the evaluation is comprehensive, many conclusions are expected and unsurprising. Provided in the form of triplets, the supervision of human preference is just another form of "classes", being very fine-grained and having flexible class definitions. This is similar to the learning signal of self-supervised contrastive learning, with a better form of data augmentation. In this regard, it is expected that human preference makes generally better representations. 2. Regarding the performance drop in fine-grained classification (especially natural classes), this might be a result of the domain gap. Given that the model is fine-tuned on all synthetic images, it is unsurprising that discriminating natural images is harder afterward. The definition of human preference can be ambiguous, and the dataset for fine-tuning can have distracting factors (eg, synthetic distribution). These factors are not tackled well in the design of comparisons and could make some conclusions unreliable.

Questions

I still find this paper could spark some interesting questions. More investigations on these aspects might increase the significance of this paper. 1. When can human preference be harmful? One motivation of this paper is to provide some empirical guidance for future vision research, if human preference is introduced as the way in the language community. As I mentioned above, the benefit of it to vision is not very surprising, but the prevention of possible risks could be valuable. This paper has shown some and more in this direction might be more valuable than the benefits (as most improvements are just marginal). 2. How should human preference be better defined and categorized? A precise definition of human preference is hard to obtain, but important. This paper has considered the level of supervision, and a discussion of other aspects could help readers. 3. What distracting factors exist in current human preference datasets and how to isolate the effects of them? This is important to ensure reliable conclusions.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations are discussed in the paper. Suggestions are flagged above.

Thank you for feedback

I acknowledge the authors for their feedback and encourage them to include in their camera ready version as much as possible clarification provided in the rebuttal. I also read the other reviews and I note that my perception is significantly more positive than the average other reviews. Nevertheless, the weaknesses raised do not seem sufficiently convincing to me, and I therefore maintain that this article deserves to be brought to the attention of the community.