DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

Summary

This paper introduces a new dataset consisting of images generated by prompting the Stable Diffusion models, aiming for studying the mid-level image similarities. Human perceptual judgments are collected for image triplets via both 2AFC and JND tests. A perceptual metric is built upon large pretrained vision models, which is empirically shown to be highly consistent with human judgments.

Strengths

+ It makes one of the first attempts to study the evaluation of AIGC images, the contribution of the new dataset is significant and the pipeline to collect human perceptual data is reasonable. + Experiments are well organized, which manifests the shortages of current perceptual metrics in evaluating AIGC images and verifies the effectiveness of the proposed metric. + The feature inversion part is interesting, aligning well with the Analysis-by-Synthesis model evaluation methodology.

Weaknesses

- The definition of distortion seems a bit confusing. In the literature, distortions are generally associated with degradation of image quality. In this work, distortions refer to mid-level changes, which do not necessarily lead to the loss of visual quality. - It is unclear how the JND experiment is conducted. When the participants are expected to give an answer of yes? Two images without any noticeable pixel changes, or perceptually the same in terms of the mid-level change? - More details of the feature inversion part are expected.

Questions

Does the proposed metric perform well on classical image quality assessment datasets?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

Not applicable.

Summary

In this paper, the authors develop a perceptual metric that assesses images similarity holistically. They first collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. During the dataset construction, the authors utilized Stable Diffusion to create synthetic data, aiming to model mid-level similarity. Based on the dataset, the authors observe that popular perceptual metrics fall short of explaining the new data, so they introduce a new metric, DreamSim, tuned to better align with human perception. Finally, authors applied this metric in image retrieval and showed that it can generalize to real images.

Strengths

Human visual similarity is an important and practical topic in the domain of computer vision. It has many potential applications such as image retrieval. One of the major contributions of this paper is the new dataset (NIGHTS), which contains human similarity judgments of 20k synthetic image triplets. I believe this dataset will be useful for fellow researchers in the related domain. Another strength of this paper is the proposed similarity metric that captures how humans naturally perceive image similarity. I also appreciate the authors detailed analyses and comparison of the metric and its performance.

Weaknesses

Though the paper is well written, and the topic is interesting and practical, the paper has several weaknesses. First of all, as the authors have mentioned, due to the potential biases and other limitations of Stable Diffusion, the authors deliberately avoided human faces during dataset construction. However, human faces are a common subject in general visual scenes and human perception of images containing human faces should be studied. Second, though the authors have applied the metric in image retrieval and used Figure 8 as a vivid proof to demonstrate that the proposed metric can be generalized to real images, I still have several concerns: first, is it possible to test the proposed metric on another benchmark dataset of real images, instead of the image retrieval task? Second, who are the users who rated the preference on the image retrieval results? More details of performance evaluation are needed. Third, not all metrics in Figure 4 are evaluated in the image retrieval task, why? Finally, while the authors avoided human faces in the proposed dataset, the models retrieved all images with human faces in the bottom right block in Fig 8. Does it show that the model focused on features other than human faces?

Questions

1. Why do the authors not compare all the metrics used in Figure 4 during the image retrieval task? 2. Who are the users who rated the preference on the image retrieval results? 3. Can the authors give a more comprehensive evaluation of the generalizability of the proposed dataset?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

A more comprehensive evaluation of the generalizability of the dataset and metric is needed. Furthermore, humans are sensitive to faces from birth. Visual similarity on images with human faces is important.

Summary

The paper expands the fundamental task of measuring perceptual image similarity to encompass factors beyond low-level and high-level similarity. To achieve this, it leverages the recent advances in generative visual AI and generate synthetic images with text prompts. These synthetic images are expected to contain multi-faceted notions of similarity, which are then handed over to humans for A/B similarity test. The proposed model then learns from these human similarity ratings and is shown to uncover the emergent hidden factors defining visual similarity. In extensive evaluations, it is shown to surpass a number of baselines, including DINO and CLIP.

Strengths

-- Research angle on visual similarity is timely, given its rather narrow scope of examinations in the past. Breakthrough on this angle can benefit vision community at large,. -- Using synthetic images for visual similarity learning is a promising technical path forward given our increasing controllability on generating target visual contents. Instead of harnessing human ratings on the similarity between any two real images and letting models to learn as a black-box, using synthetic images has the advantage of leaving the discretion to human designers on what constitutes visual similarity -- for example in this paper, although only superficially, manipulates on mid-level visual variations. -- Using LoRA to efficiently tune pre-trained visual ensembles towards a visual similarity judge is also reasonable. -- Extensive analysis is extensive.

Weaknesses

The paper is itself is quite intact by itself. So what I propose here is not necessarily a weakness, but rather with such a new tool on visual similarity, I keep wondering what can it, immediately, benefit vision tasks at my hand. The paper showcases a pilot study on the application of image retrieval, but in my understanding, the paper's contribution is best leveraged as a universal (differentiable) plug-n-play module to provide a complementary power to many computer vision tasks. For example, it would be lovely to see how this paper opens new opportunities to image quality assessment community, where the intrinsic ambiguity of ground-truth human ratings is painstaking for generalisable modelling. Can DreamSim help alleviate this problem as a regularisation force by requiring similar visual contents to be of similar quality score? Same for the problem of sketch-based image retrieval community, where sketch-image ground-truth paired data is ill-defined. In the work of [A], they have shown how leveraging a visual similarity metric like LPIPS can greatly advance the understanding of the problem. Can DreamSim bring even further breakthrough? [A] Photo Pre-Training, But for Sketch. CVPR 2023

Questions

- Would be great if the paper shows more practical use cases that benefit from such a new visual similarity judge. For example, as a plug-n-play module in diverse image retrieval tasks? Or as an essential component for better evaluating a generative visual agent?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Summary

This paper leverages recent advances in synthetic image generation (namely text-to-image) to explore the space of image similarity assessment. They generate a novel dataset based on human forced choice input on pairs of synthetic images generated with respect to a reference image as well as interleaved visual memory tasks to assess when images are noticeably different on first assessment. The metric they learn over this novel dataset appears to better capture mid-level features such as pose and viewing angle.

Strengths

The paper is overall clearly written. The experiments are well-described and compelling, with multiple complementary facets supporting each other. The paper provides both a dataset and metric which may be of interest to the image similarity domain.

Weaknesses

I apologize for a somewhat unclear criticism, but the JND task appears to be a visual working memory task, not a perception task. As we know from investigations of change blindness, there can be quite distinct differences between images that humans will only notice given sufficient time to explore the image, but if they happen to be attending the location that is different, they will immediately notice the difference. This seems to be getting at a different aspect of similarity than the perceptual choice task, which makes me want more of an explanation for why these two tasks are paired, or how they interrelate and why they are paired in this way.

Questions

- How are the categories drawn from the labels? Is it a random sampling (and if so, are duplicate categories controlled for?), a subset (and if so, how was this selected?), or another arrangement? - Although it is mentioned that 7 votes is the average for retained triplets (L148), was there a minimum number required to be retained? - How do humans react to the image ablations tested in Section 5.2? It seems unclear to me whether the model should be impervious to some of the changes made.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations seem clearly articulated.

Post rebuttal

The authors have addressed my concerns well in the rebuttal, I am willing to raise my rating to 8.