Summary
In this paper, the authors propose a feature descriptor to assess low-level image quality degradations. Based on CLIP, the proposed method first encodes input image and its degraded version to features in CLIP space; the input image is encoded by CLIP image encoder, and the degraded image feature is generated by adding a textual feature of degradation to the image feature. The Deep Degradation Response (DDR) is measured by calculating the distance (seems that cosine is used) of two features. The authors demonstrated the effectiveness of the proposed descriptor with extensive experiments and analysis including SRCC test and applications to image deblurring and super-resolution.
Strengths
1. The paper proposes to exploit CLIP feature to measure image quality under various degradations. Unlike other image quality works focusing on image-based approaches, this paper introduces a novel approach using textual features.
2. Surprisingly, it seems that the proposed method works well without training CLIP with degradation prompts (I need clarification of it in Weakness). It demonstrates that CLIP can be used as a tool for image quality assessment.
Weaknesses
1. Clarity: Some technical details are unclear, which limited the understanding of the proposed method.
- I did not understand the distribution of DDR in Figure 2. Why is the distribution of "adaptive" better than "low" and "high"? What did the authors do specifically for "optimal"?
- The authors mention in L84-85 that there are options in the disparity metric M, but in SRCC evaluation there exist a clear positive or negative direction of correlation. Since Ln and cosine metrics have different meaning for low/high values, more clarification is needed in the description of the method. It seems that the authors used cosine metric in their code.
- It seems that the method simply used pre-trained CLIP features for both images and texts. Is it correct?
- Among degradation types, Color and Content seem ambiguous. Do the authors have a clear definition of these categories?
2. Even though the DDR can score good and bad quality images in terms of (color, noise, blur, ...), it seems difficult to measure the amount of degradation such as noise level. It makes me wonder if the DDR contains useful information to measure degradations, or simply focus on visually pleasing images because positive texts usually correspond to those images? This is a reasonable question as the authors are aware of the fact that the CLIP features may have bias to high-level features rather than low-level degradations.
3. Continuing 2., although the authors framed the paper as image descriptor for degradations, I think the paper is more relevant to non-reference image quality assessment. Therefore, more prior works in this line of research need to be discussed in Sec. 2.
4. Although it is mentioned in the limitation, the proposed method is evaluated on one type of sentence per each degradation. Can the authors justify why they chose these specific words? Due to the nature of texts, there would be similar words that share similar meanings. How robust is the proposed method against the choice of words?
Questions
The paper generally addresses an interesting problem of image quality descriptor using CLIP textual features, and the results look encouraging. However, it lacks the in-depth understanding of the CLIP and proposed method. I would like to hear the authors' answers to the points raised in Weaknesses before adjusting my rating.
Limitations
The authors described two limitations of the proposed method. A potential negative societal impact would be
- The proposed method could be potentially used to discriminate a certain group of photos (e.g. race, gender, etc) by including those words in degradation prompts.