Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models

Summary

In this paper, the authors developed a joint model of CLIP and LLM to solve the task of Visual relation detection. In this model, images are encoded into a triplet,~\ie, object, subject and spatial branches. Then it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Experiments on four VRD benchmarks shows good results compared to the baseline models.

Strengths

+ Interesting and good applications of LLMs including GPT-3.5 and large multi-modality pertaining models~\eg, CLIP. + Show higher performance than baseline methods. + Good storytelling to readers understand the key idea.

Weaknesses

### Technical Novelty and main ideas 1. This paper pays its major attention to designing and using LLM~\ie, GPT-3.5 to facilitate the deduction of multi-modal models. Many contributions lie in the design and feeding prompts into LLMs. This contribution seems insignificant to me and seems not generalizable for future LLM using different architectures with GPT-3.5. Besides, the authors also lack deep investigation into the improvements of the Chain of Thought (CoT). Although the overall application and using LLMs seem interesting. but the solid contributions of this paper are not clear to me. ### Presentation and motivation issues The descriptions of CoT and experimental results are somehow unclear. Besides, the novelty of using CoT in this task is not sufficient. Designing prompts 2. The reviewers tested the same prompt using GPT-3.5, while in this case, with or without Chain of Thought (CoT) does not show significant differences. (with CoT: avg: 0.51, 0.3, 0.19 for s,o,p; w/o CoT: avg: 0.59, 0.33, 0.14 for s,o,p) The reviewer understands the results may not be stable but is still unclear about this case. 3. Why did Figure 5(a) choose 0.4, 0.4, 0.2? The authors should explain how `obviously unreasonable` on lines 188 - 190 is `obviously unreasonable`. ### Experimental issues 4. For Figure 2 and Figure 7 in the supplementary material. a) Where did the cues for the CLS baseline displayed by the sentence come from? Line 214 of the text states to use "relational CLasS-based hints (e.g., ride)", which are somehow inconsistent with the statements. Can the authors explain in more detail how the CLS baseline is tested? Where does the performance difference come from if two settings use the same prompts? 5. Although the newly proposed setting, the authors do not discuss other similar works. PEVL[1] and STIP[2]. Differences and relation discussion could help. [1] Yao, Y., Chen, Q., Zhang, A., Ji, W., Liu, Z., Chua, T. S., & Sun, M. (2022). PEVL: Position-enhanced pre-training and prompt tuning for vision-language models. _arXiv preprint arXiv:2205.11169_. [2] Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., & Chen, C. W. (2022). Exploring structure-aware transformer over interaction proposals for human-object interaction detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (pp. 19548-19557). 6. The time efficiency of different compared methods in Tab.4 should also be provided. 7. [Experimental Description] For the conjecture in line 247, does deduction give evidence for a confusion matrix? The reviewer did a cursory check of the dataset and there are a total of 117871 annotations, These provided actions seem to be rare.

Questions

Please refer to the weakness section. Besides the technical novelty, this manuscript exist many unclear implementation problems and insufficient experimental results, especially a lack of discussion with other works using LLMs. I hope the authors could solve these concerns to make this paper more readable.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors clearly discussed the limitations of the proposed method. These discussions are fair.

Summary

This paper aims to address the VRD problem using LLMs. The paper decomposes the visual features into human, object, and spatial features, and designs prompts to generate visual cues that describe each of these types of visual features. The relation classification is established by calculating the distance between visual and semantic features, and dynamic weights generated by LLMs are also integrated to enhance the training process.

Strengths

1. This paper focuses on a critical issue in visual relationship detection tasks, exploring the potential of leveraging LLMs to enhance visual relationship understanding. 2. The paper introduces a novel and reasonable approach by decomposing each predicate category into human, object, and spatial descriptions. 3. The authors thoroughly investigate various approaches to enhance the quality of prompts, encompassing both the generation of visual cues and the improvement of weights.

Weaknesses

1. It appears that the visual cues employed in the main papers are presented as mere examples, leaving uncertainty regarding the specific visual cues utilized in the experiments. Furthermore, the visual cues depicted in Figures 3 and 4 exhibit notable differences, with Figure 4 generating more complex sentences. As a result, evaluating the quality of the visual cues based on the current evidence becomes challenging. 2. The RECODE's performance gain on the HICO-DET and V-COCO datasets is marginal, and the authors did not provide error bars in their reports. Additionally, the ablation studies were solely conducted on the VG dataset, which could have substantial differences compared to the HICO-DET dataset. Consequently, it is difficult to be convinced that the RECODE is as effective as claimed by the authors. 3. The main technical contribution of this paper lies in the development of specifically designed prompts. However, the improvements made to the prompts are relatively straightforward, and the utilization of CoT is a standard practice. 4. There are many incurious statements/claims. For example, in line 47-55, it is unclear why a person has to stand while holding an object.; in lines 67-69, it is not clarified why the act of holding depends on spatial factors.

Questions

1. Could the authors provide additional examples of prompts and the corresponding visual cues generated by GPT that were utilized in the real experiments? 2. Could the authors present more empirical evidence to further support the benefits of RECODE?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

The authors have adequately discussed the limitations.

Summary

This paper proposed a novel method for zero-shot visual relation detection by leveraging LLM (e.g. GPT) and VLM (e.g. CLIP). Specifically, the proposed approach decomposes each predicate category into subject, object and spatial component and enrich each section with the help of LLMs, which can generate the description-based visual cues to help distinguish semantically similar concepts. Different visual cues are used to enhance discriminability from different perspectives, and the authors again use LLM to assign weights to different components for effective fusion. Extensive experiments on four different datasets are provided to demonstrate the effectiveness and interpretability.

Strengths

1. The proposed approach is theoretically sound and intuitive. Enriching the prompt from class-based to description-based can provide more information to enhance the relation sensitivity, and it also improves the explainability as the relation classification score can reveal the most important factors for the prediction. 2. The decomposition of subject-object pair makes it much more efficient for processing visual signals as the previous O(N^2) patches now reduce to O(N). The spatial relationship also makes sense as an abstract from real objects to just the relations. 3. The paper is well-written and easy to follow. The extensive experiments and ablation studies/visualization help a lot for understanding the model.

Weaknesses

1. The most important issues with this paper is that the evaluation section does not have important baselines. Specifically, in Table 1 and Table 2 the authors only show the performance of the proposed model with simplified version (CLS and CLSDE), which more like an ablation study. Many previous work actually attempted similar tasks and have been experimenting on the same dataset, e.g. [1][2][3]. 2. In Table 2 I guess the bolded numbers should be the highest (best)? For HICO-DET, CLS has the same performance on "Rare" category with RECODE thus should be highlighted as well I think? 3. A very very minor issue: the zero-shot chain-of-thought prompt used in most literatures are "let's think step by step" not "let's think it step by step". Formal usage should be "think" or "think about it" or "think through it", rather than "think it". [1] https://arxiv.org/abs/1804.10660 [2] https://arxiv.org/abs/1707.09423v2 [3] https://arxiv.org/pdf/2004.00436.pdf

Questions

Overall I think the paper is solid just that the evaluation is insufficient, as it doesn't have any comparison with previous methods. I will consider raising my scores if the comparison is provided in a revised version (I only listed a few papers in the weaknesses sections and I'm pretty sure they are not the latest ones, please compare against the most recent one/SOTA methods as it is more meaningful),

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

This paper doesn't discuss the limitations of the proposed methods, for example one important underlying assumption is the reliability of the LLM (responsible for decomposition and weight estimation) and the quality of the VLM model. I don't see obvious potential negative societal impact with this paper.

Summary

This paper presents RECODE, a novel method for zero-shot visual relation detection (VRD), designed to address the shortcomings of models like CLIP in distinguishing subtle relation categories and spatial discriminability. RECODE leverages large language models (LLMs) to generate detailed description-based prompts for each relation class component, thereby enhancing VRD performance. The authors also introduce a chain-of-thought method that breaks down the problem into smaller parts for LLMs, thereby assigning reasonable weights for each component. The effectiveness and interpretability of the method are demonstrated through experiments on four benchmark datasets.

Strengths

1. The approach introduces a novel framework, called RECODE, for zero-shot VRD that addresses the limitations of traditional class-based prompts. It decomposes the visual features of a triplet into subject, object, and spatial features and generates detailed descriptions of visual cues for each relation category. The use of chain-of-thought prompting for generating reasonable weights is a unique and creative approach. 2. The approach leverages large language models (LLMs), specifically GPT-3.5-turbo and CLIP, for the generation of descriptions and similarity calculations. The use of LLMs provides a strong foundation for generating informative and accurate descriptions of visual cues. The evaluation is conducted on four benchmark datasets, and the results demonstrate significant improvements over baseline methods. 3. The paper provides clear descriptions and explanations of the proposed framework, including the visual feature decomposing, semantic feature decomposing, and relation classification steps. The process of generating descriptions of visual cues and weights using LLMs is well-described, and the chain-of-thought method is illustrated with examples. The evaluation metrics and experimental setup are clearly presented. 4. The proposed approach addresses the challenge of zero-shot VRD by improving the discriminability of similar relation categories. By incorporating specific visual cues and generating descriptions, the approach enhances the performance of relation classification. The experimental results show significant improvements over baseline methods, demonstrating the effectiveness and interpretability of the proposed approach. The approach has the potential to advance the field of VRD and contribute to applications such as image understanding, scene understanding, and human-computer interaction.

Weaknesses

While the experimental results show improvements over baseline methods, the paper lacks a thorough analysis of failure cases. Understanding when and why the proposed approach fails to accurately predict relations is crucial for identifying its limitations and potential areas of improvement. Analyzing failure cases and providing insights into the challenges faced by the model would strengthen the evaluation and guide future research directions.

Questions

1. The paper mentions the incorporation of specific visual cues to improve the discriminability of related categories. Could you provide more details on how these cues were selected? What criteria were used to determine their relevance and effectiveness? Additionally, did you consider any alternative visual cues during the experimentation process? Exploring different visual cues and discussing their impact could provide further insights into the effectiveness of the proposed approach. 2. The paper demonstrates improvements in relation classification for zero-shot VRD, but it would be valuable to discuss the scalability of the proposed approach. How does the performance scale with an increasing number of related categories and visual concepts? Are there any computational or efficiency limitations that arise when dealing with larger datasets or more complex scenes? 3. The paper should provide a clear justification for the choice of evaluation metrics used to assess the performance of the proposed approach. Are there any limitations or biases associated with the selected metrics? Additionally, it would be helpful to include a discussion on the limitations of these metrics in capturing the true performance of zero-shot VRD models. 4. While the paper discusses the experimental results and improvements over baseline methods, it would be valuable to have a section dedicated to the limitations of the proposed approach. Identifying and addressing these limitations can help guide future research directions. Additionally, it would be beneficial to have a discussion on potential extensions or improvements to the proposed approach that could further enhance its performance or broaden its applicability.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. Auhors has not included the section or discussion on limitations and, if applicable, the potential negative social impact of their work.

Summary

Naively utilizing CLIP with prevalent class-based prompts for zero-shot VRD has several weaknesses, e.g., it struggles to distinguish between fine-grained relation types and neglects essential spatial information of two objects. To this end, the authors propose a novel method for zero-shot VRD: RECODE, which solves RElation detection via COmposite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, boosting performance in VRD. To dynamically fuse different cues, they introduce a chain-of-thought method that prompts LLMs to generate reasonable weights for different visual cues.

Strengths

- The framework for decomposing visual cues and using LLM to separately generate prompts for subject, object, and spatial features seems novel. - The proposed method shows noticeable performance improvements, and the authors provided an ablation study to solidly analyze the design choices of the proposed method.

Weaknesses

- The baselines in the experiments seem weak. Are the baseline methods recent enough models? To verify the effectiveness of the proposed method, the RECODE should be attached to the recent state-of-the-art model and show consistent performance improvement.

Questions

Please refer to the questions in the weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

I cannot find a potential negative societal impact in this paper.

Thanks for the additional experiments!

I've read the rebuttal from the authors and are satisfied with the response. I don't have additional questions and will raise my rating.