EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection

Summary

In order to solve the HOI detection problem in a zero-shot setting, the authors propose the Unseen-class Text Prompt Learning module. Using learnable visual prompts and textual prompts, it effectively utilizes the knowledge from large language models and Vision-Language Models and performs well in unseen class HOI detection.

Strengths

1.The work is innovative, well-written and has clear diagrams. 2.The authors' proposed learnable prompts scheme and UTPL module are novel.

Weaknesses

1. In Qualitative Results, the authors use Figure 4 for visual illustration, but it is not possible to conclude from Figure 4 that "MaPLe tends to predict seen classes with high confidence scores". The authors should have given a clearer and more accurate illustration of the figure. 2. The authors claim to have "employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. prompt learning." However, the information provided by the LLM may be misleading or inaccurate, did the authors filter the information? If not, what can be done to avoid HOI detection being negatively affected from the information provided by LLM? 3. For the UTPL module, the authors split the LLM-generated description into multiple sentences; how do they control the length of individual statements? 4. The author mentions: " A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5.", why is the threshold value chosen to be 0.5 instead of 0.7 or other? What is the basis for the choice? 5. WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear.

Questions

1. In Qualitative Results, the authors use Figure 4 for visual illustration, but it is not possible to conclude from Figure 4 that "MaPLe tends to predict seen classes with high confidence scores". The authors should have given a clearer and more accurate illustration of the figure. 2. The authors claim to have "employed an LLM to provide the nuanced differences between related seen and unseen classes, improving our method for unseen class prompt learning. prompt learning." However, the information provided by the LLM may be misleading or inaccurate, did the authors filter the information? If not, what can be done to avoid HOI detection being negatively affected from the information provided by LLM? 3. For the UTPL module, the authors split the LLM-generated description into multiple sentences; how do they control the length of individual statements? 4. The author mentions: " A prediction was deemed a true positive if the HOI classification was accurate and the Intersection over Union (IoU) between the predicted human and object bounding boxes and the ground-truth bounding boxes exceeded 0.5.", why is the threshold value chosen to be 0.5 instead of 0.7 or other? What is the basis for the choice? 5. WRITING DETAILS: Abbreviated nouns should be introduced the first time they appear.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

4

Limitations

Yes

Summary

This work presents EZ-HOI, an innovative framework that addresses the zero-shot HOI detection challenge by employing prompt learning. It integrates LLM and VLM guidance to enrich prompts and adapt to HOI tasks effectively. By learning from related seen classes, EZ-HOI overcomes the limitation of lacking labels for unseen classes, enhancing its performance on them. The framework achieves state-of-the-art results with significantly fewer trainable parameters than existing methods, showcasing its efficiency and effectiveness in zero-shot HOI detection.

Strengths

- The paper is well-written. - The Unseen Text Prompt Learning is well-designed. - Overall, the zero-shot results with less trainable parameters are good.

Weaknesses

- This paper primarily addresses the improvement of zero-shot HOI detection benchmark performance by training with the pre-definition of all HOI classes. This creates an unfair comparison with previous work. - In light of the above, the reviewer believes that the authors should provide experimental results where the pre-definition of all HOI classes is unknown during training, allowing for a fairer comparison with previous work.

Questions

N/A

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper investigates the problem of human-object interaction (HOI) detection. This paper introduces EZ-HOI, a method for efficient zero-shot HOI detection in an open-world setting. EZ-HOI also explores the use of LLM and VLM guidance for learnable prompts to enhance prompt knowledge and aid in adapting to HOI tasks. To better adapt to unseen classes, EZ-HOI establishes relationships between unseen and seen classes.

Strengths

- The state-of-the-art experimental results in the zero-shot setting. EZ-HOI demonstrates good performance in zero-shot scenarios. More importantly, by leveraging prompt tuning, the number of trainable parameters in EZ-HOI is significantly smaller compared to other methodologies. - The code in the supplementary material is provided to ensure the reproducibility of the study.

Weaknesses

- The overall method is straightforward but lacks sufficient novelty. Compared to UniHOI, which also claims to utilize Spatial Prompt Learning, the main difference in EZ-HOI lies in its use of more traditional prompt learning methods (like VPT-DEEP) and its additional modeling of relationships between unseen and seen classes when handling unseen categories. Since many previous works have already demonstrated that prompt tuning (ViT-adapter/CLIP-adapter) can achieve even higher performance than full fine-tuning, the claims of EZ-HOI seem somewhat weak. Additionally, modeling the relationship between unseen and seen classes as a technical contribution also appears somewhat limited. - The fully-supervised setting results are provided in Appendix Table 6, but the performance is worse than Uni-HOI. Firstly, this is a very important experiment and should be included in the main text. Secondly, it appears that under full supervision, EZ-HOI performs worse than Uni-HOI, which contradicts the claims in the main text. Have the authors provided an in-depth analysis of the reasons? If the performance difference is due to parameter count, what would the performance be with full fine-tuning? - The performance improvement seems to come from the baseline rather than the proposed modules. As shown in the ablation study (Table 4), the baseline of EZ-HOI achieves 37.44 on seen categories, which is already state-of-the-art (SOTA). I wonder what EZ-HOI’s baseline is and why it can achieve such strong performance on seen categories. Additionally, considering that the UTPL module increases unseen category performance by two points, and EZ-HOI shows a two-point increase compared to Uni-HOI, does this mean that EZ-HOI’s performance gain over Uni-HOI is primarily due to the UTPL module?

Questions

What is the fully fine-tuned performance? What is the baseline?

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

1

Limitations

This submission discusses limitations.

Summary

The paper proposes a novel prompt learning framework for zero-shot Human-Object Interaction (HOI) detection, which enhances the generalizability of Vision-Language Models (VLMs) by interacting with Large Language Models (LLMs) to obtain descriptions.

Strengths

1. The proposed method appears logical and achieves general improvements in performance. 2. Utilizing LLMs to generate descriptions is a promising direction. 3. The paper is well-written, and the framework diagram is clear.

Weaknesses

1. The prompt design seems to be a hybrid approach, combining visual prompt learning [1, 3] with text prompts, such as those used in COCOOP, without a specific focus on task-specific configurations. It is difficult to justify the necessity of this work for the field. 2. Despite the claim of improved performance on novel classes, the method performs worse on unseen classes compared to CLIP4HOI, while it shows improved performance on seen classes. Given that novel class performance is crucial, this result seems not consistent with this task. 3. W_down presents in both Equation 3 and Equation 5 is puzzling. Equation 5 attempts to integrate descriptions for unknown categories with similar known categories and LLM-generated details. More details on the training process would be helpful, as it appears that data training might be lacking since unknown categories are not included in the training. 4. There is substantial related work on using LLMs for new class descriptions in zero-shot scenarios, such as CuPL [2] for classification and DVDet [3] for detection, which use attribute decomposition to aid in categorization. DVDet also generates more discriminative descriptors by distinguishing confusing categories. The paper lacks a discussion on such related LLM-based works. It would be interesting to see if using action descriptors might also yield some benefits. 5. The paper could benefit from providing more visualization results to demonstrate the effectiveness of both general and discriminative descriptions generated by the method. [1] Visual Prompt Tuning [2] VISUAL CLASSIFICATION VIA DESCRIPTION FROM LARGE LANGUAGE MODELS [3] LLMS MEET VLMS: BOOST OPEN-VOCABULARY OBJECT DETECTION WITH FINE-GRAINED DESCRIPTORS

Questions

Please see the weakness.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

The paper studies the challenges behind successful adaption of pre-trained vision language models (VLMs), i.e., CLIP, to the problem of zero-shot Human Object Interaction (HOI) detection. Specifically, during finetuning, VLMs overfit to seen HOI classes observed during the training on HOI training data, preventing successful transfer on the unseen HOI classes. To overcome the aforementioned problem, authors propose prompt tuning mechanism coupled with guidance from a large language model (LLM). In particular, LLM is used to generate (i) descriptions of each HOI class; (ii) explain the difference between each unseen and the corresponding closest seen class to facilitate transfer on unseen classes. The prompt tuning technique itself learns a set of prompts that are shared between vision and text encoders. Authors employ attention mechanism to produce modality specific prompts via attending on class descriptions for text modality and via attending on the image embedding for visual modality correspondingly. Authors evaluate the proposed approach on the standard HOI benchmark and compare it to the recent baselines, showing improvements in terms of the parameter efficiency and the performance on unseen classes.

Strengths

* The problem of enabling generalization to unseen classes during VLMs adaptation is important, even outside HOI detection field * To my knowledge, learning shared prompts and producing modality specific prompts via attention mechanism is novel * The evaluation is thorough, both comparing to HOI specific prompt tuning baselines and to general purpose prompt tuning approaches such as MaPLe.

Weaknesses

My main concern is that the current narration of the proposed methodology lacks intuition and structure. For example, **Structure** * Section 3 starts with mentioning that the method will learn prompts per layer, and the subsequent section also use notation considering N layers, but the first time the reader encounters interaction with the layers is Section 3.3. It makes the paper hard to follow. **Intuition** * The main idea of the approach builds on repeatedly applying MHCA for different purposes, however these equations just stated on the high-level in Eq. (2); (3); (5) and without any intuition behind the design. * Similarly to the above, Eq. (6) and Eq. (8) state how these learnable prompts are used in the corresponding encoder layers, however the narration is on the high level and does not elucidate the particular idea behind the equations.

Questions

* How $N=9$ was chosen? Can you provide ablations for different number of $N$? * Similarly to the question above, Eq. (6) and (8) suggest that the learnable prompts are inserted in the first $N$ layers of textual and visual encoders, correspondingly. What is the motivation behind this design choice? I, in general, would assume that earlier layers already consumed broad knowledge during CLIP pre-training and task specific adaptation is required for penultimate layers. Can you please provide ablations on different positioning of the learnable prompts? * Currently, the paper narration mainly focuses on tackling generalization to unseen classes and, indeed, experimentally confirms the improvements offered by the proposed approach. The main component causing such improvements is seem to be UTPL that employs guidance from LLM. However, according to Table 4, other components also bring substantial improvements. Can you please provide the intuition behind other components' role and how they can bring these improvements?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Authors discuss the limitations of the current zero-shot HOI detection setting. In particular, that to truly enable zero-shot HOI detection, one must exclude the assumption of having in advance a set of unseen HOI classes.

Summary

This paper tackles zero-shot HOI detection via prompt tuning. To address the challenge posed by the absence of novel classes, the authors first incorporate LLM and VLM guidance to enrich learnable prompt tokens. Further, the authors utilize LLM to provide nuanced differentiation between unseen classes and their related seen classes, termed Unseen Text Prompt Learning (UTPL), to alleviate overfitting to seen classes. In experiment, the proposed method achieves state-of-the-art performance under major zero-shot HOI detection settings while costing fewer training resources.

Strengths

1. This paper tackles an important issue in zero-shot / open-vocabulary learning, i.e., overfitting to seen classes. 2. The proposed method effectively explores guidance from large foundation models, including VLM and LLM. 3. The authors achieve state-of-the-art performance with significantly fewer training resources.

Weaknesses

1. It seems the authors assume the prior knowledge of unseen class names when training the UTPL module, which conflicts with the zero-shot setting. 2. In Table 1, the proposed method underperforms CLIP4HOI (ResNet50+ViT-B) in unseen mAP although it is claimed to tackle the overfitting issue especially.

Questions

1. In Figure 2, are the VLM visual and visual encoder the same thing (i.e., CLIP)? 2. Why did the authors use LLaVA for text description instead of choosing pure language models?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

N/A

Acknowledgement

I thank the authors for the provided clarifications. After reading the other reviewers responses and the rebuttal I will maintain my positive score. I strongly suggest authors to include the provided clarifications in the future revision of the manuscript. Given that, partially, the proposed approach employs well-established techniques in the prompt tuning community, I also believe that the paper will greatly benefit from expanding discussion and positioning with respect to these existing techniques.

kicking off reviewer-author discussion

Thanks the authors for the rebuttal, and reviewers' for the comments. We are in the period of author-reviewer discussion. - Reviewers, please read through all the reviews and rebuttal and see if authors' responses have addressed your and others' important questions. - Authors, please follow up with any reviewers' further comments. Cheers, AC

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Hi dear reviewer, Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments. Thanks, AC

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Hi dear reviewer, Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments. Thanks, AC

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Hi dear reviewer, Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments. Thanks, AC

Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments.

Hi dear reviewer, Please read through rebuttal, and see if the rebuttal has addressed your concerns or you have further comments. Thanks, AC