Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

Summary

This paper proposes a framework for human-object interaction (HOI) detection that leverages vision-language foundation models and large language models to achieve universal and flexible recognition of complex interactions in images. The framework, named UniHOI, consists of three main components: a visual HOI detector that extracts three levels of features from images, a HO prompt-guided decoder that queries the foundation model for high-level relation representations associated with human-object pairs, and a knowledge retrieval module that uses a large language model to generate descriptive texts for interaction categories. The framework supports both supervised and zero-shot settings, and can handle any textual input for open-category interaction detection. The paper demonstrates the effectiveness and superiority of UniHOI over existing methods on two public benchmarks, HICO-DET and V-COCO, as well as in the wild scenarios.

Strengths

- The performance is quite impressive. - Leveraging LLM and foundational models to augment CV tasks is the future, and this work gives a try to use them simultaneously. - The authors promise to release the code to ensure reproducibility.

Weaknesses

1. There is no ablative study of each component (perhaps only one component, i.e., HO prompt decoder) under the close-set setup. 2. This work uses BLIP2 with ViT-L while existing work like GEN-VLKT typically uses CLIP with ViT-B. It is evident that the former is much more powerful. Could you provide the performance on HICO-DET using HO prompt-based learning with CLIP ViT-B under the close-set setup? As the improvement may be brought by more advanced large visual-language pre-trained models. 3. How about the inference speed? As shown in Table 8, **three** times longer than GEN-VLKT. Note that existing work like GEN-VLKT does not involve the computation of large visual-language pre-trained models at the inference stage, since all of the feature of objects or verbs is pre-computed. However, in this work, the feature for prompting should be computed for each image individually. Considering the extremely large backbone (e.g, ViT-L), there would be a heavy burden at inference. 4. The core contribution of this work is actually the HO prompt-based decoder. However, there is nothing novel with it, i.e., directly using spatial location to get foundational model output features. 5. Knowledge retrieval is solely used in open-world setup, is it possible for it to augment the closed-world setup? Overall, this is technically solid work, and LLM for knowledge retrieval is interesting. But the comparison is unfair (i.e., a much more powerful visual-language pre-trained model), the inference time is unacceptable, and the novelty of the prompt-based decoder is limited. I will be very happy to update my score if the authors can address my concerns above.

Questions

Please refer to weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

There is no discussion on limitations or failure cases.

Summary

In view of the limited scalability and the suboptimal zero-shot performance of current HOI detection methods, the authors propose a novel method for HOI detection based on VL foundation models. With in-depth analysis and adaptation of HOI detectors, the foundation model is effectively adopted to reason about HOI relationships based on human/object tokens. Furthermore, LLM is adopted as a knowledge base to diversify HOI descriptions, enabling open-vocabulary HOI detections. With the VL foundation model and LLM, extraordinary HOI detection performance is achieved upon both conventional and zero-shot setting.

Strengths

The proposed HO prompt-guided decoder is brilliant in addressing the feature alignment issue. Adopting GPT as knowledge base is an interesting idea to incorporate the recent progress in LLM with HOI detection. The performance is amazing with significant margins upon previous SOTAs, especially for zero-shot setting. Extensive experiments are conducted, providing valuable insights on the effect of VL foundation models in HOI detection. In the wild HOI detection illustration is quite impressive.

Weaknesses

The comparison between GEN-VLKT and the proposed method is not totally fair to me. It might be better to replace BLIP-2 with CLIP for a fair comparison. Fig. 2 is not very clear. It would help if annotating the encoders in the figure with corresponding notations. The baseline of ablation experiments is chosen as GEN-VLKT. However, a major difference between GEN-VLKT and UniHOI is the VL foundation model used. And there are also other differences, like VLKT is not used for UniHOI. It might be better to change the baseline to make the ablation more reasonable.

Questions

- The description on the adopted Image Encoder and Instance Decoder is not clear. Is DETR adopted? Or some more advanced detectors? Are they frozen during training? - In L177, $V^f$ and $V^f$ seems to be a typo. - The performance of UniHOI-l in Tab. 1-2 is questionable. Please check whether there are typos. - In the ablation studies, it is still not very clear that how the VL Foundation model is simply added. Is it a simple removal of HO Spatial Prompting? Or replacing the input feature of HO Spatial Prompting with the learnable queries? - Is it possible to make the HOI detector share the backbone of VL foundation model? This could be related to the proposed insight that HOI detector is a three-tier visual feature hierarchy. Ablation studies on this would be preferrable. - The proposed method seems to be applicable to arbitrary HOI detectors (if the answer to Q1 is yes). Is it practical? - Results in Tab. 5 and Tab. 3 are not consistent. The result without Knowledge Retrieval is reported in Tab. 3.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Limitations of the paper is not well-discussed in the paper. I would like to see more discussion on extending the use of LLM further than a static knowledge base. Also please refer to the weakness part.

Summary

This paper investigates the problem of human-object interaction (HOI) detection. The authors introduced UniHOI, a method for universal HOI detection in an open-world setting. They also explored the universal interaction recognition with Vision-Language (VL) foundation models and large language models (LLMs), and proposed HO prompt-based learning for high-level relation extraction aimed at VL foundation models. Experimental results show the effectiveness and significance of the proposed method.

Strengths

1. Overall, the manuscript is well-written and easy to follow. The figures are pretty and can convey the concepts clearly. 2. Pushing the problem of human-object interaction detection toward an open-world setting is of great importance. This is also a trend for most existing computer vision applications. 3. The extensive experimental results show the superiority of the proposed UniHOI method.

Weaknesses

1. In Table 4, the results from the third row come from "ConsNet [31]" but not "ATL [15]", according to the paper of "GEN-VLKT". 2. The conclusion part lacks objective reflections on the deficiencies of this study and future prospects for improvements.

Questions

Please refer to the weakness section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the weakness section.

Summary

The paper addresses human-object interaction (HOI) detection task. The authors propose a new method named as UniHOI, achieved by prompting BLIP2 using human-object paired features, as well as linguistic semantics generated by a LLM. The proposed UniHOI demonstrates significant performance gain on HICO-DET and V-COCO in both fullly supervised and zero-shot scenarios.

Strengths

If the BLIP2 is a fundamental model pre-trained on a lower-level task than HOI detection (e.g., classification, object detection), I would say this paper is an excellent work in terms of both model design and performance. Actually, I think this is the first work to transfer a fundamental model into the domain of HOI detection, which may open doors to further exploration on prompting learning for HOI. However, I cannot accept the choice of using BLIP2 as the fundamental model for HOI detection. I will explain the reasons for this in the weaknesses below.

Weaknesses

1. Prompting engineering aims to transfer a fundamental model pretrained on **lower-level tasks** to **higher-level** tasks. At a minimum, **the task for pre-training needs to be decoupled from the downstream task**. Otherwise, transferring a model pertrained on higher-level tasks to low-level tasks is not prompting, but fine-tuning. BLIP2 is a powerful model pertrained for VQA, image captioning, and similar tasks. However, as widely acknowledged, HOI detection is a sub-problem for these detailed scene understanding tasks. Namely, BLIP2 itself is a powerful HOI detector (I have tried using BLIP2 directly for HOI detection, and the performance is impressive). From this point, BLIP2 cannot be used as a fundamental model for HOI detection since it has a great capability for HOI detection by itself, and is capable of even higher-level tasks. Therefore, this paper is more like a work that fine-tunes BLIP2 on HICO-DET and V-COCO, at the cost of giving up the ability to use BLIP2 for other tasks, e.g., captioning. 2. While direct use of BLIP2 for HOI detection may fail to achieve as impressive performance as that of UniHOI on HICO-DET and V-COCO, **BLIP2 has already achieved the goal of doing HOI detection, i.e., detailed scene understanding**. Therefore, is it a case of putting the cart before the horse to use BLIP2 for HOI detection only? 3. In a real open-world scenario, I think BLIP2 is more capable of HOI detection compared to UniHOI. After all, the zero-shot HOI detection capability of UniHOI is mainly inherited from BLIP2. 4. The performance of UniHOI on HICO-DET and V-COCO is impressive. I think this is the first work that achieves a mAP being larger than 40% on HICO-DET. However, the comparison is not so fair. As aforementioned, BLIP2 itself is a powerful HOI detector, which has been pre-trained with a large amount of **interaction-specific** data. Note that, **on a fair comparison, a fundamental model should not be pre-trained using data with annotations involving downstream tasks**. Otherwise, the authors need to report the results without using these extra data. For instance, if we first collect all data involving HOI detection from the dataset used for BLIP2 pre-training. Next, we use these data to pre-train a HOI detector listed in Table 1, (e.g., GEN-VLKT) to get GEN-VLKT-2. Finally, we fine-tune the GEN-VLKT-2 on the HICO-DET and V-COCO. I think it can also get an excellent performance. This is another reason why I think that transferring a model pre-trained on a higher-level task to a lower- level task (especially when the lower-level task is a sub-task of the higher-level task) is not a prompting, but a fine-tuning.

Questions

Actually, I am very much looking forward to the work of using fundamental models for HOI detection. However, I think an instrive work is to transfer a fundamental model pretrained on lower-level task to a high-level task. This work, however, seems to be the opposite.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes a universal HOID pipeline, which utilizes decoded ho-pair feature as spatial prompts to prompt the VL foundation model with the aim to implement effective prompt-based learning on base VL model and extract HOI related features from it. It also proposes knowledge retrieval for HOID in open-category manner by large-scale pretrained language models. Experiments show the effectiveness of this approach in both generic and zero-shot HOID.

Strengths

1. It explores universal interaction recognition by tansferring the rich knowledge inside Vision-Language foundation models and LLMs to HOI pipeline, which broaden the research scope of HOID. 2. The experimental results are promising. In both generic and zero-shot setting, this approach reaches a new state-of-the-art and surpasses previous methods by a substantial margin.

Weaknesses

1.In line152, ‘P_h’ and ‘P_o’ are described as ‘excellent spatial position features’ and further utilized as HO spatial prompts. However, these features are generated by learnable position embedding and query, which is identical as many transformer-based HOID approaches before such as GEN-VLKT. Can you provide some evidence that these feature are indeed ‘excellent’, why could it provide accurate spatial information concerning ho pairs? Or are there some unique designs I overlook? 2.In Line 177, HOPD is designed for the alignment issue between VL foundation model and HOI pipeline. And the output V_f is incorperated with V_i for final predicition of interaction. But the performance of pure V_f, i.e., only utilizing V_f for prediction is unexamined. This results may more directly show the effectiveness of alignment between VL models and HOI pipeline. 3.Some typos. Line 171, ‘the guidance of’ repeated twice. Line 177, ‘V_i’ is mismarked as ‘V_f’.

Questions

Majorly the questions lies in the choice and interpretability of spatial prompts. It’s unclear why these prompts are ‘excellent spatial position features’. Or have you try some experiments on the choice of these prompts?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

None

Further Discussion with Reviewer WxZv

Dear Reviewer WxZv, We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have carefully addressed your comments and provided corresponding responses and results. We believe that these responses and results adequately address your concerns. We would value an opportunity to further discuss whether your reservations have been resolved. Should there remain any aspects of our work that are unclear to you, please do not hesitate to inform us. Once again, thank you for your invaluable feedback. Best,

Further Discussion with Reviewer X6bU

Dear Reviewer X6bU, We sincerely appreciate the time you devoted to reviewing our manuscript and the invaluable feedback you provided. We have diligently addressed your comments and provided corresponding responses and results. We believe that these responses adequately address the concerns you raised. We would be grateful for an opportunity to discuss whether your reservations have been resolved. Should there be any aspect of our work that remains unclear, please do not hesitate to inform us. Once again, thank you for your constructive insights. Warm regards,

Further Discussion with Reviewer djfM

Dear Reviewer djfM, Thank you for the time and effort you dedicated to reviewing our manuscript. We sincerely appreciate your valuable feedback. In response to your comments, we have provided thorough explanations and updated results. We believe that these address the concerns you raised. We are eager to ensure that all of your concerns have been addressed adequately. Should there be any aspect of our work that remains unclear to you, please do not hesitate to inform us. Once again, we extend our gratitude for your constructive feedback. Best,