Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models

Summary

This paper introduces DIFFUSIONHOI, a novel Human-Object Interaction (HOI) detector that utilizes text-to-image diffusion models for HOI detection. It efficiently focuses on complex relationships between objects, providing a strong basis for HOI modeling. The relation-driven approach enhances image generation capabilities for HOI detection, enriching training samples for rare interactions. Additionally, it improves detector flexibility and accuracy, achieving good performance on HICO-DET and V-COCO benchmarks.

Strengths

- The experiments are comprehensive. - The design of Inversion-Based HOI Modeling and Relation-Driven Sample Generation are intriguing.- - Achieving zero-shot generalization on SWiG-DET is impressive.

Weaknesses

- Despite having fewer trainable parameters overall, DIFFUSIONHOI has a large total parameter count, which raises concerns about unfair comparisons with existing methods. - The authors claim to set a new state-of-the-art on HICO-DET, but the reviewer found that there are better models available, as also mentioned in the paper [79]. - Quantitative results and analysis of failure cases are needed. [79] Frederic Z Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context in detecting of human-object interactions. In ICCV, 2023

Questions

N/A

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This paper introduces DIFFUSIONHOI, a new HOI detector leveraging text-to-image diffusion models. Unlike previous one-stage or two-stage models, diffusion models excel at discerning mid/low-level visual concepts as generative models and possess strong compositionality to handle novel concepts expressed in text inputs. To steer the focus of diffusion models from instance generation to the relationships between humans and objects, this paper exploits textual inversion and devises a human-object relation inversion strategy grounded in the disentanglement of HOI. Furthermore, to transfer extensive knowledge from large-scale diffusion models to assist in recognizing interactions, the paper leverages both text-prompted image generation and conditioned feature extraction capabilities of diffusion models. By embracing text-to-image diffusion models and facilitating relation-driven image generation and prompting, this method demonstrates superior performance.

Strengths

This paper is well-written and easy to follow. It proposes a new diffusion-based solution, DiffusionHOI, for the human-object interaction (HOI) task. Unlike traditional one-stage and two-stage methods, DiffusionHOI benefits from controllable image generation and HOI knowledge transfer from diffusion models. Without making design changes to the HOI decoder, this method achieves superior performance compared to previous state-of-the-art models by deriving HOI-relevant features to assist in HOI detection.

Weaknesses

1. I am curious about the number of parameters in the VQGAN used as the backbone of DiffusionHOI. According to Table 1, the authors use ViT-L as their backbone, which appears to be significantly larger compared to previous works, especially those using the Res50 network. Could the authors provide more results using a smaller backbone, such as ViT-B, to offer a clearer and fairer comparison between DiffusionHOI and previous state-of-the-art models? 2. Another concern is that the improvement primarily stems from the additional knowledge provided by the powerful diffusion model. Could this be considered unfair when comparing it to previous works that do not have this significant benefit? In this case, how does this approach differ from simply adding more data for training? As I am familiar with HOI but not as familiar with diffusion models, could the authors provide more details about the data scale and parameter scale of the diffusion model used in this paper? Additionally, I look forward to feedback from other reviewers on this matter.

Questions

Please refer to weakness.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Please refer to weakness.

Summary

This paper tackles the human-object interaction (HOI) detection task. It aims to utilize the feature of generative models like diffusion models to help human-object interaction classification. More specifically, it utilize the inversion process in the diffusion model to learn the embedding for human-object interaction. The proposed method obtains obvious improvement over existing methods.

Strengths

- The improvement on performance is significant compared with the baseline. - The figures are illustrative and helpful for understanding this paper. - The writing and overall demonstration is good. The problems to tackle are clearly explained, for example, in line 39-61. This does affect my rating. - The ablation study is though and well organized.

Weaknesses

- The cost of training with the inversion process. As the inversion process is computationally costly and slow, it could affect the training speed. I would like to see more clarification or analysis over the cost of the proposed method. - The proposed model does not achieves SOTA as it claims. For example, it falls behind PViC (iccv23) with swin-L backbone by a large margin. As the proposed method have a ViT-L backbone, I think it is not fair to not mentioning PViC-swin-L and some other methods. Besides, PViC utilize no external knowledge like CLIP or SD while this method does. - The general technical design is complicated. This is not a big problem and does not affect my rating.

Questions

- The analysis of the cost. I will look into the rebuttal and adjust my rating accordingly.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

Discussion over the limitation is provided.