Neural-Logic Human-Object Interaction Detection

Summary

This paper introduces a novel Human-Object Interaction (HOI) detection method called Logic-Guided Transformer Reasoner (LoTR). Unlike standard Transformer-based HOI detectors that accept pre-composed human-object pairs, LoTR leverages Transformers to infer plausible interactions among entities, thus enabling the creation of novel interactions. This is achieved by modifying the self-attention mechanism to reason over human-action-object triplets. The reasoning process is guided by two critical elements of HOI: affordances and proxemics, expressed in first-order logic and grounded into a continuous space. These adaptations enhance the learning of LoTR, leading to improved performance and zero-shot generalization capabilities. Evaluations on V-COCO and HICO-DET datasets demonstrate LoTR's superiority over existing methods under normal and zero-shot conditions.

Strengths

+ A novel perspective for addressing problems in the Human-Object Interaction (HOI) domain, which is inspiring. + An abundance of experimental evidence demonstrates that the method has yielded impressive results. + The logic-guided learning method in section 3.3 of the paper is interesting and can provide assistance for the development of other computer vision tasks.

Weaknesses

- Although the method has great advantages and is thought-provoking in the HOI (Human-Object Interaction) field, its application seems somewhat limited. Can it be applied to other visual tasks? If so, I am really looking forward to such interesting things happening.

Questions

Please refer to the weaknesses.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

I do not find any negative societal impact in this work.

Summary

The paper introduces a novel approach to tackle the problem of human-object interaction detection. It presents two main contributions. Firstly, there is a revisited self-attention mechanism within the decoder that focuses on the interaction queries. This aspect has received limited attention in the field. Secondly, the paper introduces a logic-based reasoner that employs affordances and proxemics to filter out infeasible interactions. This reasoner replaces conventional hard-coded rule-based logic with differentiable operations. In my opinion, these technical advancements are original and are likely to be of interest to researchers in this field. Furthermore, experimental results on HICO-DET and V-COCO datasets demonstrate state-of-the-art performance.

Strengths

- Most Transformer-based HOI detection methods rely on the vanilla self-attention mechanism borrowed from DETR [59]. DIfferently, this paper introduces a reasonable enhancement to better adapt for the combination nature of human-object interactions. The idea is original and likely to be of interest to researchers in this field. - Some prior methods (e.g., [15][49]) have utilized rule-based logic to filter out infeasible human-object interactions, resulting in improved performance. This paper takes a further step by converting the rule-based logic into differentiable operations, which is very interesting. - The paper is generally well-written. Readers with a relevant background will have no difficulty following the paper. - The proposed method demonstrates significant improvements over existing approaches on both general and zero-shot settings.

Weaknesses

There are a few minor weaknesses in the experimental section that should be addressed: - It remain unclear whether the proposed Transformer reasoner can be applied to common HOI detectors and contribute to performance improvements across different methods, or if it is specifically designed for the approach in this paper. To provide a comprehensive understanding, it would be valuable for the paper to include additional results such as GEN-VLK+reasoner, QPIC+reasoner, CDN+reasoner, and others, to showcase the potential benefits of integrating the reasoner into other existing methods. - In Section 4.2, there is a need for more in-depth analysis explaining why the proposed method can achieve better results on unseen objects and unseen verbs. Although an analysis on the reasoner is presented in L344-349, it remains unclear why TRA without the reasoner can also achieve state-of-the-art performance on the zero-shot setting, as shown in Table 4. - Regarding the baseline results in Table 3, it appears that the baseline method has already achieved better performance compared to many existing methods. The baseline does not seem to be a vanilla model with the standard Transformer encoder-decoder architecture. It would be helpful to address these concerns by providing additional implementation details. Typos: - L333: examine the efficiency of … -> examine the effect of …? - L45, L47: Fig.2 -> Fig.1 - Eq.(7): $\neg h_M(x)) \rightarrow \neg h_N(x)$

Questions

Please refer to the above section for further details. It would be greatly appreciated if the following questions could be addressed: - What are the potential benefits of integrating the reasoner into existing Transformer-based methods? - A more in-depth analysis explaining why the proposed method achieves improved results on unseen objects and unseen verbs. - Implementation details regarding the baseline method (as indicated in the first row of Table 3)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The limitations have been discussed in the supplementary materials. The primary limitation arises from the computational complexity of the proposed method.

Summary

This paper introduces LoTR: Logic-Guided Transformer Reasoner for Human-Object Interaction (HOI) detection that adapts the self-attention mechanism in standard Transformers to infer feasible interactions over entities, guided by affordances and proxemics.

Strengths

none

Weaknesses

1-The meanings of the notations in the paper are confusing. For instance, in Equation 2, the meaning of $W^{v\'}$ is unclear, and there is no clarification provided in the text. It would be helpful if the authors could elaborate on the meaning of $X_i$. Since Equation 2 forms a significant contrast to the proposed triplet-reasoning attention (Equation 5), it should be thoroughly explained for improved readability. 2-In Equation 5, the authors derive $X_{ij}$. It would be beneficial if they could provide a detailed explanation of how $X_{ij}$ differs from $X_i$. 3-In Line 213, $h_N$ should be $h_M$. Furthermore, Equations 7 and 8 both have an extra right parenthesis. 4-The method of designing a loss function using logical reasoning is interesting, but it seems rather complex. Could the authors use more intuitive language to describe $G_{v,p}$ and $G_{o,p}$? 5-The introduction of logic-induced affordances and proxemics, in essence, imposes additional constraints on spatial positions within HOI. Could this lead to overfitting on these two datasets, thereby reducing the model's generalizability on a broader range of data? For example, I don't fully agree with the statement in Line 238, "if there is a low possibility of a horse and the human box is above the object box, then the reasoning results should not be 'human-feed-horse.'" This scenario is not entirely implausible as a person could feasibly feed a horse from atop it.

Questions

How to obtain the set of infeasible interactions (i.e., triplet ⟨human, action, object⟩) {h1, · · · , h_N} and {h1, · · · , h_M}?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

none

Summary

The paper addresses the problem of human-object interaction (HOI) detection. The authors introduce a new "triplet-reasoning attention" mechanism that enumerates all possible "human-action-object" triplets. Meanwhile, the authors propose a new logic-guided loss function, which enables an interaction detector with stronger reasoning capability. The new method (LoTR) is evaluated on HICO-DET and V-COCO, achieving competitive results.

Strengths

I believe that the authors may introduce some new points to the domain of HOI detection, especifically the logic-guided learning strategy and the proposed LoTR demonstrates excellent performance. However, the paper is really hard to follow due to the poor presentation.

Weaknesses

- In Eq. 4, $\mathbf{Q}^h \in \mathbb{R}^{N_h \times 1 \times D}$ and $\mathbf{Q}^a \in \mathbb{R}^{1 \times N_a \times D}$, why $\mathbf{Q}^h + \mathbf{Q}^a \in \mathbb{R}^{N_h \times N_a}$ - The $\mathbf{X}$ are *inputs* (line 186) in Eq. 5 and *outputs* (line 191) in Eq. 6. According to my understanding，the $\mathbf{X}$ in Eq.5 are the outputs of "triplet-reasoning attention" while the $\mathbf{X}$ in Eq.6 are the outputs of the decoder, i.e., the outputs of cross-attention (omitted residual connection and FFN). - In Eq. 7 and Eq. 8, what is $x(\cdot)$ and what is x? is it a function or a variable? In Eq. 7, what is $M$? And it is really hard for me to understand L214-217. In line 213, $\{ h_1, h_2,\cdots, h_N\}$ are infeasible interactions, which yet seem to be reasonable interactions in line 217 (e.g., "human-launch-boat"). - Eq. 11 and Eq. 12 need more detailed description. What is $M$ in Eq. 11 and what is $N$ is Eq. 12? Is $h_m$ different from $h_n$? What is $x_k$? The action-position loss, $L_{v,p} = (1-G_{v,p}) \propto {1-s_k(v)[1-s_k(h_m)]}$, which decreases as $s_k(v)$ increases and $s_k(h_m)$ decreases. It can be intuitively interpreated that when an action *v* gets a high confidient score, then the interaction $h_m$ should get a low score. How to understand it?

Questions

1. Acorrding to Line 180-182, the "triplet reasoning attention" actually enumerates all human-action-object combinations ($N_h \times N_a \times N_o$), in which most are negative proposals. 2. The number of queries in this paper is $N_h \times N_a \times N_o = 32^3 = 32768$. However, in pervious approaches, the number of queries is normally less than 100. So the computational complexity should be reported.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

2 fair

Limitations

Yes, the authors have addressed the limitations.

Summary

The paper presents LoTR, a method for detecting human-object interactions (HOI) that utilizes Transformer as a reasoning mechanism to model the interactions between people, objects and their interactions. By feeding the self-attention with different feature combination as key query and value, LoTR can infer triplets and establishes meaningful interactions. The paper demonstrates the effectiveness of LoTR through experimental results on two widely used datasets, showcasing its strong performance.

Strengths

- The paper is well written and well motivated. - The idea of Triplet-Reasoning Attention is well motivated and seems working as expected. - The results are comprehensive (I do have concerns on the results as well, please see section below).

Weaknesses

1. Some of the claims may lack sufficient support or appear too strong. For instance, in lines 30-33, it is advisable to provide citations or evidence to substantiate the claim. It seems contradictory to assert that the model can accurately identify the correct pair while lacking comprehension of them. 2. The comparison of results in Table 1 and Table 2 lacks fairness. As indicated in line 249, this paper utilizes the CLIP model, whereas most of the baselines are trained on significantly smaller datasets. The authors should establish a fair comparison by employing the exact same backbone or explicitly state the pre-training set used for the backbones. 3. Additional discussion and analysis are necessary. For instance, when the proposed model performs worse than STIP on V-COCO, what could be the potential reasons? Currently, the paper criticizes STIP for its performance on other datasets but fails to explain why the proposed method does not generalize as well to V-COCO. 4. In experimental results section, more details are required. For example, what does the bolded text signify in Table 5? Additionally, when the authors claim that they selected a three-layers decoder for improved efficiency, what are the corresponding FLOPs or latencies associated with different configurations?

Questions

Please see my comments above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None

Thanks for the rebuttal

The rebuttal addressed most of my concerns, please include the additional information into the final paper. I will raise my rating.

Looking forward to your further discussion

Dear Reviewer AydW, Thank you again for your kind review and comments. We hope we have addressed all of your concerns. We wonder if there are any additional comments. We sincerely hope that we will be able to use the remaining time to engage in an open dialogue with domain experts to enhance the quality of our work. Thanks again, Authors

Looking forward to your further discussion

Dear Reviewer jaUK, Thank you again for your kind review and comments. We hope we have addressed all of your concerns. We wonder if there are any additional comments. We sincerely hope that we will be able to use the remaining time to engage in an open dialogue with domain experts to enhance the quality of our work. Thanks again, Authors

Looking forward to your further discussion

Dear Reviewer gPtK, Thank you again for your kind review and comments. We hope we have addressed all of your concerns. We wonder if there are any additional comments. We sincerely hope that we will be able to use the remaining time to engage in an open dialogue with domain experts to enhance the quality of our work. Thanks again, Authors

Looking forward to your further discussion

Dear Reviewer uwRp, Thank you again for your kind review and comments. We hope we have addressed all of your concerns. We wonder if there are any additional comments. We sincerely hope that we will be able to use the remaining time to engage in an open dialogue with domain experts to enhance the quality of our work. Thanks again, Authors

Please respond

Dear AydW, Given you are in the minority with a reject it is critical that you read the rebuttal and inquire about any outstanding issues you feel are still not addressed. Thank you, AC

Response Needed

Dear uwRp, Given you are in the minority with a reject it is critical that you read the rebuttal and inquire about any outstanding issues you feel are still not addressed. Thank you, AC