EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views

Summary

This paper describes an approach to estimate interaction between humans and objects from egocentric videos. To this end, head movement, 3D object meshes and 2D egocentric video are used as input and processed individually before being combined to predict contact regions in both object and subject. Experiments are carried out on Ego-Exo4D and GINO datasets, with additional annotations provided for both.

Strengths

+ novel model in which the flow of information makes sense + quantitative and qualitative results give a significant amount of insight into the method's performance and inner workings. It's clear that all modules contribute to the increased performance over baselines.

Weaknesses

- contributions of the paper a bit unclear, in terms of novelty over related work - critical discussion missing - paper hard to read Contribution: The approach seems to be heavily influenced by LEMON, and the comparisons in the experiment section are also predominantly with LEMON. Still, differences between the two models are not outlined. It is only when really reading through the text and the appendix that differences in architecture and (training) details because more clear. But it's left to the reader. Linking Figures 1-3 more tightly with the text can help to better understand the particular novelties of the method. Now, these figures are relatively disjoint from the story. Critical discussion missing: I would have appreciated a more critical discussion. While statements on Lavila's anticipation bias are reported, I don't see critical reflections on the performance reported for the main model. Still, given that human-object contact is modeled, the results depicted in Fig. 8 (for example) raise a number of questions, such as why the blade of the knife is identified (rather than the handle), or why the 3D model of the bottle is off, and why one of the two hands is not considered in the interaction with the bottle). Also, from the ablation study, the influence of various key components seems marginal. Leaving out many seemingly important terms still leaves a performance well higher than LEMON. Is there a very strong prior encoded? The experimental section gives me the feeling that the authors have been more critical to other works than to their own. This raises questions about the merits. Paper hard to read: I found the paper particularly verbose, and it seems the authors have prioritized colorful language over concise, precise descriptions. The result of that choice is that many parts, including abstract and introduction, are particularly hard to read. Given the complexity of the work and the sheer number of concepts and symbols, it's easy to get confused when structure and language are not to-the-point. As a result, it's difficult to understand the true contributions of the work. Evidently, there is a host of related work on capturing estimating contact in HOI, also from the first person perspective. What seems to be novel is the more connected processing of the inputs, including head motion. But I found little motivation into the specific design choices to do so. For one, it's not motivated why a geometry-semantics approach is not preferred. While it's clear that this only works for objects that have been considered during training, it seems the used data features objects that are rather standard, with affordances that can be well predicted.

Questions

1. What is the (technical) novelty of the current approach over related works (in particular LEMON)? 2. What is the influence of a learned prior on the contact regions of both object and subject? Could the authors show the variation in estimations across videos of the same interaction?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Partly. While some limitations have been addressed, it's clear that the system is rather specific in the way it's been evaluated.

Summary

This paper deals with the problem of inferring human-object interaction regions from egocentric views. It tackles the challenge of incomplete observations of interacting parties in the egocentric view by integrating the information from the visual appearance, head motion, and 3D object. It jointly infers 3D human contact and object affordance by exploring parallel cross-attention between the two parts. Moreover, 3D contact and affordance are annotated for egocentric videos collected from existing datasets to evaluate the task.

Strengths

1) The paper first tackles the task of estimating 3D contact and affordance from egocentric view by harmonizing multiple interaction clues including visual appearance, head motion, and 3D object. 2) A new framework is proposed to jointly infer human contact and object affordance regions in 3D space from egocentric videos. In the framework, object affordance and human contact are modeled through parallel cross-attention with gradient modulation to deal with the challenge of incomplete visual appearance in egocentric view. 3) To evaluate the new task, a dataset is constructed that contains paired egocentric interaction videos and 3D objects, as well as annotations of 3D human contact and object affordance. Extensive experiments are conducted on the constructed dataset to demonstrate the effectiveness of the proposed framework.

Weaknesses

Several important information regarding the technical details and the dataset are missing. For example, 1) Since a new dataset is constructed to evaluate the new task, the statistical information about the dataset (e.g., the proportations of interaction categories, ratio of contact and affordance annotations for different body parts and object categories) should be presented. 2) In Section 3.4, gradient modulation is achieved with learnable tokens to scale features to adapt to different scenarios. However, it is not clear whether these learnable tokens depend on input features. If yes, how? If not, then the tokens are fixed once the training is over and it is not possible to adapt to new scenarios during testing.

Questions

1) The latter part of equation (4) seems unnecessary and even confusion. Actually, it is easy to understand that scaling the features would affect the gradient on the model parameters. 2) The right part of Figure 3 is difficult to understand. More details are needed to explain the notations and procedure. 3) Writing needs further refinement. For example, a) line-158, Obejct -> Object b) line-186, ...that decoupled "decode" the feature... 3) line-233, punctuations error

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper investigates inferring 3D human contact and object affordance from a combination of egocentric video, human head motion, and 3D object point cloud. Inspired by real human behavior, which is based on visual observations, self-movement, and conceptual understanding, the authors propose a framework called EgoChoir. This framework utilizes modality-wise encoders to extract features from different input modalities and then infers object interaction concepts and subject intentions. The authors also construct a dataset comprising paired egocentric interaction videos (from EgoExo4D and GIMO) and 3D objects as the first test bed, demonstrating that EgoChoir outperforms existing baselines in inferring 3D Human-Object Interaction Regions.

Strengths

- The paper is well-written, with a clear and concise background and motivation. - The key idea is ingenious and intuitive, drawing inspiration from real human behaviors and making it easy to understand. - The empirical evaluation is comprehensive, covering a range of existing baselines and two recent datasets EgoExo4D and GIMO, demonstrating the effectiveness of the proposed approach.

Weaknesses

- The authors could leverage the existing annotations in EgoExo4D, such as 3D hand pose and scene annotation, to improve the quality of synchronization between the annotated affordance and video. Since most human-object interactions in EgoExo4D involve only two hands, utilizing these annotations could enhance the accuracy of human contact and object affordance annotation. - The authors mention that the egocentric scenes in the training and test sets are "almost non-overlapping," but the meaning of this term is unclear. Could the authors clarify whether "non-overlapping" refers to: no exact same videos, no same scenes or no same activities (e.g., training on cooking videos and evaluating on music videos). And how did they determine this train-test split. - It would be beneficial to provide more elaboration on the application examples of the proposed method, e.g. illustrating its potential use cases.

Questions

Please refer to the weaknesses part.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discuss the limitations in Section 5.

Summary

This egocentric paper, aims to capture 3D interactions such as human contact and objects affordance, to achieve this it uses head motion, 3d object and the visual appearance

Strengths

The add to Ego-Exo-4D the interaction data through a semi-automated process The approach appears to beat other works on this newly labelled data. There is extensive analysis of the results around the backbones

Weaknesses

How accurate is the semi-automated annotation process It's not clear where the right-hand side of Figure 3 is in the central system diagram The fusion of the modality-wise encoders is basically a few cross-attention layers It would be interesting to quantify the importance of each modality to the process

Questions

The work provides good performance and is clearly written, what is the key contribution to the model fusion network?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

nothing more than whats written above

Post-rebuttal feedback

Thank you for addressing my questions (and those of the other reviewers). I now understand better the contributions of the paper. I would really appreciate if those, and the other ambiguous issues I mentioned, can be clearly stated in the paper. That would make it much easier to read, and will facilitate the appreciation of the work. I have increased my rating.