Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos

Summary

The paper addresses challenges in using egocentric videos for robotics tasks, specifically the issues of occlusion and visual mismatch between the human hand and a robot end effector. To address these problems, this work proposes a factored representation of the scene that separates the agent (human hand) from the environment. This factorization is achieved via the proposed Video Inpainting via Diffusion Model (VIDM). Experiments demonstrate the effectiveness of VIDM in improving the inpainting quality and highlights the power of the factored representation for various downstream robotics tasks.

Strengths

1) Paper is well-written and well-structured; the descriptions of the experimental protocols for each of the applications was very helpful. 2) Experiments are very comprehensive; I was happy to see evaluation on both the inpainting quality achieved by the proposed model, as well as the utility of its associated representations for various downstream robotics tasks.

Weaknesses

1) The main weakness that stands out to me is that the current set of downstream robotics applications are not as convincing as they need to be. Specifically, in many of the tasks, it seems the inpainted environment $I_{t}^{\text{env}}$ is the key component, and not the factorized representation in its entirety.

Questions

1) It seems mostly $I_{t}^{\text{env}}$ is being used; where does the agent representation come into play? In application 3, I assume you're using $I_{t}^{\text{env}}$ to predict the GT hand pixels, so $I_{t}^{\text{agent}}$ is unused? And in applications 4 and 5, the function $g$ which abstracts $I_{t}^{\text{agent}}$ simply returns a green dot representing the position of the end effector/hand. This is a reasonable exploration of the concept, but are there more concrete use cases where $I_{t}^{\text{agent}}$ obtained from segmentation models (as described in L101-103) actually interplays equally with the inpainted $I_{t}^{\text{env}}$? 2) It seems like in application 2, what the experiment is actually proving is that ground truth environment information without the agent improves 3D reconstruction (unsurprising), and not "the effectiveness of our proposal of using factorized agent-environment representations" (L228-229). In other words, there's no connection at all between this experiment and the factorized representation that you learn from VIDM. 3) In applications 4 and 5, neither of these tasks lend themselves to needing this agent-environment factorized representation. In particular, I feel that in opening a drawer, cupboard or fridge, the occlusion by the hand/end-effector does not significantly hamper the accomplishing of the task. Even in Figure 5, I see that the (salient) occlusion by the green dot in the agent-agnostic representation is almost more extreme than the occlusion by the robotic end-effector in the raw image. What's the intuition here, or could it simply be the case that the baselines were not tuned sufficiently? I ask because in Table 4, the "inpainted only" performance is much better than the "raw image" performance, but in Figure 5 for the real-world experiments, "inpainted only" actually fails entirely compared to "raw image." Can the authors provide some explanation for this?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

Limitations are addressed well in the discussion section of the paper.

Summary

This paper proposes to use agent-environment factorization of egocentric videos to facilitate various downstream tasks (e.g., object detection, 3D reconstruction, affordance prediction, etc.). The authors leverages a pipeline to achieve agent-environment factorization. It consists of first a segmentation model that segment hands in egocentric videos, and second a diffusion-based inpainting model for filling the hand area. The benefit of the proposed pipeline is supported by improvements of baselins over various downstream tasks.

Strengths

1. The idea of agent-environment factorization is interesting and as shown by the experiments, it does facilitate downstream tasks in various scenarios. 2. I do appreciate the sufficient amount of downstream tasks evaluated according, it proves the soundness of factorization.

Weaknesses

1. One major concern on this paper is on its technical contribution. The proposed VIDM contains limited technical novelty as it is a basic segment-then-inpaint pipeline and does not propose any new modules. 2. There is no video-inpainting method compared in comparative experiments (e.g., [1]). As diffusion-based models usually perform poorly in inference speed, the current comparison with image-based inpainting models do not show whether VIDM's inference speed is enough for videos. 3. In the reward learning task (Sec. 5.5), the experiments are only conducted on 3 tasks, and all of them are open-action tasks. Would the results and analysis still hold for more complex tasks? or is it limited to the current selected domain. 4. In the real-world policy learning task (Sec. 5.6), the action space of the robot is very limited (1D) and has been placed in a very task-specific position. It might be too simple for making a point. Additionally, these experiments were only conducted on one task (still open-action) under one secneario, this makes it difficult to assess the generalizability and effectiveness.

Questions

See the Weakness section. The authors could focus on: 1. Identify the uniqueness of the proposed VIDM and show its superiority for the current task instead of a plain pipeline. 2. Showing that agent-environment factorization could be beneficial for interaction tasks. As this could be the most important factor for the proposed pipeline, the current evaluated domain and task might be too limited.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations.

Summary

This work proposes the use of a factored agent and environment representation to handle two ego-centric video problems introduced by human hands : 1. They occlude objects of interaction and induce a domain gap between the data available for learning (egocentric videos) and the data seen by the robot at execution time; 2. Removing hands from the scene by masking or inpainting abandon the information of object affordances. The paper demonstrates the ability of the factored representation across tasks spanning 2D/3D visual perception to robot learning. They also show how selectively choosing and modifying aspects of the factored representation improves performance across all of these tasks compared to existing approaches.

Strengths

Strength 1. The paper is clearly written and easy to follow. The related work provides enough information for the reviewers to get familiar with the background of ego-centric video and related tasks. 2. The proposed diffusion model, VIDM, is effective while efficient. The performance looks amazing compared to previous work. 3. The experiment part is sound and shows the effectiveness of the proposed VIDM across many benchmarks. 4. The motivation is intuitive. Combining hand pose as well as inpainting technique can provide more information than before, and thus the improvement is plausible.

Weaknesses

Weakness 1. In figure 2, what is the meaning of the big f and g? Does they stand for different functions? If so, what is the purpose of drawing them in that way? The idea of this figure is not that clear. 2. Which part in VIDM contributes the most to the performance improvement across those benchmark? There seems to be no related ablation study on this. 3. In table 1, how does the stable diffusion (fine-tuned) done? It would be great if the author could provide more detail on this.

Questions

Please see my comment in weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please see my comment in weakness.

Summary

The following work presents a factorized approach for video-based egocentric tasks. Specifically, they propose to break down the video feed into separate environment-only and hands only feeds. Intuition behind this formulation is that change in appearance of the hands may constitute a domain gap when the source of the video perspective changes (e.g. person to person, person to robot manipulator). Furthermore, explicit factorization provides the model with additional supervision on the breakdown between what is the environment and what is the manipulator. The hands are removed from the video feed using a video-inpainting model based on the latent diffusion architecture with attention-based extensions to attend to multiple past frames. Results demonstrate that their video-inpainting formulation outperforms DLFormer in both the video inpainting task (limited to their use case) as well as in improvements to downstream applications (object detection, affordance prediction, etc.)

Strengths

- Simple idea with extensive demonstration of improvements in multiple downstream tasks. - Strong results for video inpainting in the ego-centric setting with similarly very simple but pretty easy to justify design choices (optical-flow-like attention for past frames, exclusion of hands from training data)

Weaknesses

- While the superiority of their video inpainting formulation is demonstrated only within the hand-removal task of ego-centric videos, the language used to describe the method can often be misinterpreted as a broader claim for outperforming existing state of the art models in a general sense. - The authors found it helpful to exclude hands as prediction targets during training. This seems like a significant design decision that should have a corresponding ablation study. - Additional training details for comparison against DLFormer are missing: - Was DLFormer also trained with the hand-exclusion technique? - On what training data were the visual codebooks used by the LDM formulation and DLFormer derived from? - While I don't necessarily doubt the idea that the factorized formulation improves object detection, I do not see average recall as an appropriate replacement for average precision. I would much rather see average precision measured on a limited set of categories where all instances of the object category in question are annotated.

Questions

See weaknesses

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Technical limitations discussed. I'm not sure it's sufficient to simply say that this work inherits uncertain societal implications from other generative modeling works.

Summary

The paper proposes agent-environment factorization (AEF) as a representation for egocentric videos. AEF consists of 2 parts: the hand segmentation as agent part, and video inpainted environment part. The former uses an off-the-shelf hand segmentation while the latter is from finetuning an inpainting diffusion model. The authors show several downstream applications to demonstrate the AEF can improve recognition, reconstruction, and robotic tasks.

Strengths

+ The key idea is agent-environment factorization, which is shown beneficial to multiple applications, 2D recognition, 3D reconstruction, and robotic tasks. The baselines are carefully designed such that they are directly comparable and showcase where and why AEF helps. + In the design of video inpainting model, they use the pretrained image inpainting to get the spatial prior while using nearby frames to aggregate more context.

Weaknesses

1. While the factorization is shown effective, the improvement over the video inpainting model itself appears a bit incremental. This is based on 1) qualitative results in FigS3 2) some domain specific knowledge being used (e.g. no loss on the hand pixels, copy paste hand-shaped masks, etc). It is not clear how well the proposed architecture can generalize to other well-adopted video inpainting benchmark benchmarks like DAVIS or Youtube VOS. 1.1 The proposed architectures may not be as critical as the paper claims (like in Table 1, see 1.2). It seems finetuning an generative video model, e.g. a MAGVit or video diffusion model may lead to results as good as the current method shows. I wonder if the authors agree with my conjecture. 1.2 The numbers in Table 1 indicate a significant gap over current SoTA video inpainting methods. Improvement over Latent Diffusion is sensible since the model sees more context. But DLFormer is a completely different method – a per-clip model that operates in pixel space. All of the metrics would favor sharp high-frequency signals, which latent space in latent diffusions are good at but DLFormer lacks. 1.3 visualizing attention may be helpful to understand how context in nearby frames help inpainting. 2. The paper is very related to “Neural Feature Fusion Fields” which factorize the videos into background, agent, and, in addition, moving objects. Although they optimize a per-clip representation, the differences with this line of works should be discussed. 3. There are some improvements across multiple downstreaming applications but the improvements are not surprising. For example, seeing the unoccluded objects / environment improves 3D reconstruction; merging prediction of both unoccluded and original images boosts object detection; seeing the hand location indicates frame order better. 3.1 It is a minor point but in Table 2, it will be more fair to compare if the proposed region is also doubled for the baselines since the proposed method is evaluated with twice the predictions.

Questions

Overall I think it is a sound paper. The novelty are faire and mainly empirical -- showing this factorization can help several downstream tasks. See weakness. My main concern is 1.1 and 2.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The author discussed the limitation explicitly.

Concerns addressed

The authors did a great job of addressing all my concerns, as well as the concerns of many other reviews. I am increasing my rating to accept with the expectation that all changes are appropriately incorporated into the final draft.

Post-rebuttal response

Thanks the authors for the clarification, the rebuttal has addressed most of my concerns, therefore I'm willing to increase my original rating to 5.