EgoEnv: Human-centric environment representations from egocentric video

Summary

This work aims to learn human centric environment representations from first person camera views. Their novel approach utilizes a transformer-based approach that encodes the local environment state at each time-step in an egocentric video is defined as a set of objects, along with their approximate distances, located in front, left, right, and behind the camera-wearer. This work claims to outperform state-of-the-art representations in predicting visited rooms and retrieving significant moments when responding to natural language queries.

Strengths

`Significance:` This work takes an important step towards proposing an approach that models the physical surroundings of a camera from a single egocentric perspective. This work could have multiple applications in AR, VR and robot navigation. `Originality:` Even though there has been a lot of work in the domain of video understanding in 3D environments, most approaches localize the camera-wearer but do not learn representations for the camera-wearer’s surroundings. Therefore, to the best of my knowledge, this work is novel. `Quality:` The authors provide a good theoretical background to their approach and back up their claims with rigorous experimentation on multiple datasets and tasks. `Clarity:` The paper and the supplementary materials are clearly written and easy to follow.

Weaknesses

- This work does not consider the influence of motion blur if the person wearing the camera makes sudden movements, making it challenging to apply this work to real environments. - This work also uses a semantic segmentation model to identify physical objects in the camera wearers’s environment. However, that could be limiting as their approach is restricted to objects that are classified by the segmentation model.

Questions

- The authors should elaborate on how their models would deal with motion blur and sudden movements of the camera. Is there a motion model that is implicitly learned based on the camera movements or is it assumed that the camera moves at a fixed speed? - Since this method is intended for AR applications, were their any evaluations done to benchmark the real-time inference performance of this method?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors do not explicitly state the limitations of this work, and should refer to the Weaknesses section of the review and consider stating those limitations in the paper (if applicable).

Summary

In this paper, the authors address the limitation of current video understanding methods that only analyze short video clips in isolation, without considering the broader context of the camera-wearer's environment. They propose an approach that establishes a connection between egocentric videos and the surrounding environment by learning predictive representations. To accomplish this, the authors train their models using videos captured by agents in simulated 3D environments, where the environment is fully observable. An interesting finding is that despite being exclusively trained on simulated videos, the proposed approach effectively handles real-world videos from HouseTours and Ego4D datasets. It also achieves state-of-the-art results in the Ego4D NLQ challenge. I have increased the recommendation after the rebuttal. The rebuttal addressed my concerns.

Strengths

Overall, this paper introduces an innovative approach that bridges the gap between egocentric video and the camera-wearer's environment. By leveraging predictive representations, trained on simulated videos, the proposed method demonstrates improved performance over traditional clip-based approaches in various human-centric video tasks and real-world scenarios. The general idea of grounding egocentric video in its underlying world environment is very interesting. The proposed method to learn representations that are predictive of their surroundings and then enhance standard clip-based models is technically sound.

Weaknesses

In my opinion, the core of this method is a pertaining process that refines a feature into a better one by implicitly seeing its surroundings. In this sense, in all the experiments, EgoEnv features have knowledge about the surroundings however other features do not. Thus, one important experiment needed to be done is allowing other models to access the same amount of information but in a more straightforward manner. For example, giving other models not only the current clip as input, but also frames that are uniformly sampled from the whole video. From the Tables, FrameFeat and ObjectFeat baselines are already very useful. I wonder whether a simple extension of these methods that, similarly, allows a longer temporal range as input at a time, would also perform reasonably well.

Questions

1. The experiments stated in the weakness section. 2. Per the NeurIPS requirement, I hope the authors can discuss the limitations of the paper.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did not state the limitations of this work in the paper.

Summary

The paper proposes a novel framework to learn environment-aware video representations from egocentric videos. The framework can be trained on synthetic data and incorporated into various existing approaches for real-world downstream tasks including RoomPred and NLQ. Experiments demonstrate that models equipped with the framework achieve superior performances in the two downstream tasks on two real-world egocentric datasets, which demonstrates the value of synthetic data for real-world 3D understanding under egocentric videos.

Strengths

(1) The method in the paper releases the need for camera poses, making it more robust to the noise in structure-from-motion algorithms compared to existing pose-based methods. (2) Experiments fully demonstrate the effectiveness of the method. (3) Comprehensive ablation studies demonstrate the value of the core designs of the method. (4) The method is trained on only automatically-generated synthetic data yet performs better than baselines on real-world scenarios, indicating that simulated data could directly benefit real-world egocentric video understanding without techniques of sim-to-real transfer. (5) The writing is clear and well-organized.

Weaknesses

(1) The method hugely relies on the surrounding background objects in the 3D scene. When a person navigates in a scene with clean backgrounds, it might be challenging for the method to encode expressive environment features. When dealing with easy real-world instances in the RoomPred task, the method performance is slightly worse than the baselines. (2) The performance gain from the proposed method seems marginal on Ego4D.

Questions

When incorporating the ground-truth camera poses into the method, I am curious about how to input the pose to EgoEnv+pose since I do not find a network parameter that transfers camera poses to pose embeddings.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

One potential limitation is that the method has not been tested on dynamic scenes with dynamic background objects or humans such as multi-person interaction scenarios.

Summary

In the paper, a model is introduced to extract vector embeddings of environments using images from a first-person perspective. The model is trained in a simulated environment where a virtual agent moves around and collects images to learn about its surroundings. The model's performance was tested in two different tasks - predicting the layout of a room and remembering sequences of events. The experimental results showed that the proposed model outperforms other baselines in both downstream tasks.

Strengths

- The paper is well-structured and easy to follow, with a clear motivation behind it. The tables and figures are helpful in explaining the concepts. Overall, I found the paper to be an enjoyable read. - The paper tackles the very interesting and relevant problem of scene understanding from first-person images. - The proposed model is very interesting and the experiments are enlightening and helpful in utilizing environment information of egocentric images to better understand the scene. - I believe this is a solid paper that deserves to be communicated.

Weaknesses

Although the paper is well-written and structured, many important experiments and discussions were excluded and can only be found in the 18-page supplemental material. One example is the full ablation procedure which is solely available in the supplemental material. I couldn't find any information on the limitations of the agent, which is crucial in this work. For instance, how well the agent performs in outdoor environments? Moreover, it's important to know and if there are any differences in walking patterns between the virtual agent and real people that may affect the results. Additionally, it would be helpful to know how effective the vector embedding is in dynamic scenes where the environment is constantly changing during video acquisition. I believe that it would be beneficial to briefly discuss these questions that have been raised.

Questions

1) How was the algorithm the generate the walking pattern of the virtual agent? Were used different speeds or motions (e.g., running, walking, etc.) during the training? 2) Based on the images presented in the paper, it appears that the agent did not look up or down. I am curious if this lack of movement would impact its performance in videos focused on activities such as cooking, where the camera angle is often directed downwards. Has there been any analysis on this topic?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

I was unable to locate the discussion on limitations either in the paper or the lengthy supplemental material.

Summary

The work presents an approach to learn spatial environment representations for egocentric videos. Such representations encode the camera-wearer’s (seen and unseen) local surroundings/environment. Previous approaches mostly focus on learning representations over a longer temporal space, however, understanding of the physical spatial space seem to be missing which is addressed in the work via learning of environment-aware features. The method define a local environment state having a set of object in a relative direction to the camera wearer. The local state has both geometric information (relative object location) and semantic information (object labels). The model learns this local state by learning two matrices - a direction matrix which represents which object is in front, back, left or right and another metric which represents the distance of the object from the camera wearer in a discretized space. To define the matrices, pose information is needed. Since ground truth pose information is missing from real-world egocentric videos, a model first learns the pose information from simulated environment by minimizing a cross entropy loss between the pose of camera wearer and pose of object. Next, an environment memory is encoded from a video walkthrough of T frames and a query frame. Pose embeddings are generated for all the T frames and query frame using the learned pose model, and each frame is encoded with this pose embedding. Then, K video frames are sampled to construct an environment memory using a transformer encoder. This representation consists of both temporal and spatial information of the environment. The transformer decoder then uses this environment memory and the query frame to generate a EgoEnv representation which is combined with the original video features (eg. Resnet). These representations are finally used for the video downstream tasks. The method is evaluated on the room prediction and NLQ challenge on the datasets - Ego4D, HouseTours, and Matterport3D.

Strengths

1. The paper is well-written and has a detailed supplementary material covering the dataset statistics, model architecture details, and ablation experiments. 2. The work shows a thorough evaluation with the method being evaluated on multiple datasets and compared with multiple baselines.

Weaknesses

1. It would be great to discuss about the accuracy of the pose embedding learning network on the simulated network since the overall model is dependent on it. 2. There can be more discussion as to how the sim-to-real gap gets reduced in the approach.

Questions

How much time does it take to train the pose embedding network and the rest of the model too in general?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

No such limitations. The paper is well-written, covers all the implementation details, and discusses about all the aspects of the approach.