Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment

Summary

This paper proposes a self-supervised approach for learning view-invariant action features by aligning the egocentric and exocentric videos in time. The proposed method has two main components: 1) an object-centric encoder to learn discriminative features around the hands and active objects; 2) a contrastive learning paradigm to align videos from different viewpoints. Apart from the proposed method, this paper also establishes a new benchmark to evaluate the proposed method, and extensive experiments have been conducted to validate the effectiveness of the proposed method.

Strengths

1. The paper is well-motivated and well-written. Learning view-invariant features for fine-grained action recognition is important in video understanding. 2. Experiments are conducted on four different downstream tasks to show the effectiveness of the proposed method.

Weaknesses

1. The proposed method has limited novelty, and it is a combination of different existing modules. For example, emphasizing the hand-region features and contrastive learning with inverse temporal orders are widely used in many existing works. 2. The proposed benchmark contains four datasets, but each dataset has a very limited number of samples, conversing a few action classes. (a) Limited samples: e.g., the CMU-MMAC dataset contains 118 videos for training and 30 videos for validation and testing. This makes the model training easy to overfit. (b) Limited diversity: e.g., the CMU-MMAC dataset contains only a few actions related to 'breaking eggs', which casts doubt on the generalizability of the proposed method.

Questions

As the authors generate the benchmark datasets on their own, please elaborate more on the details, such as the number of action classes in each dataset.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Please see the weakness.

Summary

The paper introduces a novel approach for learning view-invariant features from exo and ego views using temporal alignment. The key idea is to utilize unpaired ego-exo data for the pretext task of temporal alignment in a self-supervised setting. Furthermore, an object-centric encoder was used to focus on hand and object regions. The authors also introduce a regularization method to maximize the margin between positive pair and their corresponding negative pairs. The negative pair is constructed by replacing the one video of the pair with its reversed.

Strengths

- The paper is well-written and easy to follow - The paper introduces a novel framework for view-invariant representation learning - The proposed approach is capable of utilizing the unpaired data where the videos of a pair do not have to be synchronized or be in the same environment - The method is compared with a reasonable set of baseline methods

Weaknesses

- The idea of sequence alignment as self-supervised video representation learning has been studied by e.g. [22, 23]. It is not clear why the methods of [22, 23] cannot be applied to ego-exo data. - In Line205-211 it is argued that frame-by-frame sampling in contrastive loss can lead to incorrect negative samples due to the periodicity of some videos, however, it is not clear how the proposed approach of using reversed videos for the negative sample does not suffer from periodicity.

Questions

See weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the limitation is addressed.

Summary

The authors proposed a novel self-supervised embedding approach AE2 to learn fine-grained view-invariant frame-wise video features from unpaired egocentric and exocentric videos. AE2 consists of two core designs: (1) an object-centric encoder that explicitly focuses on objects; (2) a contrastive-based alignment objective that leverages temporally reversed frames as negative samples. Four datasets exhibit improvement in experiments, demonstrating a feasible framework for learning fine-grained view-invariant features.

Strengths

1. This paper presents a well-motivation idea and sound arguments in general. It makes sense to learn view-invariant video features form unpaired data. 2. Ablation studies are conducted to evaluate the effectiveness of each design. 3. The visualization of learned view-invariant video features for test videos is very impressive, which demonstrates effectively capture the progress of an action while remaining view-invariant.

Weaknesses

1. In Lines 225-226 “ In creating negative samples, we opt for reversing the frames in S rather than randomly shuffling them.” Although the authors explain the superiority of the reversing frames, authors need to provide further experiments comparing reversing frames with randomly shuffling them. 2. Baseline's experimental results are so good that the baseline achieves SOTA results on most of the datasets. why is the baseline's experimental performance so good? 3. Lack of analysis of experimental results in the ablation experiment section. 4. In Training and Implementation Details, “the object-centre encoder network is optimized to minimize the loss above for 240 all pairs of video sequences in the training set, including ego-ego, exo-exo, and ego-exo pairs.” The training process introduces data from the same viewpoint and paired data, but the network structure is designed specifically for unpaired data from different viewpoints. Can separate experiments be performed only on unpaired data from different viewpoints to demonstrate the validity of the proposed method? 5. The language should be further improved by correcting some language errors, such as “ See Supp. for full implementation deatils ...” (Lines 232 in the paper).

Questions

See Weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None

Summary

The manuscript proposes a self-supervised learning approach for view-invariant representation learning from egocentric and exocentric videos. Unlike prior approaches, the proposed method is applicable to videos that are not aligned and captured in different environments. The key idea is to learn the alignment between the frames from two views using contrastive learning by using the temporally reversed version of the video as the negative sample. The manuscript also develops an object-centric encoder that focuses on the hands and active object regions in the frames. This allows the model to ignore the less informative regions such as background. The manuscript also develops a benchmark for evaluating the effectiveness of view-invariant representation approaches. Comparison of the proposed approach on the new benchmark with prior approaches clearly reveal its effectiveness.

Strengths

The manuscript addresses a problem that is not widely studied. Unlike prior view-invariant learning approaches that sets hard constraints such as availability of time-synchronized videos or videos captured in the same environment, the proposed approach can be successfully applied to unpaired videos captured at different environments. The benchmark developed could also be useful to the research community. The manuscript is written in a clear way explaining the various motivations behind the key ideas and the methodology used. The manuscript also presents an extensive set of experiments to validate its effectiveness by comparing against existing approaches. Moreover, most of the design choices adopted in the proposed approach are also validated with empirical analysis.

Weaknesses

One of the contributions of the manuscript is the use of time reversed frames as the negative sample for contrastive learning instead of random shuffling. However, there is no experimental validation of the proposed negative sampling strategy against random shuffling. Continuing from the above, the proposed negative sampling strategy could affect actions that are constituted of repetitive subactions such as cutting something, playing drums, etc. where the time reversed frames could look more or less similar to its original version. The authors mention in the supplementary that for videos containing non monotonic actions (pouring water) only the first/last half of the frames are reversed. However, it is not clear even such an approach would work on repetitive actions. It seems the proposed negative strategy will work only on a selected set of actions.

Questions

Please see the weaknesses section for the main queries. The learnable spatial embedding $\textbf{e}_{spatial}$ is mentioned as the embedding to encode the bounding box coordinates and the confidence score obtained from the hand-object detector. It is not clear how this is obtained for the global image feature $\phi _{project_g}(g_i)$. Some implementation details such as how the base encoder is initialized and whether it is frozen/finetuned during training are missing.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The manuscript has adequately addressed the limitations.

Response to authors

I appreciate the authors' efforts in providing a detailed response. All the concerns raised in the initial review stage are successfully addressed in the rebuttal. So I am keeping my initial rating.