EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding

Summary

To improve the efficiency of ego-centric video representation learning, this paper presents a distillation-based approach that learns to reconstruct heavy egocentric video clip features from the combination of sparsely-sampled video frames and the camera motion from IMU.

Strengths

1. The paper is well-written and the easy to follow. 2. Improving the efficiency of ego-centric video representation learning is indeed valuable.

Weaknesses

1. As mentioned in the paper, the dynamics in a video include both the movement of the scene and the camera (Line 34-35). I agree that IMU could provide some cues for the motion of the camera, however, I'm afraid it is difficult to deduce the motion of the scenes, which is more important for the video understanding (although some analysis is presented in Figure 6, it is still contradictory to the intuition). 2. From Table 1, we can see that the performance improvements mainly come from L_{KD}, other components like L1, L1-pretrain, and L1-NCE, bring subtle improvements. 3. It could be better to compare the model performance in Table 4 (which is mismarked as Figure 4).

Questions

Please refer to the weakness.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

1 poor

Presentation

3 good

Contribution

2 fair

Limitations

The authors have discussed the potential limitations in the paper.

Summary

The work presents a new approach and architecture, EgoDistill, which could efficiently perform action recognition through sparse video frames augmented with head motion data. The model goes through a two-stage training. A pre-training phase, which uses a contrastive loss that enables the motion encoder to learn semantic visual features, and a distillation phase, which makes the motion-visual fusion model jointly mimics the more stronger visual-only model. The results are impressive, supported by a series of ablation study. There is no major flaws in the paper, except a minor confusion on the argument on the fact that the pre-training of the model forces the motion encoder learning semantic features.

Strengths

The paper is very well written and easy to follow. The figure showcases the method very well. The authors made a keen observation in using a separate modality (head motion) jointly with egocentric frames to efficiently perform action recognition task. The proposed architecture is novel and yields impressive results.

Weaknesses

No major weaknesses.

Questions

During the pre-training phase, L1 would train the fusion network as well as the IMU encoder, while NCE loss would mix and match samples to train both the IMU predictor and IMU encoder. How is the schedule of training the IMU predictor \hat{z} match with the optimization of the IMU encoder? How to prevent IMU encoder from being affected by under-trained IMU predictor?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

I am not super convinced on the argument that the IMU encoder learns scene motion cues (line 318). The action confusion (between close/open and put/take) is a good example, but the two action pairs seem to be more relevant to time sequence rather than scene motion.

Summary

This article presents an innovative approach called EgoDistill for egocentric video understanding. The authors highlight the computational challenges faced by existing models in this field and propose EgoDistill as a solution to overcome these limitations. EgoDistill aims to distill crucial information from egocentric video frames and IMU readings, resulting in a more efficient representation. The article provides detailed technical explanations of EgoDistill, illustrating how it effectively combines semantics from video frames with IMU readings. Additionally, the authors introduce a novel self-supervised training strategy specifically designed for IMU feature learning. To validate the effectiveness of EgoDistill, the authors conduct experiments and present compelling results. Their findings demonstrate that EgoDistill surpasses state-of-the-art efficient video understanding techniques when evaluated on the Ego4D and EPIC-Kitchens datasets. Overall, this article contributes significant advancements to the field of egocentric video understanding by introducing EgoDistill and showcasing its superior performance over existing methods.

Strengths

1. EgoDistill offers several advantages over equivalent video models, including significantly reduced computational requirements in terms of GFLOPs, parameters, and inference time. This enhanced efficiency makes it a more practical option for real-world applications. Furthermore, the authors provide experimental evidence illustrating that EgoDistill surpasses state-of-the-art efficient video understanding methods when evaluated on the Ego4D and EPIC-Kitchens datasets. 2. The authors have devised a concise and effective self-supervised IMU feature learning architecture. This architecture enables the extraction of valuable camera motion features that are specifically associated with visual appearance changes. As a result, EgoDistill demonstrates improved performance in capturing and understanding these subtle visual cues, leading to enhanced overall performance.

Weaknesses

1. The main text contains some basic errors, such as confusion between the order of figures and tables, as well as incorrect citations. It is crucial to carefully review and correct these issues. 2. Undoubtedly, EgoDistill exhibits commendable performance in terms of generalization and versatility. However, the explanation provided in the paper for why EgoDistill performs better than STTS when both methods incorporate IMU features (as mentioned in the "Effect of fusing IMU data" subsection) is too superficial and lacks a profound theoretical explanation for EgoDistill. Additionally, there is limited discussion on how the balance between different components of the model is controlled. It would be beneficial to address these points and provide more insights into the underlying mechanisms and strategies employed by EgoDistill. 3. EgoDistill relies on head motion data to capture camera pose changes. It is important to explore and discuss the limitations and potential mitigation strategies for scenarios where reliable head motion data may not be accurate. 4. The authors have not thoroughly discussed any potential shortcomings or limitations of EgoDistill. It would be valuable to identify specific scenarios or use cases where EgoDistill might not be as effective or efficient. Understanding these limitations would provide a more comprehensive understanding of the applicability and scope of EgoDistill and could open avenues for future research to address these areas of improvement.

Questions

See Weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None

Summary

This paper for on Ego-centric video understanding, a promising and popular topic recently. Different from previous works that focus on improve heavy video encoder, this work try to cut the computation cost down and focus on lightweight temporal reasoning. Please reply the questions and weakness point-by-point.

Strengths

- The motivation is quite good and make sense. I like it. Actually ego-centric video have fixed background in general and learn the motion pattern based on the first frame is more easily than third-view video. Considering this area is new and not well studied, I think it's valuable. - The method is also simple but works. - The presentation is good and I can understand it very well. - The computation cost is cut a lot and provide possibly for small and mobile device.

Weaknesses

- The method section especially the IMU predictor is not solid. Actually, the idea of bring GRU for temporal modeling is quite a old technique. There also exist a lot of works that predict the temporal sequence on feature level like MemDPC two years ago. - Although the performance is not a very important measurement to me, the compared method is somewhat old. For example, Line243-248.

Questions

- Why the first row of Fig.6 sometimes seems not well matched with the action, can you give detail explanation about how to get this figure and why always no changed? The head motion is from AR glasses or? - How to define ambiguous categories in Ego4D?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- The performance is hard to compare with video encoder based method. - The proposed work also suitable for ego-centric video.

Summary

The work studies efficient video recognition via distilling IMU signals. Instead of heavily processing each frame, the authors propose to leverage very sparse frames (1-4) for visual features and IMU data. It leads to much more efficient inference cost. The proposed method has shown improvement over other multi-modality fusion techniques like AdaFuse, etc. They also show IMU trajectory provides complementary information to sparse frames.

Strengths

+ The idea to leverage IMU signals for efficiency is novel. And the empirical results support the idea. The improvement with IMU is consistent across different architectures. + The two stage training (first aligns IMU features then fusing them) is shown effective over alternative multi-modality fusion baselines. + The claims are well supported by ablation studies like two-stage training, loss term, usage of modality, etc.

Weaknesses

1. Since IMU only represents head motions, it is sensible to disambiguate activity relevant to scene motion (open / close, put /take), but it may not contain enough signals about finer grained activity temporally or spatially. But it is the problem of IMU signal, not the paper’s. 2. I wonder how much does the motion _trajectory_ help, compared to the overall displacement between the end frame and the starting frame, i.e X_t - X_0. More specifically, in table 2, how does EgoDistill with less IMU steps work and how does feeding start frame __and__ end frame works?

Questions

See weakness above

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitation is discussed

Maintaining my evaluation

Thank the author for the response! After reading the review from other reviewers, I remain my position in supporting the author. I think the overall idea is straightforward and would make a nice addition to the community.