HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness

Summary

This paper presents HOI-Swap, a diffusion-based video editing framework for object swap editing. HOI-Swap involves two stages. In the first stage, the authors train an image-editing model to swap object in one frame. In stage II, the authors first warp a video from the edited frame using the optical flow sampled from the source video. Then a video diffusion model is trained to reconstruct the whole video from the warped frame sequences. Through this two-stage approach, HOI-Swap successfully addresses three main challenges in HOI video editing: (a) HOI-aware capabilities, (b) spatial alignment of the object with the hand, and (c) temporal alignment with the source video. Experiments demonstrate that HOI-Swap outperforms both image-editing and video-editing baselines.

Strengths

1. This paper address a novel task: HOI video editing. 2. This paper pinpoints the three main challenges facing this task. 3. The video-editing stage employs optical flow from the source video to achieve temporal alignment with the source video. 4. HOI-Swap demonstrates better performance than existing video-editing approaches in HOI video editing.

Weaknesses

1. Stage I does not seem to address HOI-awareness explicitly. The authors should explain why the image-editing model can perceive HOI. 2. The authors did not mention the generalizability of their method across unseen kinds of objects.

Questions

1. How can the image-editing model in stage I perceive HOI? The authors should provide more explanation and maybe some more ablation studies to support their results. 2. The whole pipeline seems applicable to general object-swap tasks. Have the authors tested the model on other objects? 3. Are the backgrounds in the test set all seen by the model during training?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately addressed the limitations.

Summary

This work presents a novel approach for object insertion in hand-object interaction scenarios. The proposed approach consists of two stages: image-based editing to for precise alignment of hand with the inserted object, and video-based editing for motion alignment with the original video. The model is trained in a self-supervised manner and also imparts a varying level of controllability to adjust the degree of motion alignment based on object changes. Extensive experiments on HOI4D, EgoExo4D, EPIC-Kitchens and TCN Pouring datasets show the effectiveness of the proposed approach in various scenarios.

Strengths

- The proposed 2-stage approach to decompose the complexity of the task is intuitive and handles the challenges of hand-object interaction changes, correct object placement and temporal alignment of HOI motion effectively. - The first stage can also be used in a stand-alone manner for image editing tasks and the controllability in motion alignment is a useful feature. - The self-supervised training strategy circumvents the need for collecting paired training data for this task.. - Extensive experiments (Tab.1) on HOI4D & EgoExo4D show the effectiveness of the proposed approach over several baselines for both image and video editing tasks. The zero-shot generalization setting to EPIC-Kitchens and TCN Pouring datasets are also considered. - Visualizations in Fig.4,5 are helpful in understanding the capabilities of the proposed approach.

Weaknesses

- Several details about the experimental setup are missing from the main paper. These details are important to understand the scope of the claims in the paper. While the supplementary contains more details, it'd be helpful to have more clarity on the following aspects: - For evaluation on HOI4D & EgoExo4D, how is the test split created? Do the held-out videos contain different object categories or different instances from the same category or different subjects performing the experiments or different actions being performed? Having quantitative experiments in these different settings would help in understanding the benefits and limitations of the proposed approach. - Do the results in Tab.1 span all 4 datasets? Again, it'd be helpful to have a breakdown of the results in different settings to understand where the proposed approach is most effective. - Are all the baselines trained in the same setup as the proposed approach? I understand that it might not be feasible to implement/train all baselines due to differences in architecture and compute requirements, but it'd be useful to clearly describe the protocols followed for each baseline. Are these baselines retrained or used in a zero-shot manner? For example, AffordDiff [62] only takes a single RGB image of the object as input and generates the HOI image, it does not take the reference video into account so it is expected that the object orientation may not be consistent with the video (L43-50). AffordDiff also allows for controllability in the layout & orientation of the generated hand so it can be modified to match the hand in the reference video. I'm not asking to reimplement AffordDiff to adapt to this new setting, but it'd be useful to have these details in the main paper to understand the differences between baselines and the proposed approach. - While the 2-stage decomposition of the approach makes sense, it'd also be useful to quantitively verify that it is indeed better than the 1-stage analog. For example, in Fig. 3, Stage-2 can take the DINO encoding of the reference object instead of the generated image from Stage-1 to create a 1-stage approach. This is not required for rebuttal, but a suggestion to validate the 2-stage approach.

Questions

I need clarifications on the following aspects to better understand the results (more details in the weaknesses above): - For evaluation on HOI4D & EgoExo4D, how is the test split created? Do the held-out videos contain different object categories or different instances from the same category or different subjects performing the experiments or different actions being performed? - Do the results in Tab.1 span all 4 datasets? It'd be helpful to have a breakdown of the results in different settings to understand where the proposed approach is most effective. - Are all the baselines trained in the same setup as the proposed approach? It'd be useful to clearly describe the protocols followed for each baseline. Are these baselines retrained or used in a zero-shot manner?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

L330-332 in the main paper defers the limitations and failure modes to the supplementary without providing any insights. Please add these details to the main paper. --- I have read all the reviews and the rebuttal. I thank the authors for providing additional clarifications, which help me understand the paper better and I have increased my score to Weak Accept.

Summary

This article focuses on proposing a two-stage network HOI-Swap, a video editing framework designed for precise object edits with HOI awareness.To address the problem of real perception of HOIs as well as spatial and temporal alignment with the original video, HOI- Swap first stage focuses on solving HOI awareness and establishing spatial alignment by training an image restoration diffusion model to swap objects in one frame. The second phase deforms video sequences from edited frames by tracking inter-frame points with randomly sampled optical flow, and trains a video diffusion model to generate new videos.

Strengths

1. This paper is well organized and easy to understand. 2. The ablation experiment fully confirms the effectiveness and contribution of the improvement measures proposed in this paper.

Weaknesses

1. Limited innovation. The methodology of this paper is a combination of existing work and is more like a technical report than an academic paper. 2. Lack of dynamic HOI awareness transfer. The second stage does not embody HOI interaction awareness. The single-frame propagation only propagates the motion information of the object and does not embody how the HOI-aware information is conveyed in the video sequence. 3. Insufficient experiments. This paper only demonstrates HOI awareness in single-frame image restoration, and in the video aspect, either the scene with little change in hand poses or the scene with little change in object characteristics is demonstrated, but for the complex video sequences with large changes in object characteristics and the need for dynamic changes in hand pose, the paper does not give the relevant effects to demonstrate.

Questions

1.line 174, this paper suggests that data augmentation techniques such as flipping, rotating, perspective transformations, etc. are performed due to the difference in size and pose of the target and reference objects, so why not use scaling techniques to bridge the gap between large and small objects? 2.Line205, this paper proposes to apply the largest bounding box as mask to the whole video sequence, so for the case of a relatively large range of object movement, will a larger mask range lead to the proportion of sampling points that does not truly reflect the proportion of extracted object features and motion information？(For example, if the target object only occupies 1/4 size of the mask in a certain frame, then a uniform sampling ratio of 50% is also difficult to extract the object motion information). 3. Why not continue the use of DINO for the first stage network instead of switching to CLIP in the second stage network?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

Yes

Summary

The work considers the task of swapping the objects in ego-centric short clips with hand-object interactions. The manuscript claims to introduce this sub-task within the field of generative video editing. Concretely, HOI-Swap starts from RGB video, object area bounding box, and an image of the target object, and generates a new video, transferring hand movement to the new object, possibly adapting to the changing functionality of the object. The method has two stages: inpainting a single frame, and extending it to the whole sequence ensuring consistency. HOI-Swap is thoroughly evaluated against recent video and image editing methods, also the work demonstrates qualitative examples of generalization to novel datasets.

Strengths

1) The HOI-Swap method is novel and addresses a yet unexplored problem; 2) The proposed method is technically sound. It employs several techniques to ensure robustness and generalization of the resulting model: reference object augmentation technique to ensure robustness to viewpoint orientation, robust masking (square mask, consistent across frames to prevent overfitting), and randomized selection of the anchor frame. 3) Another notable feature of the method is its controllability through motion guidance. This original idea enables to variation of the amount of preserved information from the source object's motion to the target, thus allowing the method to adapt to the changing functionality of the objects. 4) The manuscript is well-written and easy to follow.

Weaknesses

A minor weak point is the lack of detailed discussion on downstream applications of the model. The Introduction section touches on this topic, however, an example of possible demos or a deeper discussion of possible usages of the model could help to better motivate the importance of the introduced task and the proposed method. Another possible area for improvement is automatic input mask annotation. Currently, the method requires a bounding box of the object as an input, however, with current advancements in the segmentation methods this requirement can be alleviated (or at least quantitatively compared to the GT bbox usage).

Questions

1) Stages 1 and 2 employ different image encoders (DINO and CLIP), what is the motivation for this choice? 2) Lines 212-220 discuss the ability to vary the number of control points sampled for motion guidance. What is the default value used for quantitative evaluations?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

A detailed discussion of limitations is present in the Appendix with complementary examples in a supplementary video.

Author-Reviewer Discussion Begins!

Dear authors, Thanks for your hard work and detailed responses. Dear reviewers, the authors have responded to your questions and comments, please read them and further provide feedback: Whether your concerns addressed? After reading all the reviews and responses, are there more questions about the clarity, contributions, results, etc? Thanks! Best, Your AC

Reply by Reviewer ih2o

I thank the authors for providing the rebuttal. The additional experiments (evaluation of CLIP vs DINO as an encoder for stage 2 and evaluation of using non-GT mask predicted on the fly) clarify my concerns and strengthen the work. After considering the authors' replies and other reviews, I have decided to increase my score to accept. The work tackles a novel problem, providing a solid baseline for future research.

Thanks for the clarifications

I appreciate all the clarifications provided by the authors. I have increased the score to Weak Accept and suggest to include these additional details in the final version.