Despite significant advancements in video generation and editing using diffusion models, achieving accurate and localized video editing remains a substantial challenge. Additionally, most existing video editing methods primarily focus on altering visual content, with limited research dedicated to motion editing. In this paper, we present a novel attempt to Remake a Video (ReVideo) which stands out from existing methods by allowing precise video editing in specific areas through the specification of both content and motion. Content editing is facilitated by modifying the first frame, while the trajectory-based motion control offers an intuitive user interaction experience. ReVideo addresses a new task involving the coupling and training imbalance between content and motion control. To tackle this, we develop a three-stage training strategy that progressively decouples these two aspects from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion module to integrate content and motion control across various sampling steps and spatial locations. Extensive experiments demonstrate that our ReVideo has promising performance on several accurate video editing applications, i.e., (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also seamlessly extend these applications to multi-area editing without specific training, demonstrating its flexibility and robustness.
Paper
Similar papers
Peer review
Summary
ReVideo presents a novel view of video editing by modifying content with input trajectory to create new content. It designs a three-stage strategy to wrestle out the problem of ignoring motion control when direct training. The main contribution of this work relies on the new task of editing motion via user-specified trajectory while keeping the original video movement. The editing results are superior and photorealistic.
Strengths
1. The first video editing work on creating new motion and content. 2. Good writing; the paper is easy to follow, and the motivation and three-stage training strategy on decoupling content and motion control is reasonable. The proposed SAFM learned a dynamic fusion weight at different timesteps. 3. The editing results are photorealistic and adhere to the original motion or follow user-specified trajectory with no artifacts.
Weaknesses
1. The author did not provide the method or explanation of how ReVideo edits the first frame, making the total editing pipeline not end-to-end for users. 2. Part of the original video motion, like mouth movement in the Zuckerberg->robot (head6) and tail movement in dog->lion, is not kept in the edited video. 3. I would like to know how the drag-based editing method handles non-rigid motion, such as the rotation of car tires from a side view. In examples like sea2 and sea2_2, where a shark and a dinosaur are added, the limbs of the animal seem unable to move, making the video look unrealistic. However, in soccer and some human-centric examples, the legs of dogs and people can move normally. Therefore, I would like the authors to add an example of a vehicle moving on the road from a side view, including the movement of the wheels, to address my concerns. This may be a limitation of the drag-based method. 3. There is no quantitative comparison of the ablation study; I understand that the image results in Fig 7 are clear, but only one video qualitative ablation is not reasonable. 4. There are no qualitative video comparisons with other methods in the supp or project page, but only Fig 6, and the automatic metrics are worse than pika even though I understand the clip scores are not accurate, which can not reflect temporal consistency accurately. I suggest the author supply the comparison video between Revideo and other methods in the rebuttal phase. 5. The training cost of three stages: even though Revideo makes great progress in creating new motion, training cost like GPU costs, time costs, memory costs and so on, is still a problem since users prefer to edit a video in a zero-shot manner when using a pretrained video generation model and the compared methods like AnyV2V is training-free.
Questions
1. The method to edit the first frame needs to be declared. 2. Non-rigid motion-like side view wheels movement of cars. 3. Qualitative video comparisons with other methods. 4. The inference time/training cost comparison with other similar methods. 5. What about ReVideo performing in editing multiple objects simultaneously in the same video? 6. Can ReVideo work on text-to-video generation models?
Rating
7
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
no significant limitations.
My concerns have been thoroughly addressed in the rebuttal. Overall, this paper is well-structured, offering significant contributions and demonstrating impressive performance. It brings fresh perspectives to the research field, so I have decided to upgrade my final rating to accept.
Thank you for your efforts in the review, improving our paper to a higher standard. We will revise our paper based on your comments and suggestions.
Summary
The paper presents a video editing method that enables precise localized adjustments to content and motion within specific areas of a video. It introduces a three-stage training strategy and a spatiotemporal adaptive fusion module to integrate edits across frames and locations effectively. This method allows for complex editing tasks such as changing content while maintaining motion, adding new motion to static content, and simultaneously modifying both elements.
Strengths
- The paper introduces a novel challenge of editing both content and motion in specific video areas and combines techniques from diffusion models and video editing to achieve nuanced control. - The three-stage training strategy enhances the robustness and effectiveness of the edits, supported by experimental validation that demonstrates superior performance compared to existing methods. - The paper is well-organized and clearly explains complex concepts, including the innovative spatiotemporal adaptive fusion module and detailed training strategy.
Weaknesses
- The decoupling training could cause some artifacts. Although the paper demonstrates these artifacts could mostly be alleviated by deblocking training. I can still see some blocky/unnatural results in the result videos. - The training is quite complicated and separated into three stages. I feel the training strategy could 'overfit' this particular video dataset. - This method is more like a direct combination of video diffusion and ControlNet. - More detailed implementation specifics, particularly regarding parameter settings and the architecture of the spatiotemporal adaptive fusion module, are needed. - The method's computational demands and potential scalability issues are not adequately addressed. For example, what kind of GPU does one need to perform training and testing? - The paper focuses heavily on technical aspects with less consideration of user interaction.
Questions
- What kind of GPU does one need to perform training and testing? - Will the authors release the training and testing code along with pre-trained models upon acceptance?
Rating
5
Confidence
3
Soundness
2
Presentation
2
Contribution
2
Limitations
Quality limited by SVD.
We need to clarify that: 1. **ControlNet is a commonly used method for condition injection, not a solution for our problem (as verified in Section 3.2). The contribution of this paper is to propose effective solutions for local video editing.** Additionally, we modify ControlNet by designing a region-adaptive fusion method to make the condition embedding more suitable for this task. 2. **Why is the decoupling training strategy not considered one of our contributions?** Dear reviewer, have you ever seen other similar solutions for video editing? The reference you provided is an image classification method—what relevance does it have to our paper? Just because two different images are mixed together? 3. **Our method is successfully validated on SVD and WebVid data, but this does not mean that our method can only be implemented using SVD and WebVid.** It can be applied to other data and base models. We believe that addressing a challenging problem by a progressive learning approach is reasonable. In many complex generation tasks, such as EMU2[1], the training difficulty is much greater than ours. [1] Generative Multimodal Models are In-Context Learners 4. Firstly, we are addressing an unexplored and challenging task that requires carefully designed training processes and fusion modules. This is not a simple engineering patchwork. Our core components did not previously exist. Secondly, what is the relevance of the CutMix to our work? Why can it replace our core contribution? Dear R2, we are eager to address these misunderstandings and look forward to your response.
1. Novelty: Modifying existing methods doesn't automatically constitute novelty. Your paper shows good designs and results but appears incremental to ControlNet. Clarify what fundamentally differentiates your approach from ControlNet beyond architectural changes. 2. Methodology: The cut-and-paste approach you're using is common in various video processing tasks, e.g., [1], not unique to your method. Referencing MixCut was to highlight this point, not to directly compare classification and video editing tasks. 3. Generalizability: You claim your method can generalize beyond SVD and WebVid, but haven't demonstrated this. Without broader validation, the method's applicability to other datasets or base models remains unclear. 4. Contributions: While some components in your work are new, they appear to be engineering efforts rather than fundamental contributions. The reference to CutMix was to point out similar data augmentation techniques in video processing, not to equate it directly with your method. [1] Liu, Yang, Zhen Zhu, and Xiang Bai. "Wdnet: Watermark-decomposition network for visible watermark removal." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.
1. ControlNet cannot accomplish our task. This is the fundamental difference. The following works all used ControlNet for conditional injection. We do not claim ControlNet as one of our contributions. It is a commonly used model, just like Stable Diffusion. [1] Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models [2] ControlVideo: Training-free Controllable Text-to-Video Generation [3] CameraCtrl: Enabling Camera Control for Text-to-Video Generation [4] EVA: Zero-shot Accurate Attributes and Multi-Object Video Editing [5] CCEdit: Creative and Controllable Video Editing via Diffusion Models [6] Text-Animator: Controllable Visual Text Video Generation [7] IMAGE CONDUCTOR: PRECISION CONTROL FOR INTERACTIVE VIDEO SYNTHESIS 2. [1] is a paper on watermark removal, where a watermark image refers to the watermark being overlaid on the original image. This overlay is a task, not a solution. We still cannot find any connection between this and our method. [1] Liu, Yang, Zhen Zhu, and Xiang Bai. "Wdnet: Watermark-decomposition network for visible watermark removal." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021. 3. Our method is the first attempt at this task and has already achieved state-of-the-art (SOTA) results. Even if adjustments are needed for other models and datasets, the issues found and solutions proposed in this work are still insightful. 4. CutMix you referenced is entirely different from our approach in both purpose and methodology. We do not think combining two images to enhance image classification performance can replace our contribution. What we propose is a motion-decoupled training strategy, not data augmentation. This decoupling strategy is effective and does not exist in our community.
I found out I severely misunderstood the operation of the motion decoupling part. The goal here is to manipulate different motions between the foreground (editing region) and background. Then, the purpose is largely different from MixCut or other cut-and-paste approaches. Could the authors further claim the purpose of the motion decoupling? Is my current understanding correct?
This understanding is correct. In our experiments, we find a significant motion coupling between the foreground (editing region) and the background (please refer to Section 3.2). The model tends to learn how to estimate the motion state of the foreground (editing region) by relying on the motion in the background, thereby neglecting the condition of motion trajectory. To address this issue, we propose the motion decoupling training as you mentioned. We may not describe this clearly enough in the paper, and we will make a revision for this part.
Dear authors, Thank you very much for the clarification. However, I still feel the paper is somewhat incremental by combining a ControlNet-like motion injection module with augmented (motion) training data. The approach and task themselves are new and novel. Furthermore, the result videos are impressive, with barely noticeable artifacts. I would like to raise my rating to borderline accept. Thanks again to the authors for the clarification and discussions!
Thank you for your efforts in the review. Your comments help improve our method, and we will continue to enhance our approach in future work.
Summary
This paper presents ReVideo, a new approach for precise local video editing of both content and motion. It introduces a coarse-to-fine training strategy to progressively decouple content and motion control, and a spatiotemporal adaptive fusion module to integrate them effectively. Experiments show ReVideo can modify local video content, customize motion trajectories, or change both simultaneously, and extend to multi-region editing.
Strengths
- This appears to be the first attempt at exploring local editing of both content and motion in videos using diffusion models. Being able to modify content and motion trajectories in specific regions is a novel capability compared to prior work. - The proposed three-stage coarse-to-fine training strategy to progressively decouple content and motion control is an interesting technical approach to deal with the core challenge. - The spatiotemporal adaptive fusion module is another novel component to integrate the content and motion conditions across sampling - steps and spatial locations. - Extending the approach to allow multi-area editing without requiring specific training demonstrates flexibility. - Most of the visual and quantitative results show improvements over prior methods Overall, this paper addresses a timely and important topic with significant potential benefits for the community. Despite some weaknesses, the reviewer recommends acceptance, considering this is a relatively new area and the paper presents promising results. The score may be adjusted based on the quality of the rebuttal.
Weaknesses
## Practicality of the Editing Workflow The current editing interface requires users to specify both a target content image and a set of motion trajectories. While this allows for fine-grained control, it may not be the most intuitive or efficient workflow for common editing tasks. Consider the scenario of object removal - the user would need to carefully craft a content image with the object removed and ensure that the remaining motion trajectories are consistent. An alternative approach could be to directly specify the regions to remove and have the model infer the appropriate content and motion changes automatically. The paper would benefit from a more detailed discussion of the practical trade-offs and usability considerations of the proposed editing framework. ## Limited Motion Control While the method allows for editing the motion of individual objects, it assumes that the overall scene motion (camera movement, background motion) remains fixed. This limits the applicability of the approach in scenarios where the goal is to modify the global motion patterns (e.g. stabilizing shaky footage, changing the camera viewpoint). ## Precise Placement and Key Contributions of this Paper While the individual technical components (e.g. coarse-to-fine training, adaptive fusion) are well-motivated, it's worth considering whether similar strategies have been explored in related domains. For instance, progressive training to handle multi-factor variation has been used in GANs, and spatially-adaptive normalization is common in style transfer. Drawing more connections to such related work would clarify the novelty of the specific adaptations made here. ## Content-Motion Entanglement - The key technical contribution of the paper is the decoupling of content and motion information through a coarse-to-fine training strategy. However, it's not clear if this decoupling is complete or if there are still some residual entanglements between the two factors. For instance, the edited content may still contain some motion information that could interfere with the specified motion trajectories, leading to artifacts or inconsistencies. A more thorough analysis of the content-motion separation and its impact on the editing quality would be informative. - Is decoupling content and motion the only way to address the issue - could a joint representation learning approach work instead? Acknowledging alternate strategies would help justify the chosen approach. - **Figure 4 is not very intuitive. It would benefit from additional justification, theoretical analysis, and insights into why such a simple composition from two videos is effective.** This is a key concern. ## Multi-area Editing - The extension to multi-area editing is a nice addition, but the paper could go further in characterizing the challenges involved. Are there issues with preserving global coherence across multiple edited regions? How does the method scale with the number of regions? Providing such details would give a more complete picture of the capability. ## Clarity and Reproducibility - Implementation details: There are some missing specifics that could hamper reproducibility. For instance: - How exactly are the editing regions defined during training - what is the procedure for randomly sampling them? - What metrics are used for the "threshold filtering" of motion trajectories and how were the thresholds chosen? - Are there any data augmentation, regularization or optimization tricks used during training? ## Evaluation Metrics The quantitative evaluation relies primarily on low-level metrics like PSNR and LPIPS, which may not fully capture the perceptual quality and coherence of the edited videos. Additional metrics could provide a more comprehensive assessment: - Metrics that specifically measure the consistency of the edited regions with the target content and motion (e.g. using an object detector or tracker). - Metrics that evaluate the temporal stability and smoothness of the edited videos (e.g. some metrics that are used in video inpainting tasks, Please refer to [this repo](https://github.com/MichiganCOG/video-inpainting-evaluation) for details). - Human evaluations of the overall realism, coherence, and faithfulness to the editing inputs (e.g. through user studies). ## Robustness Evaluation and Ablation Studies While the paper does include ablations for a few key components (e.g. SAFM, training stages), there are other design choices that are not fully explored. For instance: - How important is the choice of motion representation (trajectory vs. alternatives)? Testing with different motion inputs would reveal the sensitivity to this factor. - What is the impact of the trajectory sampling strategy and hyperparameters? Varying the number and selection of trajectories could provide insight into the robustness. - How does the performance vary with the size and shape of the editing regions? A systematic evaluation across different region properties would be informative. - Only the end-to-end video editing pipelines are compared, but not the individual technical components. For instance, how does SAFM compare to simpler fusion schemes used in prior work? - Input noise and perturbations (e.g. in the content image or motion trajectories) ## Dataset Complexity - While the approach achieves good results on the chosen datasets, it's unclear how well it would generalize to more complex video content (e.g. with dynamic backgrounds, scene changes, occlusions etc.). Discussing the potential failure modes and current limitations would help scope the contribution appropriately. - The examples shown in the paper are largely limited to simple object-level edits in relatively constrained scenarios (e.g. clean backgrounds, single objects). It's unclear how well the method would perform on more challenging videos with complex scenes, multiple objects, occlusions, camera motion, etc. Testing on a wider range of video complexity would help establish the generality of the approach. ## Editing Scenarios The paper demonstrates a few key editing applications (e.g. object addition/removal, motion editing), but there are other important scenarios that are not explored, such as: performing semantic-level edits (e.g. changing the action or interaction between objects). Showcasing the method's performance across a fuller range of editing tasks would demonstrate its versatility. ## Open Source Will the code for training and inference be released?
Questions
Please refer to the weakness section.
Rating
6
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Please refer to the weakness section.
I really appreciate the author's rebuttal. However: 1. The authors claim that the proposed method's novelty is beyond ControlNet. However, I am still not convinced. From Figure 2 in the paper, the two potential structures are just two slightly different ways of conditioning ControlNet. The trainable control module and zero convolution layers are exactly the same as ControlNet. The only difference is that you replace the condition (originally depth, canny edge map, normal, etc.) with a content map specifying the editing region and the motion trajectory. I still cannot say this analysis is a contribution. I'd say this is rather than an ablation on network architectural design. 2. In order to keep the original motion from the original video. This paper further proposes a data construction strategy, as shown in Figure 4, to decouple the motion between the editing region and the original region. However, this approach is still too naive, as it just performs CutMix [1] to combine two videos into one. However, in this case, the goal is to manipulate motion. I am still not convinced this stage is a contribution. 3. The training pipeline, separated into three stages, is very complex. There will be many hyperparameters that could influence the training in each stage. Also, the entire pipeline relies on many existing off-the-shelf techniques/datasets such as SVD and WebVid (This dataset is actually banned and no longer available due to some legal issue. Thus, the proposed method will not be reproducible. However, this is not the fault of this method.), and CoTracker. If more advanced techniques show up, one needs to tweak all the parameters of the entire training process to make the proposed method work. This makes this method not general 4. Following the previous comment, the proposed pipeline is more like an engineering work by combining existing techniques (SVD, ControlNet, CoTracker, CutMix) and using a massive computational power (4x A100), directly training for a long time (6 days) on a largescale dataset (WebVid). I really appreciate the edited video results. Some of them are amazing. In some of them, I still see artifacts (which might be the cons from SVD). In the entire paper, I do not see many novelties other than architectural designs and training data augmentation. Due to the above reasons, I am still leaning towards rejection. However, I would like to hear and discuss more with the authors and other reviewers. Best, Reviewer ZSh9 [1] Yun, Sangdoo, et al. "Cutmix: Regularization strategy to train strong classifiers with localizable features." Proceedings of the IEEE/CVF international conference on computer vision. 2019.
Thank you for your rebuttal and efforts to address the concerns raised. I still believe this work is timely and addresses an important topic. After considering the comments from other reviewers and your rebuttal, my opinion and suggestions are as follows: 1. The task scope and editing capabilities need to be defined more precisely. The initial submission did not clearly discuss the limitations and capabilities in the introduction and conclusion sections. 2. While the decoupling method is well-motivated, the proposed solution is straightforward and similar to proposals seen in many common computer vision tasks. There is little technical analysis and theoretical explanation provided, especially regarding potential artifacts introduced by the proposed method. 3. Given the current state of the field, the architecture is reasonable and enables motion control by trajectory, despite its complexity and the need for tuning many hyperparameters. However, a more concise solution may emerge in the future. To facilitate reproducibility, I recommend providing more detailed documentation of the training details in the paper.
Thank you for your constructive suggestions. Your very detailed comments in the review improve our paper to a higher standard. We will make a revision accordingly.
Dear reviewers, I would like to further explain our proposed three-stage training strategy. Its contributions mainly lie in two aspects: 1. We discover a significant coupling between content and motion in this task, which is an attribute difficult to detect. As described in Section 3.2, we find this attribute/issue after many failed attempts. This priori exploration can inspire future research in this community. 2. The proposed method successfully tackles an unexplored and challenging task. Each stage is validated and analyzed in Section 3.2. The design of each stage is effective and innovative given the current state of this research field.
Decision
Accept (poster)