VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing

Summary

This paper introduces ST-Modulator, a zero-shot framework for multi-grained video editing, enabling class-level, instance-level, and part-level modifications in diffusion models. To address semantic misalignment and feature coupling issues, ST-Modulator modulates space-time cross- and self-attention, allowing fine-grained text-to-region control and improved feature separation. Experimental results suggest that this approach performs competitively across benchmarks without additional parameter tuning, presenting a promising solution for multi-grained video editing.

Strengths

- The paper clearly defines the novel task of multi-grained video editing, motivated by practical needs in video editing. By enabling edits at class-level, instance-level, and part-level, it broadens the range of possible video modifications to better meet real-world demands. - The paper effectively uses visualizations to illustrate key points, such as in the *Analysis of why the diffusion model failed in instance-level video editing.* These visual aids provide intuitive insights into challenges with existing diffusion models and help clarify the motivation behind the proposed approach. - The proposed Spatial-Temporal Layout-Guided Attention focuses on self-attention and cross-attention mechanisms to address key issues in text-to-region control and inter-region feature separation. This method effectively targets the core issues in multi-grained editing within diffusion models. - The proposed method does not require parameter tuning, which makes it resource-efficient—a significant advantage for video editing applications where computational cost and tuning complexity are critical factors.

Weaknesses

1. **Focus on Image Editing over Video-Specific Needs:** The approach seems more aligned with image editing, as it emphasizes spatial layout control. However, a key distinction in video editing is maintaining inter-frame consistency, which is crucial for coherent video results. While the Spatial-Temporal Layout-Guided Attention is introduced, the temporal aspect appears to receive less emphasis or detailed explanation. 2. **Lack of Clarification on Additional Control Signals:** The authors mention that the method is compatible with ControlNet conditioning, but it remains unclear if the cases shown in the paper require extra control inputs. There is no ablation study on control conditions to assess the necessity or influence of these signals. This raises questions about fairness in comparisons with other models like ControlVideo, where discrepancies in control conditions might lead to potentially biased results. 3. **Dependence on SAM-Track for Segmentation:** The method relies on SAM-Track for instance segmentation, yet no ablation study examines its impact. This dependency raises concerns: it is unclear if the observed performance gain primarily stems from using SAM-Track. For example, using SAM-Track segmentations as conditional input to other video editing models might yield similar improvements, suggesting a need for direct comparisons to validate the unique contributions of the proposed approach. 4. **Lack of Code or Algorithm Details:** The paper does not provide code or a detailed algorithm description. Open-sourcing the code could help address concerns such as those in Weaknesses 2 and 3 by allowing for transparent validation of the method’s assumptions, control conditions, and segmentation impacts.

Questions

See weaknesses.

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Summary

This paper presents a new task called multi-grained video editing, which encompasses class, instance and fine-grained video editing. The empirical study shows that the key obstacles hindering multi-grained editing in diffusion models are text-to-region misalignment and feature coupling. This work utilizes a pretrained T2I diffusion model to address these problems without any training. More Specifically ,the method modulates cross-attention for text-to-region control and self-attention for feature separation. Effectiveness has been proven by extensive experiments and convincing qualitative results.

Strengths

1. This paper addresses a highly significant problem of editing granularity in video editing. It proposes a new and interesting task called multi-grained editing, which includes class, instance, and part-level editing. It allows flexible editing at any granularity and breaks the inherent feature coupling within diffusion models. 2. The empirical study is insightful, explaining why diffusion models are limited to class-level video editing. 3. The proposed method is novel and consistent with its motivation. It adjusts cross-attention for accurate attention weight distribution, enables text-to-region control, and modulates self-attention for feature separation. The modulation process is operated in a unified manner, and the overall framework is training-free. 4. The qualitative experiments are impressive, indicating that ST-Modulator surpasses previous approaches in multi-grained editing.

Weaknesses

1. Since this work proposes a new and interesting task focused on multi-grained editing, a holistic evaluation would be beneficial, such as establishing a benchmark and developing new metrics to evaluate the performance of current methods on multi-grained video editing. 2. ST-Modulator focuses on editing videos at different spatial granularities, but the temporal length is limited to 16-32 frames. I wonder whether ST-Modulator can be extended to handle longer video sequences, such as those with hundreds of frames. 3. What is the difference between multi-grained editing and multi-attribute editing?

Questions

Please kindly address the questions in the weakness section.

Rating

8

Confidence

5

Soundness

3

Presentation

4

Contribution

4

Summary

This paper presents a zero-shot approach that modulates attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt’s attention to its corresponding spatial disentangled region while minimizing interactions with irrelevant areas. Extensive experiments demonstrate the method achieves state-of-the-art performance in real-world scenarios.

Strengths

1. This is the first attempt at multi-grained video editing. 2. The results show the SOTA performance on existing benchmarks and real-world videos. 3. The method is intuitive and is easy to follow.

Weaknesses

1. The readability and presentation of the manuscript need to be enhanced. For example, in Figure 4: The "I" in "M_i^pos" denotes the i-th frame, whereas the "i" in "cross-attention modulation" appears to refer to the i-th token (e.g., "polar" is the 3rd token?). Readers should not have to guess what these symbols mean; it would be beneficial to clarify these points explicitly. The "E" in "L × E" is undefined within the context of this paper, and Figure 2 does not utilize this information. Consequently, the shape information in Figure 4 could be omitted to avoid confusion. It is unclear whether "p" in Figure 4 represents the original attention score. This should be clearly stated to avoid ambiguity. 2. The paper employs an additional model, SAM-Track, to perform instance segmentation, which might be unfair. If this is the case, one could directly use Grounding-DINO in conjunction with SAM-Track to accurately identify the edited area (mask) based on the provided text. Subsequently, methods like prompt-to-prompt or Video-P2P could be employed to control the editing process through attention maps, potentially achieving similar results. 3. A key concept of the paper is to enhance attention scores in relevant areas while suppressing them in irrelevant areas. However, it's worth noting that methods such as prompt-to-prompt already implement mechanisms to either reduce or increase attention scores for editing purposes. This aspect should be discussed more thoroughly to highlight the novelty and added value of the proposed approach.

Questions

See weakness.

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Summary

This paper introduces a new task called multi-grained video editing, which includes class-level, instance-level, and part-level editing, and proposes a zero-shot approach, ST-Modulator, to address the challenge of distinguishing distinct instances (e.g., "left man" and "right man") by modulating space-time attention mechanisms for precise, fine-grained control over video content—all without additional training.

Strengths

- The paper is well-written and organized, making complex concepts easy to understand. - The Spatial-Temporal Layout-Guided Attention method in this paper effectively addresses the challenge of precise, multi-grained video editing by modulating cross- and self-attention. This approach enables accurate text-to-region control and clear feature separation, allowing independent edits to specific subjects while avoiding unintended changes in other areas, especially in complex multi-subject scenes. - Compared to other approaches, ST-Modulator achieves high computational efficiency, showing lower memory usage and faster processing times.

Weaknesses

- Limited focus on background preservation. - In Figures 5 and 6, most examples demonstrate edits that include modifications to the background along with the main subjects. Could the authors provide additional examples where the editing focuses solely on specific subjects, allowing the background to remain unchanged? This would help illustrate the method’s capability for selective edits in multi-subject scenes.

Questions

- For part-level editing, is the method limited to adding objects, or can it also support modifications like changing the color of clothing or the color of an animal?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

General Response

We thank all the reviewers for their valuable reviews. We also appreciate their recognition of the key contributions of our framework and the superiority of our results. Overall, we are encouraged that they find that: 1. **Contributions to the new task**: This paper proposes a new and interesting task called multi-grained editing, which includes class-level, instance-level, and part-level editing, motivated by practical needs in video editing. (all reviewers) 2. The **novelty** of ST-Modulator: The proposed ST-Layout Attn is novel and effectively addresses the key challenges of precise, multi-grained video editing by modulating cross- and self-attention.” (Reviewer cwR6, T7q7, 4zxu) 3. **Superior performance** of ST-Modulator: “The results show the SOTA performance and efficiency on existing benchmarks and real-world videos.” (Reviewer cwR6, XFkN, T7q7)) 4. The merit of **training-free** framework: The proposed method does not require parameter tuning, a significant advantage for video editing (Reviewer cwR6, T7q7,4zxu) 5. The paper is well-written, and the empirical study is insightful, making complex concepts easy to understand.(Reviewer cwR6, T7q7, 4zxu) According to our understanding, the main concern from reviewers is: **Can other methods achieve similar results when equipped with SAM-Track instance masks?** We conducted thorough experiments with VideoP2P equipped with SAM-Track masks (Figure 9) and Ground-A-Video (Figure 10, 11). The results show that both methods failed in multi-grained edits, even with instance masks. While SAM-Track provides layout guidance, it does not guarantee successful edits. Furthermore, as demonstrated in Appendix B, our method can achieve high-quality results even without SAM-Track. Here, we have summarized the changes to our manuscript. Modified or added contents have been highlighted blue in the revised paper. Point-to-point responses were also included as a reply to each reviewer. 1. In **Appendix A** and **Figure 9**, we conducted experiments with VideoP2P equipped with SAM-Track masks and visualized cross-attn weight to illustrate the difference between P2P and ours. (Reviewer XFkN, 4zxu) 2. In **Appendix B** and **Figure 11**, we provide examples without sam-track masks and compare Ground-A-Video with instance conditions. (Reviewer 4zxu) 3. In **Appendix C**, we provided solely editing examples without changing the background(Reviewer cwR6) 4. In **Appendix D** and **Figure 1, 13,**, we expand part-level definition (Figure1 abd Line75) and add part-level modification examples. (Reviewer cwR6) 5. In **Figure 4** and **Section 3.4**, we revised the notations to clarify. (Reviewer XFkN) 6. In **Appendix E**, we provide the temporal focus of ST-Layout Attn (Reviewer 4zxu) 7. In **Appendix F**, we ablate the control condition signals. (Reviewer 4zxu) 8. We have already released the code at an anonymous link.