Summary
The task addressed by this paper is, given the the appearance and segmentation mask of a foreground subject in a video, for example a human running, to synthesize video backgrounds that are plausible and realistic, both in content and motion. For example, for a person running, if the ground is wet there should be splashes corresponding to footfalls, and the background scene should move along with the runner's movement.
The authors address this problem as a generative video outpainting task. They model it using a latent video diffusion model. The foreground subject appearances (encoded) and masks are concatenated with the latent feature vector, while the conditioning image (what the background should look like) is encoded via CLIP and is provided via cross-attention to the U-net.
A variety of qualitative results are included, demonstrating the ability to out-paint plausible scenes, including background motion, effects (splashing water) and even objects that the subject interacts with. Quantitative evaluations include ablations validating the main components of the model, as well as human rater studies comparing the quality of the output to competing methods.
Strengths
Originality & Significance.
The paper addresses a minor new variant on the outpainting/inpainting problem, which introduces some new challenges such as picking up on foreground contextual cues and using that to inform the background.
Quality & Clarity:
The paper is clearly written, seems correct, and demonstrates visually appealing results with a variety of different foreground subjects (person, car, duck, etc.). The supplemental videos help illustrate the model's capabilities well. The empirical results also support the quality of the generative model.
Weaknesses
The main weakness of this paper is very limited novelty, which compromises the claimed contributions of the paper.
Video inpainting/outpainting is a fairly well-known challenge. The main novelty here is that the masks are simply inverted: instead of deleting an unwanted person and generating the missing background pixels, here we retain only the desired foreground person and generate the missing pixels. I am not convinced that this difference is substantively novel.
Similarly, the modeling approach bears considerable similarity to [13] "Structure and Content-Guided Video Synthesis with Diffusion Models". In particular, compare Fig.2 from the two papers. The primary difference between the two is that this work conditions on foreground appearances and masks, while [13] conditions on estimated depth images. Again, the novelty is minor and unsurprising.
Questions
Have you tried unconditional generation, i.e. not providing any signal as to what the desired background should be? This could be quite interesting, since it would challenge the model to understand exactly what the foreground subject is doing and to invent plausible backgrounds. For example, this child seems to be interacting with an object, what could that object be? This woman is riding something… what fits?
Would you like to discuss the appearance of the Shutterstock watermarks in the generated backgrounds, particularly in figure 3? This could be an interesting example of the model using foreground context informing the background generation, as per my previous question.