VIDEO GENERATION WITH LATENT DIFFUSION PROBABILISTIC MODELS

Methods and systems for training a model include training an encoder in an unsupervised fashion based on a backward latent flow between a reference frame and a driving frame taken from a same video. A diffusion model is trained that generates a video sequence responsive to an input image and a text condition, using the trained encoder to determine a latent flow sequence and occlusion map sequence of a labeled training video.