FutureGAN: Anticipating the Future Frames of Video Sequences using Spatio-Temporal 3d Convolutions in Progressively Growing GANs

We introduce a new encoder-decoder GAN model, FutureGAN, that predicts future\nframes of a video sequence conditioned on a sequence of past frames. During\ntraining, the networks solely receive the raw pixel values as an input, without\nrelying on additional constraints or dataset specific conditions. To capture\nboth the spatial and temporal components of a video sequence, spatio-temporal\n3d convolutions are used in all encoder and decoder modules. Further, we\nutilize concepts of the existing progressively growing GAN (PGGAN) that\nachieves high-quality results on generating high-resolution single images. The\nFutureGAN model extends this concept to the complex task of video prediction.\nWe conducted experiments on three different datasets, MovingMNIST, KTH Action,\nand Cityscapes. Our results show that the model learned representations to\ntransform the information of an input sequence into a plausible future sequence\neffectively for all three datasets. The main advantage of the FutureGAN\nframework is that it is applicable to various different datasets without\nadditional changes, whilst achieving stable results that are competitive to the\nstate-of-the-art in video prediction. Our code is available at\nhttps://github.com/TUM-LMF/FutureGAN.\n