Our goal in this work is to generate realistic videos given just one initial\nframe as input. Existing unsupervised approaches to this task do not consider\nthe fact that a video typically shows a 3D environment, and that this should\nremain coherent from frame to frame even as the camera and objects move. We\naddress this by developing a model that first estimates the latent 3D structure\nof the scene, including the segmentation of any moving objects. It then\npredicts future frames by simulating the object and camera dynamics, and\nrendering the resulting views. Importantly, it is trained end-to-end using only\nthe unsupervised objective of predicting future frames, without any 3D\ninformation nor segmentation annotations. Experiments on two challenging\ndatasets of natural videos show that our model can estimate 3D structure and\nmotion segmentation from a single frame, and hence generate plausible and\nvaried predictions.\n