Recent image inpainting methods have made great progress but often struggle\nto generate plausible image structures when dealing with large holes in complex\nimages. This is partially due to the lack of effective network structures that\ncan capture both the long-range dependency and high-level semantics of an\nimage. We propose cascaded modulation GAN (CM-GAN), a new network design\nconsisting of an encoder with Fourier convolution blocks that extract\nmulti-scale feature representations from the input image with holes and a\ndual-stream decoder with a novel cascaded global-spatial modulation block at\neach scale level. In each decoder block, global modulation is first applied to\nperform coarse and semantic-aware structure synthesis, followed by spatial\nmodulation to further adjust the feature map in a spatially adaptive fashion.\nIn addition, we design an object-aware training scheme to prevent the network\nfrom hallucinating new objects inside holes, fulfilling the needs of object\nremoval tasks in real-world scenarios. Extensive experiments are conducted to\nshow that our method significantly outperforms existing methods in both\nquantitative and qualitative evaluation. Please refer to the project page:\n\\url{https://github.com/htzheng/CM-GAN-Inpainting}.\n