Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions

Generative Adversarial Networks (GANs) are commonly used for modeling complex\ndistributions of data. Both the generators and discriminators of GANs are often\nmodeled by neural networks, posing a non-transparent optimization problem which\nis non-convex and non-concave over the generator and discriminator,\nrespectively. Such networks are often heuristically optimized with gradient\ndescent-ascent (GDA), but it is unclear whether the optimization problem\ncontains any saddle points, or whether heuristic methods can find them in\npractice. In this work, we analyze the training of Wasserstein GANs with\ntwo-layer neural network discriminators through the lens of convex duality, and\nfor a variety of generators expose the conditions under which Wasserstein GANs\ncan be solved exactly with convex optimization approaches, or can be\nrepresented as convex-concave games. Using this convex duality interpretation,\nwe further demonstrate the impact of different activation functions of the\ndiscriminator. Our observations are verified with numerical results\ndemonstrating the power of the convex interpretation, with applications in\nprogressive training of convex architectures corresponding to linear generators\nand quadratic-activation discriminators for CelebA image generation. The code\nfor our experiments is available at https://github.com/ardasahiner/ProCoGAN.\n