Traditional approaches for complementary product recommendations rely on\nbehavioral and non-visual data such as customer co-views or co-buys. However,\ncertain domains such as fashion are primarily visual. We propose a framework\nthat harnesses visual cues in an unsupervised manner to learn the distribution\nof co-occurring complementary items in real world images. Our model learns a\nnon-linear transformation between the two manifolds of source and target\ncomplementary item categories (e.g., tops and bottoms in outfits). Given a\nlarge dataset of images containing instances of co-occurring object categories,\nwe train a generative transformer network directly on the feature\nrepresentation space by casting it as an adversarial optimization problem. Such\na conditional generative model can produce multiple novel samples of\ncomplementary items (in the feature space) for a given query item. The final\nrecommendations are selected from the closest real world examples to the\nsynthesized complementary features. We apply our framework to the task of\nrecommending complementary tops for a given bottom clothing item. The\nrecommendations made by our system are diverse, and are favored by human\nexperts over the baseline approaches.\n