Cross-domain disentanglement is the problem of learning representations\npartitioned into domain-invariant and domain-specific representations, which is\na key to successful domain transfer or measuring semantic distance between two\ndomains. Grounded in information theory, we cast the simultaneous learning of\ndomain-invariant and domain-specific representations as a joint objective of\nmultiple information constraints, which does not require adversarial training\nor gradient reversal layers. We derive a tractable bound of the objective and\npropose a generative model named Interaction Information Auto-Encoder (IIAE).\nOur approach reveals insights on the desirable representation for cross-domain\ndisentanglement and its connection to Variational Auto-Encoder (VAE). We\ndemonstrate the validity of our model in the image-to-image translation and the\ncross-domain retrieval tasks. We further show that our model achieves the\nstate-of-the-art performance in the zero-shot sketch based image retrieval\ntask, even without external knowledge. Our implementation is publicly available\nat: https://github.com/gr8joo/IIAE\n