This paper discusses how distribution matching losses, such as those used in\nCycleGAN, when used to synthesize medical images can lead to mis-diagnosis of\nmedical conditions. It seems appealing to use these new image synthesis methods\nfor translating images from a source to a target domain because they can\nproduce high quality images and some even do not require paired data. However,\nthe basis of how these image translation models work is through matching the\ntranslation output to the distribution of the target domain. This can cause an\nissue when the data provided in the target domain has an over or under\nrepresentation of some classes (e.g. healthy or sick). When the output of an\nalgorithm is a transformed image there are uncertainties whether all known and\nunknown class labels have been preserved or changed. Therefore, we recommend\nthat these translated images should not be used for direct interpretation (e.g.\nby doctors) because they may lead to misdiagnosis of patients based on\nhallucinated image features by an algorithm that matches a distribution.\nHowever there are many recent papers that seem as though this is the goal.\n