Deep learning has thrived by training on large-scale datasets. However, in\nmany applications, as for medical image diagnosis, getting massive amount of\ndata is still prohibitive due to privacy, lack of acquisition homogeneity and\nannotation cost. In this scenario, transfer learning from natural image\ncollections is a standard practice that attempts to tackle shape, texture and\ncolor discrepancies all at once through pretrained model fine-tuning. In this\nwork, we propose to disentangle those challenges and design a dedicated network\nmodule that focuses on color adaptation. We combine learning from scratch of\nthe color module with transfer learning of different classification backbones,\nobtaining an end-to-end, easy-to-train architecture for diagnostic image\nrecognition on X-ray images. Extensive experiments showed how our approach is\nparticularly efficient in case of data scarcity and provides a new path for\nfurther transferring the learned color information across multiple medical\ndatasets.\n