Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources

Current transfer learning methods are mainly based on finetuning a pretrained\nmodel with target-domain data. Motivated by the techniques from adversarial\nmachine learning (ML) that are capable of manipulating the model prediction via\ndata perturbations, in this paper we propose a novel approach, black-box\nadversarial reprogramming (BAR), that repurposes a well-trained black-box ML\nmodel (e.g., a prediction API or a proprietary software) for solving different\nML tasks, especially in the scenario with scarce data and constrained\nresources. The rationale lies in exploiting high-performance but unknown ML\nmodels to gain learning capability for transfer learning. Using zeroth order\noptimization and multi-label mapping techniques, BAR can reprogram a black-box\nML model solely based on its input-output responses without knowing the model\narchitecture or changing any parameter. More importantly, in the limited\nmedical data setting, on autism spectrum disorder classification, diabetic\nretinopathy detection, and melanoma detection tasks, BAR outperforms\nstate-of-the-art methods and yields comparable performance to the vanilla\nadversarial reprogramming method requiring complete knowledge of the target ML\nmodel. BAR also outperforms baseline transfer learning approaches by a\nsignificant margin, demonstrating cost-effective means and new insights for\ntransfer learning.\n