AutoWeka4MCPS-AVATAR: Accelerating Automated Machine Learning Pipeline Composition and Optimisation

Automated machine learning pipeline (ML) composition and optimisation aim at\nautomating the process of finding the most promising ML pipelines within\nallocated resources (i.e., time, CPU and memory). Existing methods, such as\nBayesian-based and genetic-based optimisation, which are implemented in\nAuto-Weka, Auto-sklearn and TPOT, evaluate pipelines by executing them.\nTherefore, the pipeline composition and optimisation of these methods\nfrequently require a tremendous amount of time that prevents them from\nexploring complex pipelines to find better predictive models. To further\nexplore this research challenge, we have conducted experiments showing that\nmany of the generated pipelines are invalid in the first place, and attempting\nto execute them is a waste of time and resources. To address this issue, we\npropose a novel method to evaluate the validity of ML pipelines, without their\nexecution, using a surrogate model (AVATAR). The AVATAR generates a knowledge\nbase by automatically learning the capabilities and effects of ML algorithms on\ndatasets' characteristics. This knowledge base is used for a simplified mapping\nfrom an original ML pipeline to a surrogate model which is a Petri net based\npipeline. Instead of executing the original ML pipeline to evaluate its\nvalidity, the AVATAR evaluates its surrogate model constructed by capabilities\nand effects of the ML pipeline components and input/output simplified mappings.\nEvaluating this surrogate model is less resource-intensive than the execution\nof the original pipeline. As a result, the AVATAR enables the pipeline\ncomposition and optimisation methods to evaluate more pipelines by quickly\nrejecting invalid pipelines. We integrate the AVATAR into the sequential\nmodel-based algorithm configuration (SMAC). Our experiments show that when SMAC\nemploys AVATAR, it finds better solutions than on its own.\n