Deriving Realistic Mathematical Models from Support Vector Machines for Scientific Applications
In many scientific applications, it is necessary to perform classification, which means discrimination between examples belonging to different classes. Machine Learning Tools have proved to be very performing in this task and can achieve very high success rates. On the other hand, the “realism” and interpretability of their results are very low, limiting their applicability. In this paper, a method to derive manageable equations for the hypersurface between classes is presented. The main objective consists of formulating the results of machine learning tools in a way representing the actual “physics” behind the phenomena under investigation. The proposed approach is based on a suitable combination of Support vector Machines and Symbolic Regression via Genetic Programming; it has been investigated with a series of systematic numerical tests, for different types of equations and classification problems, and tested with various experimental databases. The obtained results indicate that the proposed method permits to find a good trade-off between accuracy of the classification and complexity of the derived mathematical equations. Moreover, the derived models can be tuned to reflect the actual phenomena, providing a very useful tool to bridge the gap between data, machine learning tools and scientific theories. 1 THE NEED FOR DATA MINING TOOLS IN BIG PHYSICS EXPERIMENTS In many fields of science, the complexity of the problems investigated is such that it can become difficult, if not impossible, to describe the phenomena to be studied with theoretical models based on first principles. A typical example in physics is the case of magnetic confinement thermonuclear fusion, whose plasmas are so complex that various levels of modelling (particle, fluid, kinetic etc) coexist without providing a satisfactory description of many aspects of the physics (Wesson, 2004). On the other hand, in the last decades much more data have become available, due to the diffusion of cheap sensors and powerful computers. For example, the Big Physics European experiments are affected by a data deluge. At CERN, the ATLAS detector can produce Petabytes of data per year. The Hubble space telescope managed to send to earth Gigabytes of data per day and the data warehouse of the Joint European Torus exceeds 350 Terabytes. Therefore the inadequacies of theoretical models and the vast amounts of information available have motivated the development of data driven tools, to complement hypothesis driven theories. In this perspective, various machine learning methods have been developed. They range from Neural Networks and Support Vector Machines to Fuzzy Logic classifiers; a series of examples from the field of thermonuclear fusion can be found in (Rattà, 2010; Vega, 2014; Murari, 2009). Manifold learning tools, such as Self Organising Maps and Generative Topographic Maps, and simple classifiers based on the Geodesic distance on Gaussian manifolds, have provided very Murari A., Peluso E., Talebzadeh S., Gaudio P., Lungaroni M., Mikulin O., Vega J. and Gelfusa M. Deriving Realistic Mathematical Models from Support Vector Machines for Scientific Applications. DOI: 10.5220/0006517401020113 In Proceedings of the 9th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (KDIR 2017), pages 102-113 ISBN: 978-989-758-271-4 Copyright c © 2017 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved good results also in terms of describing the space in which the relevant physics takes place (Cannas, 2013; Murari (A), 2013; Vega, 2009). Even if these data driven tools are providing quite impressive performance, their main problem is the mathematical formulation of their models. They have shown the potential to learn very efficiently from the provided examples but their results are expressed in such a way that does not allow an easy interpretation of the physics behind the phenomena under study. This aspect is quite worrying and has hampered the penetration of many machine learning tools in scientific disciplines such as physics. Some of the main problems are: a) poor “physics fidelity” i.e. excessive difference between the mathematical form of the models and the physical reality of the phenomena investigated b) difficulties to interpret the results in terms of traditional mathematical formulations c) consequent impossibility to compare the obtained results with traditional mathematical models and theories d) lack of extrapolability of the results. In order to overcome these limitations, a new methodology has been developed to profit from the knowledge acquired by the machine learning tools, but presenting it in a more traditional format, in terms of manageable formulas. The techniques, developed in the framework of the activities presented in this paper, address the basic goal of classification. This is a very important task in many scientific applications, both “per se” and as a preliminary step to more sophisticated investigations. The main idea behind this work consists therefore of combining the learning capabilities of the machine learning tools with the “realism” and interpretability of more traditional mathematical formulations. The sequence of steps required to implement the proposed technique is: 1. Training of the SVM with the available data; 2. Populating the hypersurface on a suitable grid of points 3. Identification of the hypersurface equation with Symbolic Regression via Genetic Programming 4. Double checking of the obtained equation using the SVM This approach reconciles the prediction and knowledge discovery capability of machine learning tools with the need to formulate the results in such a way that they can be related to scientific theories and models. It is worth emphasizing that the objective of the present work is not simply improving interpretability of machine learning tools, on which significant work has already been done. The most important aspect indeed is “physics fidelity” i.e. the formulation of the results in mathematical terms, which can be compared with theories and models of the various scientific disciplines. Therefore, the proposed method must have the potential to derive mathematical expressions, which reflect the underlining dynamics of the phenomena to be investigated. In the presented approach, the first knowledge discovery step is based on Support Vector Machines (SVM), whose mathematical background is summarized in Section 2. The choice of SVM is mainly due to their structural stability, their capability to maximize the safety margins in the classification. Indeed in many applications, SVM can classify with a success rate well in excess of 95%; therefore their hyperplane can be considered a good approximation of the boundary between the classes (Murari (A), 2013). As a consequence, the equation of their hypersurface in the original space can be considered an excellent approximation of the boundary. On the other hand, their mathematical representation of the boundary is extremely non intuitive (see Section 2). Again referring to a complex system of the complexity of the Joint European Torus, in the case of disruptions, the equation of the hypersurface can comprise hundreds of support vectors and therefore the equation of the hypersurface contains an equal number of addends. More importantly, in addition to presenting serious problems for human understanding, the SVM models do not reflect the actual dynamics of the phenomena under study. It has indeed been shown, also with many numerical examples (see Section 6), that the models provided by SVM have absolutely no relation with the ones generating the data. A simple methodology has already been proposed and applied to complex problems, to recover the equation of the boundary in the case of linear kernels (Gaudio, 2014). In this paper, a new technique is developed, which is fully general. Indeed the proposed method can be applied to SVM with any type of kernel and therefore it has a much wider range of applications than the more traditional techniques. This aspect is very important in many scientific fields, whose phenomena cannot be simply modelled by linear tools or logistic regression. To formulate the outputs of SVM in way suitable for scientific investigations, extensive used is made of Symbolic Regression (SR) via Genetic Programming (GP); these tools are therefore described in Section 3. Symbolic regression is basically used to fit points on the hypersurface found by the SVM, which is the boundary between the classes. The actual combination of the various tools, to provide the equation of the boundary between two regions of the operational space in a physically relevant form, is described in detail in Section 4. The results of a systematic series of numerical tests, proving the potential of the proposed methodology, are the subject of Section 5. Some examples of application to experimental databases, covering different scientific disciplines, are provided in Section 6. Discussions and lines of future developments are the subject of the last Section 7. 2 INTRODUCTION TO SVM FOR CLASSIFICATION SVM are mathematical tools which can perform more general tasks, such as regression, but which are used as classifiers for the studies described in this paper. In intuitive terms, given a set of input examples, which belong to two different classes, SVM map the inputs into a high-dimensional space through some suitable non-linear mapping. In this high dimensional feature space, an optimal separating hyperplane is constructed in order to minimize the risk of misclassification. The minimization of the error risk is obtained by maximizing the margins between the hyperplane and the closest points, the support vectors, of each class. This is achieved by a careful selection of the constraints of a suitable functional to minimize. In the case of non-separable problems, the points to classify are projected into a highe
Paper
Full text
Deriving Realistic Mathematical Models from Support Vector Machines for Scientific Applications
Semantic Scholar · Computer Science · 2017
Abstract
In many scientific applications, it is necessary to perform classification, which means discrimination between examples belonging to different classes. Machine Learning Tools have proved to be very performing in this task and can achieve very high success rates. On the other hand, the “realism” and interpretability of their results are very low, limiting their applicability. In this paper, a method to derive manageable equations for the hypersurface between classes is presented. The main objective consists of formulating the results of machine learning tools in a way representing the actual “physics” behind the phenomena under investigation. The proposed approach is based on a suitable combination of Support vector Machines and Symbolic Regression via Genetic Programming; it has been investigated with a series of systematic numerical tests, for different types of equations and classification problems, and tested with various experimental databases. The obtained results indicate that the proposed method permits to find a good trade-off between accuracy of the classification and complexity of the derived mathematical equations. Moreover, the derived models can be tuned to reflect the actual phenomena, providing a very useful tool to bridge the gap between data, machine learning tools and scientific theories. 1 THE NEED FOR DATA MINING TOOLS IN BIG PHYSICS EXPERIMENTS In many fields of science, the complexity of the problems investigated is such that it can become difficult, if not impossible, to describe the phenomena to be studied with theoretical models based on first principles. A typical example in physics is the case of magnetic confinement thermonuclear fusion, whose plasmas are so complex that various levels of modelling (particle, fluid, kinetic etc) coexist without providing a satisfactory description of many aspects of the physics (Wesson, 2004). On the other hand, in the last decades much more data have become available, due to the diffusion of cheap sensors and powerful computers. For example, the Big Physics European experiments are affected by a data deluge. At CERN, the ATLAS detector can produce Petabytes of data per year. The Hubble space telescope managed to send to earth Gigabytes of data per day and the data warehouse of the Joint European Torus exceeds 350 Terabytes. Therefore the inadequacies of theoretical models and the vast amounts of information available have motivated the development of data driven tools, to complement hypothesis driven theories. In this perspective, various machine learning methods have been developed. They range from Neural Networks and Support Vector Machines to Fuzzy Logic classifiers; a series of examples from the field of thermonuclear fusion can be found in (Rattà, 2010; Vega, 2014; Murari, 2009). Manifold learning tools, such as Self Organising Maps and Generative Topographic Maps, and simple classifiers based on the Geodesic distance on Gaussian manifolds, have provided very Murari A., Peluso E., Talebzadeh S., Gaudio P., Lungaroni M., Mikulin O., Vega J. and Gelfusa M. Deriving Realistic Mathematical Models from Support Vector Machines for Scientific Applications. DOI: 10.5220/0006517401020113 In Proceedings of the 9th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (KDIR 2017), pages 102-113 ISBN: 978-989-758-271-4 Copyright c © 2017 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved good results also in terms of describing the space in which the relevant physics takes place (Cannas, 2013; Murari (A), 2013; Vega, 2009). Even if these data driven tools are providing quite impressive performance, their main problem is the mathematical formulation of their models. They have shown the potential to learn very efficiently from the provided examples but their results are expressed in such a way that does not allow an easy interpretation of the physics behind the phenomena under study. This aspect is quite worrying and has hampered the penetration of many machine learning tools in scientific disciplines such as physics. Some of the main problems are: a) poor “physics fidelity” i.e. excessive difference between the mathematical form of the models and the physical reality of the phenomena investigated b) difficulties to interpret the results in terms of traditional mathematical formulations c) consequent impossibility to compare the obtained results with traditional mathematical models and theories d) lack of extrapolability of the results. In order to overcome these limitations, a new methodology has been developed to profit from the knowledge acquired by the machine learning tools, but presenting it in a more traditional format, in terms of manageable formulas. The techniques, developed in the framework of the activities presented in this paper, address the basic goal of classification. This is a very important task in many scientific applications, both “per se” and as a preliminary step to more sophisticated investigations. The main idea behind this work consists therefore of combining the learning capabilities of the machine learning tools with the “realism” and interpretability of more traditional mathematical formulations. The sequence of steps required to implement the proposed technique is: 1. Training of the SVM with the available data; 2. Populating the hypersurface on a suitable grid of points 3. Identification of the hypersurface equation with Symbolic Regression via Genetic Programming 4. Double checking of the obtained equation using the SVM This approach reconciles the prediction and knowledge discovery capability of machine learning tools with the need to formulate the results in such a way that they can be related to scientific theories and models. It is worth emphasizing that the objective of the present work is not simply improving interpretability of machine learning tools, on which significant work has already been done. The most important aspect indeed is “physics fidelity” i.e. the formulation of the results in mathematical terms, which can be compared with theories and models of the various scientific disciplines. Therefore, the proposed method must have the potential to derive mathematical expressions, which reflect the underlining dynamics of the phenomena to be investigated. In the presented approach, the first knowledge discovery step is based on Support Vector Machines (SVM), whose mathematical background is summarized in Section 2. The choice of SVM is mainly due to their structural stability, their capability to maximize the safety margins in the classification. Indeed in many applications, SVM can classify with a success rate well in excess of 95%; therefore their hyperplane can be considered a good approximation of the boundary between the classes (Murari (A), 2013). As a consequence, the equation of their hypersurface in the original space can be considered an excellent approximation of the boundary. On the other hand, their mathematical representation of the boundary is extremely non intuitive (see Section 2). Again referring to a complex system of the complexity of the Joint European Torus, in the case of disruptions, the equation of the hypersurface can comprise hundreds of support vectors and therefore the equation of the hypersurface contains an equal number of addends. More importantly, in addition to presenting serious problems for human understanding, the SVM models do not reflect the actual dynamics of the phenomena under study. It has indeed been shown, also with many numerical examples (see Section 6), that the models provided by SVM have absolutely no relation with the ones generating the data. A simple methodology has already been proposed and applied to complex problems, to recover the equation of the boundary in the case of linear kernels (Gaudio, 2014). In this paper, a new technique is developed, which is fully general. Indeed the proposed method can be applied to SVM with any type of kernel and therefore it has a much wider range of applications than the more traditional techniques. This aspect is very important in many scientific fields, whose phenomena cannot be simply modelled by linear tools or logistic regression. To formulate the outputs of SVM in way suitable for scientific investigations, extensive used is made of Symbolic Regression (SR) via Genetic Programming (GP); these tools are therefore described in Section 3. Symbolic regression is basically used to fit points on the hypersurface found by the SVM, which is the boundary between the classes. The actual combination of the various tools, to provide the equation of the boundary between two regions of the operational space in a physically relevant form, is described in detail in Section 4. The results of a systematic series of numerical tests, proving the potential of the proposed methodology, are the subject of Section 5. Some examples of application to experimental databases, covering different scientific disciplines, are provided in Section 6. Discussions and lines of future developments are the subject of the last Section 7. 2 INTRODUCTION TO SVM FOR CLASSIFICATION SVM are mathematical tools which can perform more general tasks, such as regression, but which are used as classifiers for the studies described in this paper. In intuitive terms, given a set of input examples, which belong to two different classes, SVM map the inputs into a high-dimensional space through some suitable non-linear mapping. In this high dimensional feature space, an optimal separating hyperplane is constructed in order to minimize the risk of misclassification. The minimization of the error risk is obtained by maximizing the margins between the hyperplane and the closest points, the support vectors, of each class. This is achieved by a careful selection of the constraints of a suitable functional to minimize. In the case of non-separable problems, the points to classify are projected into a highe