How to tune the RBF SVM hyperparameters?: An empirical evaluation of 18 search algorithms

SVM with an RBF kernel is usually one of the best classification algorithms\nfor most data sets, but it is important to tune the two hyperparameters $C$ and\n$\\gamma$ to the data itself. In general, the selection of the hyperparameters\nis a non-convex optimization problem and thus many algorithms have been\nproposed to solve it, among them: grid search, random search, Bayesian\noptimization, simulated annealing, particle swarm optimization, Nelder Mead,\nand others. There have also been proposals to decouple the selection of\n$\\gamma$ and $C$. We empirically compare 18 of these proposed search algorithms\n(with different parameterizations for a total of 47 combinations) on 115\nreal-life binary data sets. We find (among other things) that trees of Parzen\nestimators and particle swarm optimization select better hyperparameters with\nonly a slight increase in computation time with respect to a grid search with\nthe same number of evaluations. We also find that spending too much\ncomputational effort searching the hyperparameters will not likely result in\nbetter performance for future data and that there are no significant\ndifferences among the different procedures to select the best set of\nhyperparameters when more than one is found by the search algorithms.\n