On the Impact of Regularization Variation on Localized Multiple Kernel Learning

Abstract

This brief analyzes the effects of regularization variations in the localized kernel weights on the hypothesis generated by localized multiple kernel learning (LMKL) algorithms. Recent research on LMKL includes imposing different regularizations on the localized kernel weights and has led to varying formulations and solution strategies. Following the stability analysis theory as presented by Bousquet and Elisseeff, we give stability bounds based on the norm of the variation of localized kernel weights for three LMKL methods cast in the support vector machine classification framework, including vector <inline-formula> <tex-math notation="LaTeX">$\ell _{p}$ </tex-math></inline-formula>-norm LMKL, matrix-regularized <inline-formula> <tex-math notation="LaTeX">$(r,p)$ </tex-math></inline-formula>-norm LMKL, and samplewise <inline-formula> <tex-math notation="LaTeX">$\ell _{p}$ </tex-math></inline-formula>-norm LMKL. Further comparison of these bounds helps to qualitatively reveal the performance differences produced by these regularization methods, that is, matrix-regularized LMKL achieves superior performance, followed by vector <inline-formula> <tex-math notation="LaTeX">$\ell _{p}$ </tex-math></inline-formula>-norm LMKL and samplewise <inline-formula> <tex-math notation="LaTeX">$\ell _{p}$ </tex-math></inline-formula>-norm LMKL. Finally, a set of experimental results on ten benchmark machine learning UCI data sets is reported and shown to empirically support our theoretical analysis.