Optimizing Approximate Leave-one-out Cross-validation to Tune Hyperparameters

For a large class of regularized models, leave-one-out cross-validation can\nbe efficiently estimated with an approximate leave-one-out formula (ALO). We\nconsider the problem of adjusting hyperparameters so as to optimize ALO. We\nderive efficient formulas to compute the gradient and hessian of ALO and show\nhow to apply a second-order optimizer to find hyperparameters. We demonstrate\nthe usefulness of the proposed approach by finding hyperparameters for\nregularized logistic regression and ridge regression on various real-world data\nsets.\n