Implicit Differentiation for Hyperparameter Tuning the Weighted Graphical Lasso

We provide a framework and algorithm for tuning the hyperparameters of the Graphical Lasso via a bilevel optimization problem solved with a first-order method. In particular, we derive the Jacobian of the Graphical Lasso solution with respect to its regularization hyperparameters.

cs.LG math.OC stat.ML Mathematics Computer Science

Paper

References (19)

01Implicit differentiation for fast hyperparameter selection in non-smooth convex learningQuentin Bertrand, Quentin Klopfenstein, Mathurin Massias et al.2021 · Journal of machine learning research · 39 citations In Library

02The Generalized Ridge Estimator of the Inverse Covariance MatrixW. V. van Wieringen2019 · Journal of Computational And Graphical Statistics · 19 citations

03GLAD: Learning Sparse Graph RecoveryH. Shrivastava, Xinshi Chen, Binghong Chen et al.2019 · International Conference on Learning Representations · 39 citations In Library

04Proximal approaches for matrix optimization problems: Application to robust precision matrix estimationA. Benfenati, É. Chouzenoux, J. Pesquet2017 · Signal Processing · 27 citations

05First-Order Methods in OptimizationA. Beck2017 · 2.2k citations

06Forward and Reverse Gradient-Based Hyperparameter OptimizationLuca Franceschi, Michele Donini, P. Frasconi et al.2017 · International Conference on Machine Learning · 478 citations In Library

07Local Linear Convergence of Forward-Backward under Partial SmoothnessJingwei Liang, M. Fadili, G. Peyré2014 · Neural Information Processing Systems · 102 citations In Library

08Stein Unbiased GrAdient estimator of the Risk (SUGAR) for Multiple Parameter SelectionCharles-Alban Deledalle, Samuel Vaiter, M. Fadili et al.2014 · SIAM Journal of Imaging Sciences · 126 citations

09Newton-Like Methods for Sparse Inverse Covariance EstimationP. Olsen, Figen Öztoprak, J. Nocedal et al.2012 · Neural Information Processing Systems · 114 citations

10Practical Bayesian Optimization of Machine Learning AlgorithmsJasper Snoek, H. Larochelle, Ryan P. Adams2012 · Neural Information Processing Systems · 9.7k citations In Library

11Random Search for Hyper-Parameter OptimizationJ. Bergstra, Yoshua Bengio2012 · Journal of machine learning research · 11k citations

12Convex OptimizationStephen P. Boyd, L. Vandenberghe2010 · IEEE Transactions on Automatic Control · 41k citations

Scroll for more · 7 remaining

Similar papers

Implicit differentiation of Lasso-type models for hyperparameter optimizationQuentin Bertrand, Quentin Klopfenstein, Mathieu Blondel et al.2020 · International Conference on Machine Learning · 75 citations In Library

Efficient Regularization Parameter Selection for Latent Variable Graphical Models via Bi-Level OptimizationJoachim Giesen, Frank Nussbaum, Christopher Schneider2019 · International Joint Conference on Artificial Intelligence · 4 citations In Library

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine LearningAnkur Sinha, Tanmay Khandait, Raja Mohanty2020 · arXiv (Cornell University) · 22 citations In Library

Gradient-based bilevel optimization for multi-penalty Ridge regression through matrix differential calculusGabriele Maroni, Loris Cannelli, Dario Piga2024 · European Journal of Control In Library

Regularization Parameter Selection for a Bayesian Multi-Level Group Lasso Regression Model with Application to Imaging GenomicsF. Nathoo, K. Greenlaw, M. Lesperance2016 · International Workshop on Pattern Recognition in NeuroImaging · 5 citations In Library

Thresholded Adaptive Validation: Tuning the Graphical Lasso for Graph RecoveryMike Laszkiewicz, Asja Fischer, Johannes Lederer2020 · arXiv (Cornell University) · 8 citations In Library

Distributionally Robust Formulation and Model Selection for the Graphical LassoPedro Cisneros-Velarde, Sang-Yun Oh, Alexander Petersen2019 · International Conference on Artificial Intelligence and Statistics · 16 citations In Library

A Globally Convergent Gradient-based Bilevel Hyperparameter Optimization MethodAnkur Sinha, Satender Gunwal, Kumar, Shivam2022 · arXiv (Cornell University) · 3 citations In Library

Value Function Based Difference-of-Convex Algorithm for Bilevel Hyperparameter Selection ProblemsLucy L. Gao, Jane J. Ye, Haian Yin et al.2022 · International Conference on Machine Learning · 37 citations In Library

Implicit differentiation for fast hyperparameter selection in non-smooth convex learningQuentin Bertrand, Quentin Klopfenstein, Mathurin Massias et al.2021 · Journal of machine learning research · 39 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal