Real applicability: BO over molecular properties with Tanimoto kernel and more
Thank you very much for your reply! We go on to describe our ongoing work on the application of SGD GPs to molecular property prediction as well as how the method can be applied more generally, for instance, to spatiotemporal modeling.
----
1. We are currently working on **applying SGD inference to molecular binding energy prediction** using a dataset of 250k molecules introduced by [1]. We are using the Tanimoto kernel for graphs, which admits random features ([2]). In particular, we are searching for molecules which have a high probability of binding to proteins of interest using Bayesian Optimization.
* The Tanimoto kernel only has 1 hyperparameter, the marginal kernel variance. For this task, the **authors of [1] provide an optimized kernel hyperparameter** value, which they used in their experiments. Additionally, the **authors of [3] show how marginal kernel variances can be learnt using only GP posterior samples.** Thus, our SGD-based inference can be directly applied to this setting for learning the Tanimoto kernel’s hyperparameter.
* Although there is not enough time to conclude these experiments before the end of the discussion period (the 21st), we would be happy to include them in the camera-ready version of the paper.
2. **We think that GP inference methods can be useful even without hyperparameter learning.** A simple but general and effective approach to select hyperparameters is to **maximize the marginal likelihood on clustered subsets of the data**, as we do in our paper (See Appendix A.1). This yields results competitive with, and in some datasets outperforming, the hyperparameters learnt via conjugate gradients of [4]. This approach is particularly well-suited to length scale hyperparameters, which are of key importance in spatiotemporal modeling.
3. Next, **ill-conditioning appears consistently for large enough datasets or when the kernel distance between observations is small** (in fact, *often provably so* - see Section 2.3 of [5]), so one does not need to search particularly hard to find examples. The latter is bound to occur in Bayesian optimization, since methods often explore near previously-found well-performing locations.
4. Finally, there is strong precedent in the Gaussian processes where (a) **a novel method with significant advantages but important limitations was introduced**, and (b) **the limitations were addressed through follow-up work**.
* For example, Titsias [6] introduced the variational-inference-based view of sparse Gaussian processes, developing a novel formalism for inducing points, whose complexity is $O(NM^2)$ - larger than for instance certain subset-of-data methods. Then, Hensman et al. [7] reduced this to $O(M^3)$ - a major improvement when $N$ is in the millions - by applying stochastic optimization to the variational inference objective. Achieving this improvement was only possible because the variational viewpoint had been developed previously.
* Mirroring this example, **we expect that follow-up work, using for instance bilevel optimization techniques, can address limitations around hyperparameter learning** (for examples of such techniques in a neural network context, see [8,9]). This would start from the ideas we developed, but would likely introduce enough additional theoretical and methodological contributions, as well as experimental evaluation specific to hyperparameter optimization, to constitute another paper.
[1] *DOCKSTRING: Easy Molecular Docking Yields Better Benchmarks for Ligand Design.*
Miguel García-Ortegón*, Gregor N. C. Simm, Austin J. Tripp, José Miguel Hernández-Lobato, Andreas Bender, and Sergio Bacallado
[2] *Tanimoto Random Features for Scalable Molecular Machine Learning*.
Austin Tripp, Sergio Bacallado, Sukriti Singh, José Miguel Hernández-Lobato
[3] *Sampling-based inference for large linear models, with application to linearised Laplace.*
Javier Antorán, Shreyas Padhy, Riccardo Barbano, Eric Nalisnick, David Janz, José Miguel Hernández-Lobato
[4] *Exact Gaussian Processes on a Million Data Points.*
Ke Alexander Wang, Geoff Pleiss, Jacob R. Gardner, Stephen Tyree, Kilian Q. Weinberger, Andrew Gordon Wilson
[5] *Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees*.
Alexander Terenin, David R. Burt, Artem Artemev, Seth Flaxman, Mark van der Wilk, Carl Edward Rasmussen, Hong Ge
[6] *Variational learning of inducing variables in sparse Gaussian processes*. Michalis Titsias
[7] *Gaussian processes for big data*. James Hensman, Nicolò Fusi, Neil Lawrence.
[8] *Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation*. Ross M. Clarke, Elre T. Oldewage, José Miguel Hernández-Lobato
[9] *Generalized Inner Loop Meta-Learning*. Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, Soumith Chintala