Summary
The paper proposes a new method for training sparse Gaussian processes (GPs) for large-budget Bayesian Optimization (BO), designed to facilitate the sequential decision tasks inherent in BO. This is achieved by introducing the “expected utility lower bound (EULBO),” which formulates sparse GP training as a joint optimization problem of BO queries and posterior approximation. The proposed framework is compatible with other significant advancements in the field, such as Expected Improvement (EI) and Knowledge Refinement (KR) acquisition functions, TuRBO, and batch optimization. This compatibility was demonstrated across several BO benchmarks.
Strengths
The method proposed in the paper is novel and original. One of the main strengths is how the newly proposed EULBO integrates with important points in the literature. The authors effectively demonstrate how EULBO can be used in various contexts, such as with EI, KR acquisitions, and batch optimization. This helps distinguish their work from others in the field. The submission is technically sound, with derivations and experiments convincingly supporting the claims of the paper. The work is significant, and I anticipate it will be extended in the future or that some of the ideas presented will be adopted elsewhere.
Weaknesses
While the paper is generally well-written, many derivations were abbreviated and some of the notation was not properly explained, making the paper more difficult to read than necessary. These issues, however, are easily amendable. I provide some suggestions for improvement in the following subsections.
Questions
Introduction:
(line 51): '\emph{minorize-maximize}'
Section 2:
(Fig. 1): It would be beneficial to show the SVGP model fit for EULBO as well.
(line 110): Could you define \lambda, m and S?
(Eq. between 114-115): q_\lambda is initially presented with one argument, then two. If the intention is for q_\lambda to represent all approximate distributions (in this case, p(f, u) ), please state this explicitly. It is natural to assume that q_\lambda(.) is just the Gaussian pdf (as defined in line 110), which creates confusion.
(line 116): k(.,.) as a function also needs to be explicitly defined.
Section 3:
(line 123): This should probably refer to the equation between lines 114-115, not Equation 3.
(Eq 4): Please define \Lambda.
(Eq 6): Please use double integrals to indicate nested integration. Additionally, I don't understand the emergence of l(D_t |f) and the normalizing constant Z. I was under the impression that l(.|.) is the Gaussian likelihood as defined in line 113. Notice that D_t, as defined in line 92, contains both x and y.
(line 147): 'corresponds to'
(line 165): Please clarify where exactly the requirement for the utility to be strictly positive plays a role or comes from.
(lines 173-175): Would using Gauss-Hermite quadrature for EI make EULBO significantly slower than ELBO?
(line 189): What is y^{+}_t and what its relation to y_t?
Question:
Because you formulate the BO queries and GP approximation as a joint optimization problem without necessarily reducing the dimensionality of the problem, how much slower is this method compared to ‘approximation-ignorant’ BO?
Limitations
The authors are open about the limitations of their work, which they leave for future exploration. Specifically, the increased computational complexity seems to be a primary limitation. It would be beneficial if the authors could quantify this increased complexity, perhaps through reporting empirical runtimes or other relevant metrics.