Gaussian processes are the model of choice in Bayesian optimization and active learning. Yet, they are highly dependent on cleverly chosen hyperparameters to reach their full potential, and little effort is devoted to finding good hyperparameters in the literature. We demonstrate the impact of selecting good hyperparameters for GPs and present two acquisition functions that explicitly prioritize hyperparameter learning. Statistical distance-based Active Learning (SAL) considers the average disagreement between samples from the posterior, as measured by a statistical distance. SAL outperforms the state-of-the-art in Bayesian active learning on several test functions. We then introduce Self-Correcting Bayesian Optimization (SCoreBO), which extends SAL to perform Bayesian optimization and active learning simultaneously. SCoreBO learns the model hyperparameters at improved rates compared to vanilla BO, while outperforming the latest Bayesian optimization methods on traditional benchmarks. Moreover, we demonstrate the importance of self-correction on atypical Bayesian optimization tasks.
Paper
Similar papers
Peer review
Summary
The paper proposed a novel function SAL used for active learning based on distribution difference. Then the author further incorporated the SAL function into Bayesian optimization (BO) to improve BO performance. Most numerical experiments are conducted on synehtic benchmakr functions. Better performances are achieved by using the proposed method compared to several baseline functions.
Strengths
Overall, the paper is easy to follow despite several confusions.
Weaknesses
Lack of real-world experiments. I would suggest the authors be precisely clear on their symbols. I have some questions about the technical details.
Questions
In general, I grasp the main idea of this paper. However, I suggest that the authors distinguish among three different things: (i) the true underlying function $f(x)$, (ii) the observed noisy function value $f(x)+\epsilon$, and (iii) the GP model $GP(x)$, which is an approximation to $f(x)$. The authors should carefully examine the usage of each symbol throughout their paper and ensure that they properly distinguish between (i)-(iii). Currently, the authors sometimes misuse these symbols, which hinders the understanding of the paper. Below, I have listed several questions and would appreciate it if the authors could provide answers. This will help me gain a better understanding and assessment of the paper. 1. The authors state that the mean function $m(\cdot)$ is set to a constant. Why is this constant not included in the set of hyperparameters? 2. The numerical experiments primarily focus on benchmark functions, with the exception of DNA classification task and cosmological estimation. It would be beneficial to include more real-world experiments. For example, Bayesian optimization is commonly used for tuning network parameters in automl. It would be valuable to add experiments similar to that. 3. What is the specific form of $p(\theta|D)$ in step 4 shown in Algorithm 1? 4. What is $p(f|\theta,D)$ in Step 6? And what does Step 6 do? The symbol $f$ is used to represent the underlying function in equation (1). I am confused about how to sample an $f$ from $p(\cdot|\theta,D)$. Since $f(\cdot)$ represents the underlying function and does not depend on the GP model, I assume that $p(f|\theta,D)$ means sampling a GP instance given $\theta$ and $D$. If this is the case, I suggest the authors use another symbol instead of $f$ here. 5. One advantage of Bayesian optimization is that its acquisition function does not require calling the real function. Instead, it only needs predictions based on the GP (e.g., EI = mean of GP - std of GP). Thus, Bayesian optimization can save functional evaluation times. Typically, Bayesian optimization is employed in situations where observing the function value $y_x$ at a specific $x$ is time-consuming or expensive. However, I am confused here. Does $y_x$ represent the observed function value at $x$ according to equation (1)? Does this mean that in order to evaluate equation (9), we actually need to simulate or obtain $y_x$ at a specific $x$? In other words, does the proposed acquisition function (9) require the observation of $y_x$? If so, it seems to me that the method is not valid, as it sacrifices the fundamental benefit of Bayesian optimization. 6. Continuing from the previous question, how do we evaluate the distribution value $p(y_x|\cdots)$ in Step 7? Moreover, how do we evaluate $p(y_x|D)$ in (9)? 7. What is the difference between $p(y_x|\theta,D)$ used in equation (7) and $p(f|\theta,D)$ used in Step 6? Are they different in terms of considering $\epsilon$ or not (i.e., in equation (1), $y_x=f(x)+\epsilon$)? 8. I am confused about the condition on $(x^\star,f^\star)$. What do the authors mean by $(x^\star,f^\star)$? Are they the true optimals related to the underlying function $f$ or are they the optimals related to a GP model? 9. The considered numerical experiments all seem rather low-dimensional. 10. I am concerned about the insufficient technical contribution. It appears to me the main contribution could be summarized in one equation, Eq. (7). Previous similar papers usually also provided theoretical guarantees (e.g., the JES paper). Solely a new function form seems rather insufficient as a standalone paper. I would appreciate it if the author could justify their contribution. **Writing suggestions** 1. I assume "HP" in Figure 1 means "hyper-parameter." However, at first sight, "HP" is confusing, especially since many words have abbreviations as "HP." I suggest that the authors write the full name when it first appears. 2. Line 72 contains a typo: "$\epsilon\sim N(0,\sigma^2_\epsilon)$" instead of "$\epsilon^2$." 3. Keep the symbol format consistent: In line 76, $\theta$ is not bold, while in line 80, $\theta$ is bold. 4. The Hellinger distance has the general form shown in equation (4). Why is there no general form provided for the Wasserstein distance? 5. In line 221, "robust. than" needs to be revised for clarity.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
The authors provided limitations in the last section.
Summary
The paper introduces a technique called statistical distance-based active learning (SAL) for the purpose of learning the hyperparameters of a Gaussian process. Additionally, SAL is integrated with information-theoretic Bayesian optimization (BO) in a framework referred to as self-correcting BO (SCoreBO). This integrated approach enables simultaneous learning of GP hyperparameters and optimization of the black box function. The experimental results demonstrate the enhanced performance of SCoreBO compared to traditional benchmarks, including unconventional BO tasks.
Strengths
The paper introduces new active learning algorithms that leverage the Query-by-Committee strategy and utilize two statistical distances. It also addresses the practical challenge of simultaneously learning hyperparameters while optimizing a black-box function, which holds significant importance in real-world applications.
Weaknesses
The main weakness of the paper is inadequate literature review, resulting in major concerns on the novelty of the proposed problem and solution and the experimental results. 1. Although BALD is widely recognized as a highly effective active learning strategy (e.g., employed in PES, MES, JES), it is surprising that BALD is not considered as a baseline in the active learning experiments. Conversely, the other baselines employed in the active learning experiments are relatively weak, such as utilizing only the posterior variance (BALM), the posterior mean (BQBC), or simply adding BALM and BQBC. 2. Equation (7) (the proposed SAL) is essentially the Jensen-Shannon divergence, representing the mutual information between $\theta$ and $y_x$—in other words, BALD when $d$ corresponds to the KL divergence. Surprisingly, this crucial aspect is not discussed in the paper, raising the question of why the preference is given to the Hellinger distance or Wasserstein distance instead of the KL divergence. 3. The paper introduces the problem of optimizing a black box function with unknown Gaussian Process (GP) hyperparameters, asserting that this aspect has received limited attention, without providing references to existing Bayesian Optimization (BO) works that address unknown GP hyperparameters. However, it is worth noting that in the early work PES, there exists a dedicated section that tackles the issue of unknown GP hyperparameters by learning the posterior distribution of the hyperparameters and averaging PES over samples of GP hyperparameters. This crucial detail is missing from the paper, despite frequent mentions of PES throughout. Furthermore, this implies that other information-theoretic acquisition functions such as MES and JES could also be extended to handle unknown GP hyperparameters using a similar approach. Consequently, the novelty in the problem appears to be overstated and a significant concern arises regarding the absence of several essential baselines in the experiments, including PES, MES, and JES averaged over samples of GP hyperparameters. 4. The motivation behind the proposed SCoreBO in Equation (8) needs further enhancement. Given that the goal of Bayesian Optimization (BO) is to optimize a black-box function, it seems appropriate to exclude $\theta$ from the second argument in $d$ in Equation (8). This is because it is redundant to learn the hyperparameters of the Gaussian Process (GP) if they do not impact the difference between $f^*$ or $x^*$ under different GP hyperparameter samples. Essentially, the acquisition function should be a JES, with the GP hyperparameters marginalized out. To substantiate this approach, it would be valuable to present compelling experimental results comparing it with such a JES that marginalizes out the GP hyperparameters.
Questions
Please clarify the above weaknesses. ======================== After reading author's response The authors have made substantial efforts in providing supplementary experimental outcomes and elaborating the reasoning behind the integration of hyperparameters into the acquisition function. I believe that these experiments and explanations have the potential to provide stronger motivation for the paper and make the proposed approach's performance more convincing. As a result, I improve my rating accordingly.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
The authors have adequately addressed the limitations.
Summary
The paper presents two algorithms/methods, SAL and SCoreBO, for active learning and Bayesian optimization using GPs. Both build on the premise that hyperparameters are critical to effective AL or BO; hence it is important to learn both the function and the hyperparameters through an acquisition function that accounts for both needs. Contributions: - The SAL acquisitions function for AL considers the statistical distance between a condition (on the hyperparameters) and the marginal posterior. - The SCoreBO algorithm for BO, which uses SAL in combination with Thompson sampling/posterior sampling - Empirical evaluation on several relevant benchmark problems - Comparison with a set of alternative methods
Strengths
- Well-written and clear (with only a few unclear aspects) - Relatively simple but effective approach (based on various other works in fully Bayesian BO including [42]) - Thorough evaluation of a suitable set of both AL and BO benchmarks, including non-standard problems - Comparison with what appears to be a sensible set of baselines, showing favorable performance on the BO tasks
Weaknesses
The following are mostly just questions and comments, not all weaknesses per se: - Lack of details regarding a fully Bayesian treatment - The paper mentions that it applies a fully Bayesian treatment, yet I am (as a reader) still a bit unsure about the specific inference being used. I think more details are needed here to make it precise. In particular, discussing $p(\theta | D)$ would be helpful to ensure the paper presents a self-contained view. - Justification of distance metrics and alternatives - I feel the paper lacks justification for the choice of distance measures. I appreciate a choice needs to be made at some point, but alternative statistical distance measures also have closed-form, e.g., (symmetric) KL, and even approximations for mixtures. - I am slightly concerned about the sensitivity to the distance function for the AL (and BO as shown in the appendix). For the AL task, have the authors performed longer experiments than presented in Figure 6 (i.e. why stop at 100/150/200 iterations)? - What's the complexity? It would be helpful with an indication/summary of the complexity of SCoreBO relative to a standard fully Bayesian BO approach. - (The need for) theoretical bounds: What is the prospect of providing theoretical bounds for the specific algorithm? Minor: - As a sanity check, have the authors compared SCoreBO with a box standard BO approach without a fully Bayesian treatment of hyperparameters (i.e., ML-II)? - Figure 1: For clarity, I'd suggest specifying what a "BoTorch prior" is - Figure 7: There are two dashed lines; perhaps clarify the caption.
Questions
Included in the above.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Included in the above.
Summary
Standard acquisition functions in Bayesian Optimization (BO) only aim to finding the optimum, but do not directly consider the problem of hyperparameter learning in Gaussian Processes (GP) which has considerable impact on the optimization performance. This paper introduces a new acquisition function for Active Learning (AL) and Bayesian Optimization (BO), whose goal is to learn both hyperparameters and the location of the optimum. Specifically, the acquisition function for AL is based on extending similar previous proposals of Active Learning via disagreement, by using different statistical distances. This new acquisition function is then adapted for the BO task by conditioning on sampled locations/value of the optimum. The new acquisition functions are shown to work (slightly) better at AL and BO, especially with unusual BO tasks.
Strengths
- **Originality:** While natural, the problem of hyperparameter learning in BO has rarely been addressed, so this paper tackles an open and underestimated problem in the field. The proposed solutions are interesting. - **Significance:** Given the prominence of BO in machine learning and other fields, this work is potentially very significant. - **Quality:** The general quality of the work is good, although there are some open questions (see below). - **Clarity:** The paper is generally clear. As general comments, Statistical distance-based Active Learning (SAL) is well-motivated and explained, a nice generalization of previous proposals. It is also nice that the paper explores the use of two different distances (Wasserstein and Hellinger), and that it gives approximations on how to compute them. The proposed "motivated heuristic" for extending SAL to BO via conditioning over the location-value of the optimum is interesting. ### Post-rebuttal Thanks to the authors for addressing the points I raised. I am glad that they managed to run additional experiments with a more realistic benchmark, showcasing the effectiveness of their method. Overall, I am satisfied with the paper and the score of 6 reflects my current evaluation of the paper ("Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.").
Weaknesses
- The paper showcases many synthetic functions but there are only a couple of real-world applications. Real-world applications are particularly important for this work because they are exactly the situations at risk of breaking active learning (e.g., due to model misspecification). Indeed, the Cosmological Constant task in the paper is the one in which ScoreBO does not show any advantage over vanilla Expected Improvement. For this reason, it would have been nice to show other real examples in which ScoreBO does improve performance. - As a minor comment, please double-check the paper and bibliography for typos or errors. At a quick glance I spotted a few mistakes (e.g., the authors of reference [43] "Fast information-theoretic Bayesian optimisation" are in a wrong order; occasionally "Bayesian" appears lowercase, etc.).
Questions
- This paper would strongly benefit from showing ScoreBO at work on other real-world tasks, to showcase that indeed this new acquisition function and its focus on active learning does not harm when deployed on real problems (e.g., RL, hyperparameter tuning, and other typical applications of BO which are conspicuously missing here). - Minor: Fix the few typos in the paper and bibliography.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
The conclusion quickly states a few limitations but it'd be good to have a separate **Limitations** section which very explicitly mentions them.
Response to the rebuttal
Thanks to the authors for addressing the points I raised. I am glad that they managed to run additional experiments with a more realistic benchmark, showcasing the effectiveness of their method. Overall, I am satisfied with the paper. In the discussion with the other reviewers, I will argue for acceptance.
Additional reponse to ta8V
We are pleased that the reviewer valued our additional experiments and thankful for their willingness to support our paper in discussions with other reviewers. In light of the reviewer’s updated judgement, we would very much appreciate if they also considered updating their score on OpenReview.
Thanks to the authors for clarifications and additional experimental results. However, my concerns remain: 1. I believe it is important for the paper to discuss existing BO works that handle the unknown hyperparameters issue such as PES. Considering that the issue of unknown hyperparameter is not new, the novelty of the problem seems to be overstated in the paper. 2. Section 2.3 in PES is not exactly about a fully Bayesian treatment of the hyperparameters. It proposes to average the PES acquisition function over $M$ samples of the hyperparameters (see Equation 10). This is different from a fully Bayesian hyperparameter treatment which marginalizes $p(y|x) \approx 1/M \sum_{i=1}^M p(y|x,\phi_i)$. While this may not be a "correct" way, the paper has shown superior empirical performance. Hence, I do not understand why the experiments did not incorporate this PES baseline. 3. I am still confused why JS divergence is not equivalent to BALD. Let say the posterior distribution of the hyperparameters is approximated with $M$ samples of the hyperparameters, then BALD, i.e., the mutual information between $y$ and the hyperparameters $\phi$ is $I(y;\phi|x) \approx JSD(p(y|x,\phi_1), p(y|x,\phi_2), \dots, p(y|x,\phi_M)) = 1/M \sum_{i=1}^M D(p(y|x, \phi_i) || p(y|x))$ where $D$ is the KL divergence. Equation (7) in the paper is this formulation but replacing the KL divergence with other divergences. As the authors demonstrate in their response, the competitive performance of BALD reduces the need of shifting from using the KL divergence to the suggested divergences. 4. I have a different perspective regarding the authors' assertion that: having an accurate hyperparameters is crucial for the efficiency of the BO algorithms. This is due to the fact that obtaining precise hyperparameters requires utilizing samples. If the core objective of BO is to locate maximizers (not to learn the hyperparameters), then what justifies the explicit incorporation of hyperparameters into the acquisition function? I believe we should only reduce the uncertainty in hyperparameters that specifically contributes to uncertainty in the maximizer (which is achieved by a fully Bayesian treatment of PES, for example). Therefore, incorporating the hyperparameters directly into the acquisition function doesn't appear to be particularly persuasive. I understand that there may be an empirical performance gain as the fully Bayesian treatment of the hyperparamters is not exact (approximated with MCMC). As a result, from my personal view, the inclusion of hyperparameters in the acquisition function doesn't seem to be a substantial contribution.
Further comments on marginalization
We'd like to express our sincere appreciation to the reviewer for their consistent engagement throughout the review process and for taking the time to participate in this rebuttal phase. The authors are committed to addressing the concerns raised in the ensuing discussion and especially to clarify the misunderstanding about the marginalization of the hyperparameters. ____ __Point 2:__ We now appreciate the misunderstanding of the terminology, which is one that is deeply seeded in the BO community. The reviewer makes a distinction between 1. Marginalizing the __acquisition function__ over the hyperparameter uncertainty, $E_\theta[(\alpha(y)|x,\theta)]$, and 2. Marginalizing the __model__ over the hyperparameter uncertainty, $\alpha(E_\theta[p(y|x,\theta)])$. In the ML literature, it'd be natural to call 2) fully Bayesian treatment, which is what the reviewer does. However, the BO community has been referring to a fully Bayesian treatment as directly marginalizing the acquisition function over the hyperparameters, which is 1). This definition of fully Bayesian treatment in BO was given by Snoek et al. (2012) and used even earlier by Osborne et al. (2010). It has subsequently been used by PES and a number of other BO methods, including MES and JES. From a general ML perspective this may be confusing because the PES authors refer to 1) as a fully Bayesian treatment (see Section 2.3. of their paper). SCoreBO uses 1) as well. To summarize, both PES and Snoek refer to the same definition 1) of fully Bayesian treatment, which marginalizes the acquisition function over the hyperparameters. That is the same definition that we use in all the baselines of the paper. This could be misleading and we will make this difference explicit in the CR. As a result, since PES performs the same operations to the hyperparameters as other baselines, one would expect that PES should not perform better than the other methods. We have run this version of PES, and added a table of the hyperparameter values for all methods (and reference values) for PES, JES, and SCoreBO on the 25D Ackley task. It is attached as the last comment in this post. We will add PES on all tasks for the CR; it has been run and performs marginally worse than MES on average. Its final performance is added for all tasks as a separate post. ____ __Point 1:__ We agree with the reviewer that the topic of handling unknown hyperparameters for BO is not new — All optimizers must handle unknown hyperparameters in some way, whether through MLE, MAP, fully Bayesian treatment or adapting the acquisition function to actively manage uncertainty across hyperparameters. Since the reviewer is familiar with BALD and PES, we would like to highlight the differences between them to align with the reviewer: - PES reduces the uncertainty over the optimum, - BALD reduces the uncertainty over the hyperparameters, and - SCoreBO reduces uncertainty over optimum **and** hyperparameters. The authors would like to emphasize that a Bayesian treatment of the hyperparameters does not entail reducing hyperparameter uncertainty, i.e., a Bayesian treatment “acknowledges” parameters uncertainty, but doesn’t reduce it. It can still suffer from having large variance in the parameter distribution — That is a crucial difference that makes SCoreBO outperform other acquisition functions on the SAASBO experiments. Furthermore, the Bayesian treatment of the hyperparameters in PES is based on Snoek et. al. (2012) and it is not a contribution of PES. PES is one of the many approaches that use marginalization of the hyperparameters in this way (MES and JES also did that, following Snoek et al and PES). We use the same marginalization over the hyperparameters used in PES, MES and JES. Additionally, we cite and compare against all these baselines. We thus disagree with the statement that we do not discuss existing BO works that handle unknown hyperparameters. The main contribution of SCoreBO is to combine the reduction over the uncertainty of the location of the optimum, such as in PES, and the reduction of the uncertainty over the hyperparameters. This __joint__ uncertainty reduction is novel. We are committed to improving the introduction of the paper in the camera ready, to make sure that our contribution is not confused with general handling of unknown hyperparameters as the reviewer suggests.
Further comments on BALD equivalence
__Point 3:__ The reviewer is correct in their assessment, and we are __greatly thankful__ for their persistence! We note that the equality does not only hold in the finite-sample MC approximation that the reviewer put forward, but also in the general case, i.e. the JSD for an infinite number of distributions. This was an oversight on our end, and we believe it provides another nice link to existing methods, such as BQBC. We will happily include this relation, and the short proof, in the CR. With this help by the reviewer, we believe that SAL becomes a convenient umbrella term for multiple existing methods. Naturally, this means that there is a corresponding SCoreBO variant using KL-divergence/BALD, which we are happy to include as well. We will acknowledge the reviewer's contribution in formalizing the explanation of the methods in the acknowledgement section of the CR. We will also run the reverse SAL-KL (since BALD is already included) and provide the results. The proof for the aforementioned equivalence is provided in the end of our post. ____ __Point 4:__ The reviewer raises a valid point regarding the motivation behind the formulation of SCoreBO in Equation (8). While the aim of Bayesian Optimization (BO) is indeed to efficiently optimize a black-box function, we would like to emphasize that accurate hyperparameter estimation can significantly impact statistical efficiency. The experiments reveal that in scenarios with inaccurate hyperparameter distribution, optimizing the objective directly can be inefficient (please refer to the fully Bayesian treatment & BoTorch prior in Fig. 1 of the paper). In such cases, the lack of a precise model of the response surface hinders effective BO. By dedicating iterations to better understand the problem, even though it initially incurs a cost, we observe improved optimization performance later on. It's important to note that we do not claim that this trade-off always guarantees to be beneficial. Our intention is to demonstrate instances where it proves advantageous and a few instances where it doesn't. For instance, in Fig. 1 of the rebuttal PDF, we highlight a case where the posterior distribution is notably imprecise, rendering the sampled points non-representative of the underlying function, as the posterior distribution at large suggests that all dimensions are unimportant. Prior to the introduction of SCoreBO, the ability to deliberately allocate attention to active hyperparameter learning for enhanced optimization performance was lacking. This underscores a significant contribution of our paper. ____ We remain open to any additional questions, suggestions, or concerns that may arise as we proceed in this interactive dialogue. Your comments have been immensely helpful in guiding our responses and enhancing the clarity of our contributions. We look forward to further productive exchanges.
FB-PES & BALD Hyperparameter convergence
The below table shows the (log) hyperparameter values (mean and standard deviation across the samples in the FB treatment) for all methods after 100 iterations of 25-dimensional Ackley with 4 active dimensions, across 10 repetitions. $\ell_i$ close to -1 suggests that the dimension $i$ is active, and very important to optimization performance. This is true for $\ell_1, \ldots, \ell_4$. The inactive dimensions $\ell_5, \ell_6$ should have a confident and large value. The true noise $\sigma_\epsilon^2$ is $-1$. Fully Bayesian PES is and displays inaccurate and inconfident values for all parameters, as seen previously with MES, JES and EI. BALD (previously not in plot) produces accurate values since it actively learns the hyperparameters. However, __SCoreBO is even more accurate, and confident, on all parameters__ despite doing optimization at the same time. PES subsequently produces similar to JES and MES on the task, since it does not know which dimensions to explore. | | JES | MES | ScoreBO | PES | BALD | |:--------------------|:------------|:-----------|:---------------|:-----------|:-----------| | $\sigma_\epsilon^2$ | -1.73±0.73 | -2.37±0.94 | -0.99±0.28 | -1.97±0.72 | -1.32±0.42 | | $\ell_1$ | 0.11±0.92 | 0.7±0.84 | -0.89±0.1 | 0.46±1.42 | -0.68±0.35 | | $\ell_2$ | 0.24±1.02 | 0.46±1.04 | -0.89±0.1 | 0.86±1.15 | -0.75±0.49 | | $\ell_3$ | -0.02±1.29 | 0.45±1.15 | -0.89±0.1 | 0.45±1.32 | -0.71±0.29 | | $\ell_4$ | 0.66±0.78 | 0.68±0.9 | -0.88±0.12 | 1.05±0.93 | -0.68±0.32 | | $\ell_5$ | 0.8±0.64 | 1.0±0.71 | 1.01±0.3 | 1.28±0.62 | 0.74±0.69 | | $\ell_6$ | 0.7±0.82 | 0.5±1.06 | 1.01±0.31 | 0.97±1.15 | 0.69±0.69 |
Synthetic BO & DL performance
### Novel results for fully Bayesian PES only Log regret on Synthetic BO tasks (mean and standard error) including PES and previous methods. PES performs worst on three tasks and best on one. SCoreBO wins on four tasks. All methods employ fully Bayesian acquiisition function treatment (i.e. Section 2.3 in PES) | | Branin | Hartmann (3D) | Hartmann (4D) | Hartmann (6D) | Rosenbrock (2D) | Rosenbrock (4D) | |:-------------|:-----------|:------------|:------------|:------------|:--------------|:--------------| | NEI | -1.64±0.06 | -1.4±0.05 | -1.07±0.06 | -0.6±0.06 | -0.92±0.07 | -0.22±0.05 | | JES | -1.84±0.08 | __-1.62±0.06__ | -1.26±0.07 | -0.88±0.05 | -1.07±0.08 | -0.29±0.05 | | MES | -1.89±0.09 | -1.58±0.06 | -1.03±0.07 | -0.78±0.08 | -1.0±0.07 | -0.34±0.06 | | ScoreBO | __-2.11±0.08__ | -1.56±0.06 | __-1.34±0.07__ | __-1.06±0.05__ | __-1.23±0.08__ | -0.16±0.06 | | PES | -1.9±0.09 | -1.24±0.07 | -0.94±0.06 | -0.47±0.08 | -1.0±0.09 | __-0.44±0.06__ | Performance on deep learning HPO tasks (mean and standard error) including PES and previous methods. fully Bayesian PES (i.e. Section 2.3 execution) performs approximately on par with fully Bayesian MES as the second-worst or worst method. | | WMT | CIFAR | LM1B | |:-------------|:-----------|:------------|:-----------| | NEI | 29.1±0.16 | 18.45±0.12 | __60.32±0.14__ | | JES | 29.04±0.14 | 17.82±0.17 | 60.53±0.13 | | MES | 29.63±0.16 | 18.57±0.11 | 60.57±0.13 | | ScoreBO | __28.82±0.12__ | __17.77±0.14__ | 60.42±0.08 | | PES | 29.21±0.16 | 18.01±0.1 | 60.57±0.09 |
While my concern regarding the novelty has not been entirely addressed, I deeply value the authors' efforts to conduct additional experiments and address my inquiries. I hope that the authors will share the code to aid other researchers working on similar subject. I have improved my score accordingly.
Thanks
We greatly appreciate the reviewer's efforts, as they have undeniably improved the quality of our paper through the SAL-KL discussion and beyond. Thank you for your remark about sharing code to aid other researchers working on similar subjects. We fully agree that this is very important to the community. While the program chairs have instructed authors to not post URLs during the rebuttal period, we invite the reviewer to check the URL for our public anonymous repository linked in our original paper on page 7, Lines 226-227, which includes complete code for our method and all baselines in BoTorch (recently updated to include all the experiments and methods added during the rebuttal), as well as thorough instructions and scripts for all experiments in the paper, which produce all the required metrics (MLL, observed values, inference values, hyperparameter sets) to reproduce our work. If the reviewer has any additional comments or questions following our most recent set of responses, we would be keen to address them with the remaining time until the end of the rebuttal phase.
Thanks to the authors for addressing my questions, in particular on the details of the fully Bayesian treatment and choice of divergence. I note these aspects have been discussed further and in-depth with reviewer LR3N, including explicit links to BALD, etc. I welcome the insights gained from the discussion with LR3N (and other comments). While I still feel the paper addresses a relevant and interesting aspect of BO, the discussion reveals that a more complete and coherent narrative is required to present the method/results than the case in the submitted paper. The many required/suggested changes lead to some doubt about what the final paper will look like, and I usually lean towards recommending the paper go through a full review cycle in such cases. I’d encourage the authors to summarize their proposed changes in a single place, before the end of the discussion phase on the 21st.
Thanks for the clarfications
I would like to express my sincere appreciation to the authors for taking their time to address my previous concerns and clarifying my misunderstandings regarding this paper. I have several followup questions. 1) Could the authors provide clarifications on the highest dimension of the experiments discussed in the paper? 2) With regards to the JES paper authored by Ben Tu et al., I understand that the provided proofs are not intended to establish a definitive convergence rate. But it is good to know some theoretical properties of JES, so is for this paper. 3) Could the authors give intuitive reasons why the proposed method works, and performs better than the other methods?
Further response
Thanks to the reviewer for their continued engagement. We hope to clarify the unique strengths of our proposed method and clarify the additional questions below: --- > __1.__ Could the authors provide clarifications on the highest dimension of the experiments discussed in the paper? --- The highest dimensional benchmark in the experiments is the Lasso-DNA real-world task with 180 variables. The problem constitutes finding the hyperparameters of a weighted Lasso model for a microbiology classification task (Mills 2020). This benchmark has been considered in several papers for high-dimensional hyperparameter optimization (Šehić et al., 2022, Papenmeier et., al. 2022, Ziomek and Bou-Ammar, 2023). As traditional BO methods are inefficient at such high dimensions, we adopt the Sparse Axis-Aligned Subspace hyperparameter prior (SAAS, Eriksson and Jankowiak, 2021) which encourages the model to disregard inactive dimensions. This approach was used for Lasso-DNA in Papenmeier et. al. (2022). SCoreBO is able to infer the effective subspace under the SAASBO prior within tens of iterations by actively learning the model while optimizing, while conventional acquisition functions fail to infer the effective subspace at all. As such, SCoreBO achieves a 25% performance increase over JES and EI, relative to the initial design configurations in Fig. 7. --- > __2.__ With regards to the JES paper authored by Ben Tu et al., I understand that the provided proofs are not intended to establish a definitive convergence rate. But it is good to know some theoretical properties of JES, so is for this paper. --- We agree with the reviewer that a deeper theoretical contribution in this area would be very valuable, as the broader set of BO acquisition functions with fully Bayesian treatment of the hyperparameters lack such insights. SAL was developed with BQBC in mind, but generalizes BQBC to allow a broader class of disagreement metrics than just the difference in mean. As such, SAL encompasses a broad class of possible AL methods. A theoretical connection with existing methods came through the discussion with reviewer LR3N: SAL with KL divergence used as a distance measure is equivalent to the popular BALD active learning method. We theoretically prove this connection in our discussions with reviewer LR3N.
Further response, part 2
--- > __3.__ Could the authors give intuitive reasons why the proposed method works, and performs better than the other methods? --- The efficiency of surrogate model-assisted optimization is often highly dependent on that the model yields accurate predictions, which in turn relies on well-specified hyperparameters for the task at hand. The hyperparameters are estimated from the observed data, but __no previous BO method actively selects data to quickly adapt the model to the problem__. Instead, conventional BO methods (EI, UCB, JES) must hope that the selected points happen to be sufficient to learn a good model. This is where SCoreBO differs from previous approaches, and the reason why SCoreBO outperforms previous methods in the experiments. By selecting points where the model disagrees the most, not only do we quickly learn the hyperparameters, but we __focus on learning the hyperparameters that most strongly impact the model prediction__. By combining this with optimizing the objective, SCoreBO presents a reliable acquisition function that learns the model early and, subsequently, __uses this more accurate model__ to efficiently find good solutions. In the high-dimensional case of Ackley (rebuttal PDF Fig. 1, response to LR3N), SCoreBO identifies the most important dimensions as fast as the pure active learning method BALD, which effectively turns the 25D optimization problem into a 4D one using only tens of iterations. Conventional BO acquisition functions fail on this high-dimensional task, as they do not quantify utility based on any notion of uncertainty in the surrogate. From the hyperparameters in the rebuttal PDF in Fig. 1, the reviewer can see that JES & MES seemingly randomly suggest dimensions as being important. The result is persistent for Hartmann and Lasso-DNA, which shows the potential impact of a joint objective. A counterexample is for the Rosenbrock (4D) synthetic function (Appendix C.3), where SCoreBO performs worse relative to other acquisition functions because the model is already accurate enough. On Rosenbrock, the hyperparameter values increase over time instead of converge, which suggests that the latent function is not part of the model class. Thus, the self-correction effort of SCoreBO is not helpful towards efficient optimization in that case. SCoreBO is __the first acquisition function to have a joint optimization/active learning objective__, which ultimately makes it very distinct to JES, or any current BO acquisition function. We believe it has an important role to fill in practical applications with complex models, which we have hopefully exemplified both with the SAASBO and HEBO experiments, as well as a theoretical concept to inspire new methods focusing on better learning the surrogate model.
Thanks for the clarification
I appreciate the authors' clarifications, and vote for acceptance.
Summary of changes
We thank the reviewers for their further engagement. We acknowledge that additional aspects of the SAL/SCoreBO have been brought to light during the rebuttal, and the comments regarding BALD from LR3N have been very helpful to the quality of our work. To summarize, the following is currently being worked into the camera ready version of our paper: 1. __SAL-KL/BALD equivalence:__ An expanded discussion on BALD in the background. The definition of SAL-KL and the proof of its equivalence to BALD, stated in Section 3.1 and proved in the Appendix. (Presented in discussion with __LR3N__, SCoreBO-KL in a subsequent post). 2. __Intuition for distance metrics:__ Clarification of the benefits and drawbacks of various distance metrics, including plots to match the intuition in the Appendix (__fp2C__). 3. __Deep learning benchmarks:__ Inclusion of the tasks, employing the warpings from HEBO, from the rebuttal PDF as an additional Section 4.4. (Rebuttal PDF, Fig 4.) 4. __Details regarding fully Bayesian treatment:__ A paragraph in Section 2 outlining it generally and its use in acquisition functions specifically. (__LR3N__). An extended explanation of the sampling procedure for the fully Bayesian treatment in the Appendix and a shorter description in Section 3.3. (__fp2C__). 5. __Complexity of method:__ A description of the complexity of SCoreBO and fully Bayesian treatment in Section 3.2 as compared to other methods (__fp2C__). 6. __Explicit limitations section:__ A new Section 5.1 outlining the pitfalls of using SCoreBO for misspecified models, and its dependence on fully Bayesian treatment of the hyperparameters (__ta8V__). Minor (non-structural): - Fix references and typos (__ta8V__, __BCnP__). - Include EI-MAP (fp2C) and fully Bayesian PES (__LR3N__). - Show prolonged AL experiments (__fp2C__). - Specifying BoTorch prior (currently in Appendix 1, __fp2C__). - Clarify meaning of dashed lines (__fp2C__). We note that almost all experiments, figures and commentary _can already be found in this rebuttal_. As such, we hope that the reviewers feel confident that the CR will be representative of the material already presented, and result in an even higher-quality paper. We are currently working on the CR and are happy to provide additional details on any of these bullets if the reviewer desires.
SCoreBO-KL
To complete the set of experiments, we have now added SCoreBO-KL to the synthetic benchmarks. SCoreBO-KL displays very competitive performance, ranking as the second best method overall (after SCoreBO-Hellinger = SCoreBO-HR), achieving the best regret on one task and the second best on an additional three. Best __log regret__ on each task is bolded and cursive, second best is bolded only. | | Branin | Hartmann (3D) | Hartmann (4D) | Hartmann (6D) | Rosenbrock (2D) | Rosenbrock (4D) | |:-------------|:-----------|:------------|:------------|:------------|:--------------|:--------------| | NEI | -1.64±0.06 | -1.4±0.05 | -1.07±0.06 | -0.6±0.06 | -0.92±0.07 | -0.22±0.05 | | JES | -1.84±0.08 | __-1.62±0.06__ | __-1.26±0.07__ | -0.88±0.05 | __-1.07±0.08__ | -0.29±0.05 | | MES | -1.89±0.09 | -1.58±0.06 | -1.03±0.07 | -0.78±0.08 | -1.0±0.07 | ___-0.34±0.06___ | | ScoreBO-KL | __-2.05±0.1__ | ___-1.68±0.06___ | -1.08±0.07 | __-0.92±0.06__ | -0.98±0.08 | __-0.32±0.06__ | | ScoreBO-HR | ___-2.11±0.08___ | -1.56±0.06 | ___-1.34±0.07___ | ___-1.06±0.05___ | ___-1.23±0.08___ | -0.16±0.06 |
All intended changes
We thank the reviewers for their engagement — This message is a follow-up to the camera ready summary (the message to all reviewers on August 19). The aspects that have been brought to light during the rebuttal are very helpful to the quality of this work and we are grateful to the reviewers. We fully agree with reviewer fp2C that it’s helpful to see the intended revisions in order to make a confident assessment, and while we‘re not allowed to upload the final revised paper we would like to do the next best thing and describe the changes related to the enumerated updates in detail. We also make explicit reference to the line in the current OpenReview PDF where we will add this text. ### 1. SAL-KL/BALD equivalence: __Line 103:__ Bayesian Active Learning by Disagreement (BALD) [21] was among the first Bayesian active learning approaches to explicitly focus on learning the model hyperparameters: $$\alpha_{BALD} (x) = I(y_x; \theta) = H(p(y_x)) - E_{\theta} [H(p(y_x | \theta))]$$ __Line 147:__ Notably, SAL generalizes both BQBC and BALD, which are exactly recovered by choosing the metric $d$ to the difference in mean or the forward KL divergence, respectively. __Proposition 1:__ $E_\theta [KL(p(y_x|\theta, {D}) || p(y_x|{D}))] = I(y_x; \theta |D).$ The proof is shown in Appendix E. __Appendix E - SAL-KL equivalence with BALD:__ We show that SAL, using the KL divergence as the metric $d$, is equivalent to BALD. _Proof_: $$ BALD = I(y_x,\theta) = H(p(y_x)) - E_{\theta}[H(p(y_x|\theta))]= $$ $$ -\int_{-\infty}^{\infty}\int_{\theta} p(\theta)p(y_x|\theta)~log[p(y_x)] d\theta dy_x + \int_{\theta} p(\theta)\int_{-\infty}^{\infty} p(y_x|\theta) log [p(y_x|\theta)]dy_x d\theta = $$ $$ \int_{\theta} p(\theta)\int_{-\infty}^{\infty} p(y_x|\theta) log\left[\frac{p(y_x|\theta)}{p(y_x)}\right]dy_x d\theta = \int_{\theta} p(\theta) KL(p(y_x|\theta) || p(y_x)) d\theta $$ $$ = E_{\theta}[KL(p(y_x|\theta)||p(y_x))] = SAL.KL $$ ### 2. Intuition for distance metrics: __Line 117:__ We focus on three metrics which have closed forms for Gaussian random variables. __Line 119:__ The KL divergence is a standard asymmetrical measure for dissimilarity between probability distributions. For two probability distributions $P$ and $Q$, it is given by $$ D_{KL} (P || Q)= \int_{X}P(x)log(P(x)/Q(x))dx. $$ The KL divergence mainly prioritizes same order-of-magnitude variances, and will initially query the same location multiple times to assess noise levels. Thereafter, it tends to query in an axis-aligned fashion, close to previous queries, to attain information regarding the length scales, but places a low priority on global exploration. __Line 124:__ The Hellinger distance seeks to minimize the ratio between difference in mean and the sum of variances, which punishes outlier predictive distributions of high confidence. Similar to KL, initial queries have a tendency to be axis-aligned to attain selective length scale information. __Line 127:__ The Wasserstein distance, also commonly referred to as the earth-mover distance, seeks to minimize the difference in the first and second moment. In practice, this places a premium on matching large-variance regions, leading to higher global exploration which can be detrimental for global optimization. ### 3. Deep learning benchmarks: __Page 8:__ We will add a new figure with the new DL results as seen in the rebuttal PDF. We will tentatively call the new figure Figure 10. __Line 285:__ We show the performance of SCoreBO on three (4D) hyperparameter tuning tasks from the PD1 benchmarking suite [Wang et. al., 2023], Fig. 10 — Two involving large language models, and one from computer vision. We adopt the same application setting as in HEBO [Cowen-Rivers et. al. 2020], where input and output warpings [Snoek et, al 2014] are used to account for the heteroskedasticity prevalent in real HPO tasks. We observe that SCoreBO outperforms the other methods on 2 out of 3 tasks, placing second on the third.
All intended changes, part 2
### 4. Details regarding fully Bayesian treatment __Line 80:__ The posterior probability of observing a value $y_x$ for a point $x$ is given as [Lalchand \& Rasmussen, 2020]: $$ p(y_x|D) = \int_{\theta} \int_{f} p(y_x|f, \theta)p(f| \theta, D) p(\theta| {D})df d\theta $$ where ${D}$ is the observed data. The inner integral is equal to the GP predictive posterior, $$ \int_{f} p(y_x|f, \theta)p(f|\theta, {D})df = p(y_x|{D}, \theta). $$ However, the outer integral is intractable and is estimated using Markov Chain Monte Carlo (MCMC) methods. The resulting posterior prediction $$ p(y_x|{D}) = \int_{\theta}p(y_x|{D}, \theta)p(\theta|{D})d\theta\approx \frac{1}{M}\sum_{j=1}^M p(y_x|{D}, \theta_j), \theta_j \sim p(\theta_j|{D}), $$ is a Gaussian Mixture Model (GMM). Within BAL and BO, the fully Bayesian treatment is often extended to involve the acquisition function, such that the acquisition function $\alpha$ is computed as an expectation over the hyperparameters [Osborne 2010, Snoek et. al., 2012] $$ \alpha(x|\mathcal{D}) = E_\theta[\alpha(x|\theta, \mathcal{D})] \approx \frac{1}{M}\sum_{j=1}^M \alpha(x|\theta, \mathcal{D}), \theta{}_j \sim p(\theta{}|\mathcal{D}). $$ This is also the definition of fully Bayesian treatment considered in this work. ### 5. Complexity of method __Line 193:__ Add a new paragraph inside the SCoreBO Sec. 3.2: The conditioning on the fantasized data point involves a rank-1 update of $\mathcal{O}(n^2)$ of the GP for each draw. As such, the complexity of constructing the acquisition functions is $\mathcal{O}(MNn^2)$ for $M$ models, $N$ optima per model and $n$ data points. We utilize NUTS [Hoffman & Gelman, 2014] for the MCMC involved with the fully Bayesian treatment, at a cost of $\mathcal{O}(Dn^3)$ per sample. ### 6. Explicit limitations section __Line 321:__ _Replace:_ Moreover, the potential downside of self-correction is displayed when the model structure does not support the task at hand, or when self-correction is not required to solve the task. _with:_ __Limitations__ SCoreBO displays the ability to increase optimization efficiency on complex tasks that necessitate accurate modeling. However, SCoreBO’s efficiency is ultimately contingent on the intrinsic ability of the GP to model the task at hand, i.e., whether the black-box function is a draw from the class of functions defined by the chosen kernel. Appendix C3 demonstrates this issue for the Rosenbrock (4D) function, where SCoreBO performs worse relative to other acquisition functions. On Rosenbrock, the hyperparameter values increase over time instead of converge, which suggests that the latent function is not part of the model class. Thus, the self-correction effort of SCoreBO is not helpful towards efficient optimization. Moreover, increasing the model capacity, such as in Sec. 4.4, comes with increasing resources allocated towards self-correction. In highly-constrained-budget applications, such resource allocation may not yield the best result, especially if increased model complexity is unwarranted. This is evident from the synthetic AddGP tasks, where despite identifying the additive components more efficiently, SCoreBO does not provide substantial performance gains over EI. Lastly, SCoreBO’s reliance on fully Bayesian hyperparameter treatment makes it substantially more computationally demanding than MAP-based alternatives, limiting its use in high-throughput applications.
Decision
Accept (poster)