Response to Reviewer NoA7 (1/n)
We sincerely appreciate your constructive comment to improve our work.
> What does the figure specifically illustrate?
We illustrated $p(\mathbf{x} | y \leq y^\dagger, \mathcal{D})$. We would like to emphasize that we do not model $y \leq y^\dagger$ and instead it indicates some class, e.g., Class 1, which implies points that might have function values smaller than $y^\dagger$. By following the conventional expression of machine learning, the probability we illustrated is expressed as $p(\mathbf{x} | c = 1, \mathcal{D})$ where the probability of interest is the probability of $c = 1$.
> Unfortunately, the efficacy of proposed method in the performance of optimization over iterations is hardly distinguishable in Figs. 5 or 6.
We disagree with this comment. Our methods quickly converge to solutions compared to other baseline methods. In particular, **Figure 6** shows the superior performance of our methods.
> The bigger problem is that it is unclear from the text on what condition and why the proposed method outperforms the existing approaches.
Based on the use of unlabeled points and semi-supervised learning, our methods can estimate more informative probabilities, thereby easing the over-confidence problem discussed in **Figures 1 and 8** and **Section 1.1**.
> How is matrix $\mathbf{A}$ designed? What covariance is taken for this disbiribution, for what? What are lower and upper bounds $\mathbf{l}$ and $\mathbf{u}$? Are they boundaries of search space $\mathcal{X}$? Is it assumed to be rectangular?
Covariance matrix was designed as an identity matrix, and then $\mathbf{A}$ can be calculated. Since the search space of Bayesian optimization is often assumed to be a hypercube, we also used a hypercube space. Thus, $\mathbf{l}$ and $\mathbf{u}$ can be determined by the search space given. We have updated our submission accordingly.
> Does fixed-size pool scenario mean some finite number of candidate points $\mathbf{x}\_i$ are selected in advance and that optimized over these points? If true, is the set of points dynamically updated or fixed in advance?
Yes, it is optimized over a fixed number of points. The set of points is not dynamically updated by following the general formulation of pool-based optimization.
> What is the definition of simple regret? Is it $f(\mathbf{x}\_n) - f^*$ in the $n$th iteration, where $f^*$ is the minimizer of function $f$. I am wondering why the regret is monotonically decreasing in Fig. 2. Does this plot $\min\_{n} f(\mathbf{x}\_n) - f^*$?
The definition of simple regret is $\min\_{n} f(\mathbf{x}\_n) - f^*$, which is widely used in comparing Bayesian optimization algorithms. We have included this definition; please see **Equation (12)**.
> Are they all two? This is crucial information on the difficulty of black-box optimization problems.
Yes, they are two-dimensional problems. We have added the specifics of the synthetic benchmarks; see **Section D.2**.
> How do you define the distance $|\mathbf{x}\_i - \mathbf{x}\_j|$, especially when categorical values are involved such as activation function and learning rate schedule.
To handle categorical and discrete variables, we treat them as integer variables by following the previous literature (Garrido-Merchan & Hernandez-Lobato, 2020). After converting them to integer variables, the distance between two integer variables is easily defined. We have updated **Section D.3** accordingly.