End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes

Summary

The authors propose Neural Acquisition Process, a novel method for Bayesian optimization by meta-learning a function that jointly performs the surrogate and acquisition steps. The function, i.e. a transformer, is able to work as a policy that predicts the action **a** given a state **s**, where the action is the hyperparameter to observe. The state comprises the history of the observed samples. Also, the transformer can predict probabilistic the performance of a hyperparameter. They use reinforcement learning to train the policy and, additionally, they use negative log-likelihood of the predicted performance for meta-tasks as an auxiliary loss. The authors test the method on 4 different benchmarks and delineate its difference from previous work, demonstrating the relevance of their approach.

Strengths

* They present a novel and effective method for pre-training a transformer to perform Bayesian optimization. The method is well-founded and all the parts make sense from an engineering perspective. For instance, the use of auxiliary losses and reinforcement learning appears as a very important and interesting approach. * They establish a difference from prior work in terms of some parameters such as history order invariance, F values, number of tokens, etc. * The authors provide a strong experimental protocol, comparing to relevant baselines in a broad set of datasets and benchmarks, which makes the empirical results very strong.

Weaknesses

* The method might be overkill for small search spaces and nonexpensive functions. However, many black-box functions nowadays, such as deep neural networks, are expensive to evaluate, thus this approach might be highly relevant. * The authors do not present a comparison against time. How does the method perform in terms of time?

Questions

* Why is there no confidence bar for the MIP dataset?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

No important limitation is detected.

Summary

This paper proposed an end-to-end transformer-based framework for meta-Bayesian optimization (meta-BO). It formulated the meta-BO as a RL problem, defining a MDP in which a policy can be trained to solve the meta-BO problem. To help the training of RL algorithm, this paper also proposed an inductive bias via an auxiliary loss. The experiments had demonstrated the effectiveness of the proposed method.

Strengths

1. This paper deals with a good problem, which has a lot of applications in real world. The formulation of MDP is also reasonable. 2. This paper is well written, and also easy to follow. 3. The proposed inductive bias via an auxiliary loss Section 3.3. is insightful.

Weaknesses

1. The experiments is insufficient. In the current experiments, the author only provided the evaluation results of the proposed method NAP and other baselines on several benchmarks. However, it would be better to do some ablation studies to further analyze NAP. For example, it should be important to investigate the effect of the auxiliary loss Eq(3). 2. The applications are limited. RL is known to be sample inefficient, especially when the state/action spaces are of high dimensionality. Furthermore, NAP utilized a transformer to extract historic information, which is also relatively hard to train. Therefore, NAP may be implicitly restricted into situations when x is of low dimension, limiting its applications.

Questions

Please refer to weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

The authors present a hyperparameter optimization method that is based on a transformer. In contrast to other recent work on HPO with transformers, this method does not rely on an acquisition function but instead trains an end-to-end model that outputs the acquisition scores directly.

Strengths

**Diversity of benchmark tasks** Experiments on 4 different types of tasks, comparison against state-of-the-art methods. Demonstrate some improvements over baselines. **Clear Motivation** Use of transformers have shown promising results. End-to-end learning is always a good idea. The authors combine these previously existing ideas.

Weaknesses

**Complexity of the Method** The authors combine transformers, reinforcement learning and neural processes. There are most likely a lot of knobs and using this method effectively might be very difficult. Meta-BO is a prime example of a method that adds quite a bit of complexity and it turned out that it did not generalize to any other setup than considered by the authors. **Novelty** This is neither the first paper to use transfer learning, transformers, RL, or end-to-end learning for HPO. This is a paper that combines all of it to a new method. A paper not discussed in this work: Bing-Jing Hsieh, Ping-Chun Hsieh, Xi Liu: Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization. NeurIPS 2021: 7718-7731 It also trains an end-to-end model and the authors demonstrated some improvements over FSBO. **Some Overclaiming of Contributions** In multiple locations, the authors claim to present the " first end-to-end training protocol for meta-BO". This is not the case, see reference above and MetaBO.

Questions

This is a complex method that probably introduce several hyperparameters itself. How do you set them? How robust are they? Do you think there is a chance that this method can be directly applied to a different task or will it require significant work to get it running? Why is NP-EI missing for HPO-B?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors do not address the limitations that come with the complexity of this method, i.e., additional hyperparameters.

Summary

This work proposes an approach to meta-Bayesian optimization (BO) that features a single transformer neural process architecture for both the meta-surrogate function and meta-acquisition function, which are respectively trained via standard neural process maximum likelihood and model-free meta-reinforcement learning (in the style of RL^2) frameworks. The authors demonstrate the empirical benefits of such an approach over prior meta-BO methods on four suites of black-box optimization problems.

Strengths

### Originality My background is in meta-learning and reinforcement learning, not Bayesian optimization. It appears that the proposed approach is novel for meta-BO in that it makes minimal modelling choices, opting for a completely black-box neural process for the meta-surrogate function as opposed to e.g. a Gaussian process, and a completely black-box acquisition function as opposed to e.g. expected improvement (EI). ### Quality The empirical evaluation seems reasonably thorough (though again, I am not an expert in BO), with experiments based on multiple data sources and a healthy pool of competing methods. I also appreciated the described engineering effort in adapting previous methods to handle datasets for which they were not originally designed for. ### Clarity I'm not sure that Lemma 3.1 adds much substance in favor of the paper's design decisions. Instead, perhaps we can more loosely but intuitively argue that typical acquisition functions such as EI depend on the surrogate function's predictions of the underlying objective function (as it well should, to avoid no free lunch), but this dependency is dropped with a completely end-to-end black box acquisition function. The neural process meta-surrogate training can then be justified as softly reintroducing this dependency a la multi-task learning (multi in the sense of surrogate function modeling and acquisition function policy learning, not multiple BO tasks) via a shared architecture. ### Significance I imagine the demonstrated empirical benefit of an end-to-end solution in a domain where handcrafted solution components have been standard will have considerable significance.

Weaknesses

- A minor point -- it is unusual to see one maximizing a "loss" (Eq. 3 and Alg. 1).

Questions

- How is your framing of the acquisition function as a policy with order-invariant processing of the current optimization trace related to ideas in context-based/prototype-based meta-reinforcement learning [A]? It seems like the former is a specific instance of the latter where the state space for each MDP is singleton. [A] Rakelly & Zhou et al., Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables, ICML 2019.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

In this work, they developed the first end-end transformer based architecture for meta Bayesian Optimization and demonstrated empirically state-of-the-art regret minimization on hyperparameter optimization, antibody and chip design problems. They propose using a novel transformer architecture based Neural Process to learn the acquisition function and train it E2E compared to prior work in meta-learning acquisition functions. They identified logarithmic reward sparsity patterns in RL and introduced an auxiliary loss maximizing log-likelyhood of making the correct predictions on their labeled source task datasets as an inductive bias to stabilize training.

Strengths

Demonstrates strong empirical results on their tasks and ablation experiments are included in the appendix demonstrating the importance of their inclusions. Proposes novel combination of transformer network and E2E learning of the acquisition function for Bayesian optimization with a new formulation and inclusion of auxiliary loss. Well written and well positioned with existing work.

Weaknesses

Somewhat limited novelty due to prior separately have utilized transformers for Bayesian optimization and other prior work having meta-learned acquisition functions. Somewhat limited experimental evidence since no hyperparameter search was conducted on the proposed method or baseline. Results were only included on four tasks.

Questions

Did you tune the weighting of the auxiliary loss? How were the hyperparameters chosen for the implementation if no hyperparameter search was conducted.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Discussed limitations limiting to 5000 BO steps due to their use of the transformer network.

Summary

By combining a Transformer-based architecture and reinforcement learning objective, this paper proposes the first end-to-end framework for meta-Bayesian optimization. The proposed framework is claimed to resolve the inefficiency in the existing two-stage approaches.

Strengths

- The meta-Bayesian optimization is an important topic that can be useful in various domains. - The presented motivation appears to be compelling, while the proposition of employing a Transformer-based architecture seems reasonable. - The experiments include practical problem settings.

Weaknesses

- I am not sure if RL is really the optimal way to achieve an end-to-end meta-BO framework. I think this paper lacks a justification for why RL is a necessary component. - Although this paper claims to tackle the inefficiencies from two-stage approaches, introducing RL makes training much harder. The authors add an auxiliary task to solve this issue, but due to this, the proposed method does not seem to reduce the complexity and the objective discrepancy in the previous two-stage approaches. Please note that I do not have much expertise in the relevant fields. I currently do not see any major flaw in this work, but I am also not fully convinced. Comments from other reviewers and further discussion may largely change my score.

Questions

- Is there any factor that necessitates the use of RL?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes

Clarification

The authors may excuse my unclear comment with respect to time. What I was interested in was test time and costly it is. For example, if you consider the compute overhead of your smart searcher into account, is a different able to evaluate more configurations in the same time and therefore find a better solution? Effectively Figure 2 with wall clock time on the x-axis. Edit: Sorry, not my review. I'm interested anyway :D

Raising Rating to 5

I would like to thank the authors for responding to my questions and for their clarifications. After reviewing their response and the discussion with other authors I will be raising my Rating to a 5 due to the clarifications on hyperparameter search and E2E differentially.