ExPT: Synthetic Pretraining for Few-Shot Experimental Design

Summary

The authors introduce a novel approach SynTO to address a challenging setting - few-shot black-box optimization. Specifically, SynTO adopts a pretraining-adaptation pipeline. SynTO can be pretrained using synthetic functions and then adapt to downstream tasks via an in-context learning manner. Comprehensive experiments demonstrate the effectiveness of the proposed method in multiple settings.

Strengths

1. The proposed few-shot BBO is more real-world applicable and generalizable to multiple optimization tasks. Generally speaking, the paper is technically solid. 2. It is interesting to pretrain models with synthetic data from families of other functions. 3. The paper writing is clear, and the presentation is satisfying.

Weaknesses

1. SynTO assumes the access to large amounts of unlabeled data, which may cause unfair comparisons as the other methods are trained only with a few labeled data. 2. In Table 1, albeit with the extra unlabeled data, SynTO seems to perform inferior in some settings. 3. It would be better to give the efficiency of SynTO compared with previous methods, since it further introduces an additional pretraining step.

Questions

1. Can you explain why SynTO degrades more on Max scores under the poorest setting? 2. In Table 4, can the forward and inverse modeling methods be directly compared regarding their qualitative results, considering the two approaches generate different kinds of outputs?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

The paper tackles black-box-optimization from few-shot examples, by pretraining a transformer model on synthetic proxy tasks using in-context learning, and evaluating with the same procedure but with real data. The synthetic tasks are generated by using 1) real unlabeled data and 2) a synthetic generative process that generates different tasks (i.e. input/targets pairs $x_i$, $y_i$) given the unalbeled data (inputs) for the task of interest. The procedure studied to generate the synthetic tasks is Gaussian processes with a radial basis function kernel. During training, the model receives D input/target value pairs generated by the process, and is tasked to predict the the distribution of the input given a target value ($p(x_y|y_i)$) and the context points using a VAE. At inference time, the model receives the few-shot examples context points, and is tasked to predict the distribution for the current task. The approach results in competitive performance against existing methods, which generally outperforms existing methods.

Strengths

- The method proposed is simple and sound, and incorporates recent progress from transformer-based modelling, such as in-context learning which allows to efficiently adapt the model during inference time without backpropagation. - Despite its simplicity on how to train the system and use it during inference, it achieves competitive performance on a wide range of benchmarks. - The experimental section is thorough, giving lots of insights about the method. For example, it gives insights on how the training with GP performs when the (synthetic) tasks during inference are clearly out-of-distribution (Section 4.1), as well as a thorough analysis of alternative design choices such as random vs sorted selection for context/target points, which are shown to be suboptimal. - The paper proposes pretraining on fully synthetic data, which is easy to construct and has less ethical concerns than pretraining on real data.

Weaknesses

The main point of the paper is pretarining using synthetically generated data, and more inisights on why this works at all would make for a stronger paper. My main concerns for the paper are regarding this issue: - For example, regarding the following sentence: L50-51: "Our key insight is, if a model learns to perform few-shot learning on a diverse and challenging set of functions, it should be able to adapt quickly to any objective function at test time." Generating tasks that are "diverse and challenging" from the unlabelled data does not seem enough for the method to perform well, and the concepts of "diversity and challenging" are not well defined in the paper nor in references, neither theoretically nor experimentally. For example, one could generate a totally random process without any type of correlations, which would be both "challenging and diverse" and the method would likely not perform well during inference. Making this notions clearer and relating it to the choice of Gaussian processes, either theoretically or through experimental comparisons with pretraining with other types of processes would make the paper more complete. - More insights on why the selected random process (Gaussian processes) is good, and analysis with other alternatives would make for a stronger paper. For example, although the functions used for synthetic evaluation in 4.1 are much more simple, it would be good to use them as pretraining at least to illustrate this point. - If the only requirement is the data being "diverse and challenging", can real data from similar domains (or other domains that can be adapted to match the desired data statistics) be used for pretraining? Why training with a fully synthetic process is an advantage? - It would be good to perform an analysis of how the statistics of the pretraining data compares to the statistics during inference (e.g. precision/recall as in [1] or similar metrics), to see if the inference task are statistically different or not when compared to the pretraining tasks. This or similar would illustrate if the the diversity of the pretraining data covers all modes of the inference tasks (high recall) and/or whether the diversity is too high compared to the distribution of data for the inference task (low precision). [1] Assessing Generative Models via Precision and Recall. Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, Sylvain Gelly

Questions

Despite the paper showing strong evidence that the proposed method is sound and outperforms alternatives, more insights on why GP works would make for a stronger paper. See weakness for questions.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes, the authors adequately addressed the limitations

Summary

The paper presents a method for tackling few-shot black-box optimization problems in which the model queries a few hundred data points from the black-box function. The proposed method utilizes synthetic pretraining, where a family of synthetic functions is employed to generate data for in-context learning of a transformer-based model. After pretraining, the model adapts to downstream function using few-shot data.

Strengths

- The paper proposes and addresses the problem of few-shot black-box optimization, a problem with many potential real-world applications. - The proposed inverse modeling approach is robust to the quality of synthetic data and allows for gradient-free optimization during testing. - As Gaussian Processes (GPs) are used to generate synthetic functions, there are no extra costs associated with data generation. - Experimental results show performance comparable to previous state-of-the-art methods, while also demonstrating increased robustness to the quality of few-shot data.

Weaknesses

- There's no guarantee that real-world downstream functions follow a Gaussian Process with a Radial Basis Function (RBF) kernel. In other words, performance might be sensitive to the similarity between downstream functions and those generated by a GP with an RBF kernel. While the authors mention the universal approximation property, it isn't certain that the pretraining stage covers a sufficient range of functions over a sufficient number of training iterations in practice. For instance, the hyperparameters of the RBF kernel used in the experiments are bounded, and the range was heuristically chosen.

Questions

- How sensitive is the proposed approach to the choice of synthetic functions? For instance, if we change the kernel of the GP or use randomly initialized neural networks, how is downstream performance affected? - How does the downstream performance vary with the context size? Additionally, does the proposed method perform well when the downstream context size differs significantly from those used during pretraining?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have addressed the limitations.

Summary

This paper investigated the problem of few-shot black-box optimization, and presented Synthetically pre-trained Transformer for Optimization (SynTO). By combining synthetic pretraining with in-context learning to enable few-shot generalization, SynTO demonstrate its superior performance on Design-Bench.

Strengths

+ Synthetically pre-trained Transformer for Optimization (SynTO). + Combining synthetic pretraining with in-context learning to enable few-shot generalization. + Demonstrating superior performance on Design-Bench. + The paper is well written.

Weaknesses

- Can the problem in this paper be directly solved by few shot learning methods? If so, Some experiments may be required to compare the proposed method with the existing few shot learning methods. Else some discussion may be required to explain this issue. - It seems that SynTO shares similar structure with BONET. Thus, some discussion is required to explain the performance gain of SynTO w.r.t. BONET. - Ablation studies are required to clarify which components of SynTO explain the performance superiority.

Questions

- Can the problem in this paper be directly solved by few shot learning methods? If so, Some experiments may be required to compare the proposed method with the existing few shot learning methods. Else some discussion may be required to explain this issue. - It seems that SynTO shares similar structure with BONET. Thus, some discussion is required to explain the performance gain of SynTO w.r.t. BONET. - Ablation studies are required to clarify which components of SynTO explain the performance superiority.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Thanks for the rebuttal

The argument that "A Gaussian Process with an RBF kernel [...] is a universal approximator to any function [1], which means "[...]" it will eventually see data from a synthetic function which is close to the downstream function." is weak to justify the choice, as in practice to perform well on downstream tasks you want both diversity. I understand that, for the particular applications studied, a GP with an RBF kernel (with some ranges of the hyperparameters, as the ones found to be good) may be a good choice and likely to yield functions that are similar to the downstream ones, but this comes from human expertise on this domains, and is not likely to generalize to other domains. E.g. training with Gaussian noise, will also produce a correct image once in a while, but it's a bad choice to train image models. With this, my concerns regarding this process vs alternatives remain. For real data from other domains, I refer to real data that is easy to collect (not data from engineering problems), like images/sound/video... that is rasterized and adapted to match simple statistics of the downstream data, like the mean and variance of the process. This would provide data of high diversity and with potentially useful correlations. I understand that this is out of the scope of the paper.