Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks

Summary

The authors consider the task of learning curve extrapolation, i.e., the aim is to predict the performance of a given model wrt. e.g., accuracy/log-likelihood over time, given current observations. Their proposal primarily relies on Prior-data fitted networks (Müller et al., 2022) trained on samples from a prior set of curves adapted from their sole comparison method by Dunham et al. (2015). Compared to this prior MCMC-based method (Dunham et al., 2015), the authors can show greatly improved performance in most setups as well as very strong improvements in runtime.

Strengths

- The paper focuses on an important task in the AutoML literature that has so far not received a lot of attention. - The authors evaluate their approach extensively on a varied set of experiments properly evaluating all their claims. - The paper is overall well written with two minor deficiencies (see next section). - The authors provide an extensive implementation of the model. However, it is also necessary as replicability would be difficult given the coarse level of detail in the written part. - With respect to originality. The contribution is primarily in the application. The model itself relies on the PFN model by Mueller and the prior is an adaptation of Dunham et al. This can be seen as a weakness, yet given that the application of PFNs to this field is novel and the application is an important one with clear results, I see this only as a very minor weakness if at all. - Concerning significance, I lack a deep understanding of the AutoML literature to properly judge the significance. But from what I know of the field, the results look very promising and should be of great interest to many readers.

Weaknesses

- It has two deficiencies in the writing/structure of the paper. 1. Its backbone is a prior-data fitted network. However, that model itself is only briefly introduced in a single paragraph with a figure that is barely understandable without reading the original paper. The paper requires a proper discussion of this approach in either the main text or the appendix. 2. Training details and hyperparameters are barely discussed with many essentials missing, e.g., the reader has to guess the meaning of `nb_data`, `emsize`, `nlayers`. It is an easy guess, but a guess nevertheless. Adding the abbreviations to the paragraph in l157 would quickly fix this. - As stated above, the novelty is almost solely in the application which could be considered a weakness. But in my opinion this is only a very minor weakness if at all. - Table 3 lacks error bars. ## Minor - Figure 2 is missing details on the colors of the arrows - The caption of Table two should be above - Figure 3 contains an unexplained horizontal line - l319-320 the sentence is broken (interesting appears twice)

Questions

- In l216 the authors mention that the best MCMC-PP method took over 80 seconds. In Table 3 the 80s method does, however, not not improve upon the 30-second version. Can the authors clarify this statement?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss limitations of their method, but no potential negative societal impact of their work. The second is of lesser importance in this work anyway.

Summary

The authors in this submission applied the prior-data fitted neural networks (PFNs) in the learning curve extrapolation task, for which the main goal is to predicit the performance of a machine learning model in later epochs, based on the information from earlier epochs. The target is modeled as a linear combination of basis growth curves, and the proposed LC-PFN algorithm aims to minimize the cross entropy loss as in PFN. The authors tested the proposed method across a few datasets and showed that PFN (i) achieves better prediction and (ii) is computationally efficient, when compared with its competitors.

Strengths

1. The idea of applying PFNs in the learning curve extrapolation task sounds reasonable, and is intuitively more efficient when compared with MCMC. 2. The application of LC-PFN in early stopping could be valuable, as it can be helpful for the model selection step. The experiments in Section 4.3 also provide some promising results.

Weaknesses

1. In my opinion the selection for the 3 parametric basis curves seems a bit ad-hoc. It's not convinving whether they are sufficient to fit different learning curves. Furthermore, selection of hyperparemters in priors there also lacks details. 2. The results could be more convincing if the authors can compare aganist more previous methods, instead of MCMC only. For instance, how about the one by Klein et al., 2017, which used Bayesian neural networks for learning curve prediction?

Questions

1. In Table 1, is the support for x integers, as it corresponds to $f_k (t | \theta_k)$ in the model? 2. For the equation just below L132, what does $<$ mean? 3. Do you need to assume $t > T^{'}$ in your LC-PFN model during training?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

The authors propose applying prior-data fitted NNs to learning curve extrapolation. The authors demonstrate that this approach outperforms MCMC inference and is substantially faster.

Strengths

The paper has a novel idea of applying approximate inference via meta-learning learning curves. The method appears to work in experimental evaluation, but more experiments would make the paper stronger. The manuscript (all but experimental section) is clear and relatively easy to follow. The authors provide the code.

Weaknesses

The paper contains only two experiments. One is based on synthetic data. Table 1 has no error bars. Table 1 indicates that while the performance of MCMC is stable across the hyperparameters, PFN score is dependent on the network size indicating it might not generalize well to other problems. While the difference in time is clearly significant, the differences in LL are rather small. While the idea of applying the described method in the considered context is novel, much of this work is a combination of previous work by Domhan 2015 and Muller 2022. The experimental section could have been written in a more clear way.

Questions

Is poor performance of MCMC attributed to slow mixing or initializing at a poor point? What about MCMC inference initialized at MAP? If MCMC has substantially more computational time will it outperform PFN? At the end of the day exact inference should outperform approximate inference.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The authors used prior fitted networks to perform learning curve extrapolation. Crucially, they demonstrate that their method vastly outperforms approximate Bayesian inference via MCMC (both in terms of inference time and predictive log likelihood). Moreover, they demonstrate that the proposed approach outperforms heuristics for ending unpromising training runs commonly used in hyper-parameter optimization.

Strengths

This paper prevents a super practical method that not only performs well *but* also runs fast, allowing it to be used in many real-time applications. The practicality of the method is extremely bolstered by the impressive suite of empirical experiments performed. Lastly, the paper was written well and was pretty easy to follow (though I have some comments).

Weaknesses

The biggest weakness to me is that I don't think the authors spent time on the potential difficulty of using MCMC for this problem. Specifically, the prior, and the corresponding posterior, is constrained on some non-standard subset that prevents standard MCMC algorithms to be used. The geometry of this will heavily affect the performance of MCMC algorithms as samples that are proposed outside of the set will always be rejected requiring substantial tuning. That being said, I am a little concerned with the number of samples the chain was run: 4,000 samples seems too low for this problem.

Questions

Below I will list some comments 1. In figure 1 there is a black dashed line but I can't find in the text what it represents. Without prior knowledge, I thought that maybe it was the true log posterior predictive distribution, which would make it seem like MCMC is outperforming the proposed method. Also, error bars here would be great. 2. For all results, it would help if it was made clear if higher values are better or the converse. 3. For Fig. 4, errors bar would be very informative, especially given the range of values. I suggest plotting 20% and 80% quantiles. 4. In Fig 5., it isn't clear what the error bars are, i.e. standard error, quantiles, etc.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors did a good job explaining the limitation of the methods.