Active Learning for Derivative-Based Global Sensitivity Analysis with Gaussian Processes

Summary

This paper considers active learning strategies for global sensitivity analysis of expensive black-box functions to efficiently learn the importance of different input variables. Novel active learning acquisition functions are proposed to target key quantities of derivative-based global sensitivity measures (DGSMs) under Gaussian process surrogate models. The study showcases the application of active learning directly to DGSMs and develops tractable uncertainty reduction and information gain acquisition functions for these measures. Through evaluation on synthetic and real-world problems, it shows that active learning acquisition strategies enhance the sample efficiency of DGSM estimation, especially with limited evaluation budgets.

Strengths

1. The paper is well structured and the main points are clearly outlined. 2. Novel acquisition functions are derived to target key quantities of derivative-based global sensitivity measures (DGSMs) under Gaussian process surrogate models. 3. The performance of the proposed active learning acquisition strategies are evaluated by numerical experimens.

Weaknesses

1. The main goal of global sensitivity analysis is to determine the impact of input variables on the output of a model. The most effective performance metric for this purpose is yet to be identified. 2. There is a need for a practical application that showcases the real-world value of using acquisition functions as presented in the article.

Questions

1. In the right panel of Figure 1, why are the curves in the middle and bottom graphs so similar to each other? 2. In Figure 2, why do some curves rise again as the number of active learning iterations increases? 3. Even if the global sensitivity estimation is very accurate, in practical applications, is it still necessary to use another model to accurately estimate f?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

The authors propose eight acquisition functions for active learning of functions of the gradient of a Gaussian Process, which is motivated by the use of the gradient for global sensitivity analyses. They provide experimental evidence assessing the relative performance of these acquisition functions.

Strengths

The paper’s contribution (a novel application of active learning algorithms and the derivation of several acquisition functions tailored to this application) seems strong. The experiments are thorough, and the paper is well-written and the method clearly presented.

Weaknesses

- Although the application is novel, the proposed acquisition functions are straightforward applications of standard acquisition criteria in the active learning literature. - The authors provide experimental evidence on the performance of their proposed method(s), but do not provide explicit convergence guarantees or other theoretical results. - There appear to be many assumptions implicit in the authors’ method, such as that the sensitivity measures are well-approximated by the GP gradient. I would be interested in seeing more discussion of whether these assumptions are required for the effectiveness of their proposed method(s).

Questions

- On pg. 2, the authors state “To learn the squared DGSM, active learning selects points that are adjacent to existing observations, as that adjacency is valuable for the derivative estimate.” This suggests to me that a natural baseline comparison in the experiments would be a heuristic method that selects pairs of points some small distance apart. Do the authors have thoughts on how their proposed acquisition functions would compare to such a heuristic method? - On line 135, should $m_{\boldsymbol{x}}$ be $m_X$? - Why is an information gain acquisition function not derived for the absolute gradient?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors discuss limitations of their work.

Summary

In this paper, the authors study how to select observation data to improve the efficiency of sensitivity analysis. The focus is on measuring sensitivity through a function’s gradient variability. The authors provide a derivation on gradient variance formulae based on Gaussian process surrogates. Various acquisition functions based on gradient, absolute gradient, and squared gradient are described. Empirical evaluations show that the information gain of derivatives and squared derivative perform best in the majority of experiments but in high-dimensional problems, max variance of derivative and squared derivative performs the best. Moreover, the proposed methods can effectively discover the correct ranking for different variables in the sensitivity analysis.

Strengths

1. This work is novel in its systematic study of active learning in DGSM (derivative-based global sensitivity measure). The exposition is clear to me even though I am not an expert in sensitivity analysis. 2. The authors provide comprehensive empirical evaluations of their proposed approaches including both synthetic functions and real-world applications. The results of RMSE estimations show significant improvements over baseline methods.

Weaknesses

1. It seems that some categories are missing some methods. For example, in section 4.3, there is no acquisition function based on information gain for absolute gradient. In the baseline methods, there is no $f_{V_r}$, i.e., maximizing variance reduction on function values. These omissions make the empirical evaluations incomplete. 2. The NDCG (normalized discounted cumulative gain) results are weak compared to the RMSE results. Can the authors elaborate on the practical implication of this? Is RMSE typically more important than NDCG?

Questions

1. What GP kernels did the authors use in their experiments? 2. In Figure 2, RMSE sometimes went up with more active learning iterations. Why is that? 3. Based on the discussion in section 5.1, would it be possible to combine the information gain-based acquisition function with the max variance-based acquisition function to produce a new acquisition function that works well in both low and high-dimensional problems?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discussed limitations on some of their acquisition functions.

Summary

The authors develop and compare sequential design criteria for Gaussian-process-based estimation of gradient-based sensitivity metrics to assess the importance of individual variables.

Strengths

The problem of learning variable sensitivities is well-motivated and a matrix of criteria are proposed to address it, from considering different gradient quantities (the gradient itself, its absolute value, and its square) and different measures of sequentially learning them. This paper clearly addresses an interesting problem and presents itself well. The numerical experiments are sufficient in rigor and quantity to establish the performance of the proposed method.

Weaknesses

As the authors point out on the checklist, theoretical guarantees on the performance of the acquisition strategies are not available but the numerical experiments are sufficient to establish them.

Questions

I think it's great that the authors considered so many different approaches in their experiments, but it's disappointing that this does not result in practical takeaways for the reader in the conclusion section. If I'm an engineer and want to learn about the sensitivity of my simulator to its parameters, which of your acquisitions would you recommend I start with? Such a discussion would benefit the practioner-reader. Maybe just a sentence or two (beyond what is in 5.1) saying "starting with XYZ is our general purpose recommendation" or some such. If there is any possibility of adding execution timings to this article I think it would make it more useful to practitioners, regardless of what those timings show.

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The numerical experiments consider a variety of functions, but consist of very limited budgets. There appears not to be any discussion of the overhead required to conduct the sequential design. The authors mention that "All methods were implemented to be auto-differentiable and, therefore, are efficiently optimized with gradient optimization". It is great that the methods were implemented in a framework allowing for automatic differentiation. But this does not guarantee that the optimization will be efficient. It's true that BO literature often glosses over the overhead required, and that for experiments of very high cost, it is negligible by comparison. But it is still important to report the overhead execution time. As they discuss in the checklist, there is some discussion of the complexity order in Appendix B but no reporting of actual execution timings. This is important to give practitioners thinking about using this method when faced with an expensive-but-not-overwhelmingly-so simulator to decide whether the overhead associated with this method is tolerable relative to their particular application or not.

Official comment by Reviewer Uip4

Thanks for the response that basically addresses my major concern. I'll raise my score later as a response.