Physics-Driven ML-Based Modelling for Correcting Inverse Estimation

Summary

This paper builds on recent advances in black-box optimization for science and engineering inverse problems to find a satisfactory state with small physical error while decreasing the query times to the physical evaluation. The authors propose an error correction method GEESE, which is composed of a hybrid surrogate error model and a generative twin state selection approach. The hybrid surrogate error model is defined by both explicit and implicit errors, with the implicit error modeled as an ensemble of fully-connected neural networks. The generative twin state selection method adopts an exploration and exploitation approach, reducing the cost associated with physical error collection by not directly choosing states via a comprehensive space search. Experiments in real-world engineering inverse design settings demonstrate that GEESE decreases the number of queries needed to find satisfactory states with low error tolerance.

Strengths

The paper introduces a novel correction algorithm, GEESE to find a qualified state within an error tolerance threshold after querying the physical evaluations as few times as possible. Overall architecture is novel, and a twin selection strategy, its key component that simultaneously propose a potentially near-optimal state and a potentially informative state, is effective to decrease the query times. The experiments demonstrate that the proposed method outperforms the baselines in terms of both failure and query times.

Weaknesses

It is still unclear how existed ML model in ML-branch (in Figure 1) is incorporated in GEESE algorithm. Even though a state estimated by the existing ML-model is used for activating GEESE algorithm, what else is the ML-model used for? Is the model also used to generate initial data $D_{0}$? (If so, does the performance of the ML model affect the failure time and query time in the experiments?) If ML-branch is used only for activation of GEESE branch, what is the reason of having ML-branch? In this case, as long as there is a pair of observation and estimated state (not necessarily being a state estimated by ML-model), GEESE is available for the pair. In other words, GEESE is applicable to no matter what ML-models or estimators we have. **Minor comments** line 149: Taking advantage of the robustness of nsemble learning [52, 53] … (typo)

Questions

Is there any theoretical evaluation between the estimated state that activates GEESE and a satisfactory state by GEESE, such as distance between them?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Yes, the limitations are discussed in the main text.

Summary

This paper studies the problem of estimating states from observations (inverse problem). The problem is important since there are many applications such as engine design for aircraft. The current solution usually uses a black box simulator to simulate the observations given the estimated states from the model and compare them with the ground-truth observations to adjust the estimation of the state. This is based on the gradient-based method. However, the current state-of-the-art approach is time-consuming, in which it requires many quires and gradient update steps to find a good solution. As a result, this paper aims to increase the sample efficiency of such a method. The proposed method is called "Generative Exploitation and Exploration guided by hybrid Surrogate Error (GEESE)". It consists of several parts: (1) a surrogate error model that consists of an ensemble of neural networks to provide a fast estimation of the physical error, and (2) a generative twin state selection algorithm that consists of two neutral networks for generating the state's distribution. The algorithm proceeds as follows: it first proposes an "exploitation" state that aims to search for a better solution that is near optimal, and then an "exploration" state that aims to explore other states that might lead to a better solution. The proposed method is tested in three real-world inverse problems and shows a competitive performance.

Strengths

1. The writing is very clear, and the notation is great. 2.The method is simple and intuitive, and it shows good results based on Table 1.

Weaknesses

1. The method requires the training data to train the neural networks to estimate the error. This would require a lot of computational cost and time to collect high-quality data. 2. The proposed method essentially tries to learn a "system dynamics model" using neural networks and exploits this neural network to do optimization. As a result, you do not need to query the simulator many times. 3. Following on the above point, this means that the quality estimation heavily relies on the performance of the hybrid surrogate model. How many datasets do you use? If I understand correctly, first you "pre-train" the model, and then use that for getting the results in Table 1? 4. There is no explanation about what are the tasks presented in Table 1. I knew it states that they are in the appendix, but the reviewers are not required to read the appendix. So it would be nice to include some detail about the tasks. 5. The observation dimension in the paper is quite low, compared to some prior works that estimate the state from image observations (https://proceedings.neurips.cc/paper_files/paper/2022/file/6d5e035724687454549b97d6c805dc84-Paper-Conference.pdf, https://arxiv.org/abs/2102.06794).

Questions

1. In Table 1, there is a metric called query times. Could you please report the world-clock time in seconds? I am curious to see if these three problems simulator actually requires an hour/minute to run. 2. How will the model perform if the size of the observation is in the order of thousand?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See the weakness section. Overall, I think this paper needs some further discussion.

Summary

This paper proposes a sample-efficient method for correcting failed states in surrogate inverse problems. This is achieved through error estimation of optimized surrogate states, where some computationally expensive errors are approximated with a neural network, while the simple errors are computed explicitly. Exploitation and exploration networks sample new states to verify errors for, finding feasible states that will not lead to failure. Several science and engineering applications are shown for the proposed method, and it outperforms the baselines for all tasks in query time and failure times.

Strengths

Well written abstract, concise, showing core idea and significance in the field. The proposed method is well explained, and easy to understand. The authors did a thorough comparison with baselines, and ablation study to investigate the different components of the work.

Weaknesses

1) Minor note: The introduction could be shorter, its purpose is very clear in that it leads to the contributions and methodology of the proposed work in the last paragraph, but there might be too much information leading up to that. 2) The D-dimensional state space first mentioned in Line 151 lacks a practical introduction, the reader might find it hard to imagine what such a state space could look like. Is it a meshed airfoil shape, is it the 3D position and velocity vectors of some object, etc. This is mainly important since in Line 183 the authors explain the generative model for a candidate set of these states, which is then again more difficult to understand without knowing what exactly these states are. 3) The implicit errors should be more thoroughly explained, it is unclear how "expensive to collect" is defined, and once again, a practical example of how these look like would be appreciated. Most important of all, it is unclear why these should be estimated with a surrogate in the first place. If it is due to computational complexity, an analysis should be done on how slow the computation is, and how much of a speedup the surrogate can offer, all the while showing the limitations of the surrogate in terms of generalizability, since the error estimation likely worsens when the explored states are far off from existing data.

Questions

1) In line 99 for the related works paragraph on RL, is it correct to assume that there is no practical benefits of RL for inverse problems as opposed to the black box optimization mentioned in the paragraph above? At least there does not seem to be any advantages mentioned in this paragraph on RL for inverse problems. Is RL able to find better optima, but is just resource-intensive? 2) Why is the exploration network using a generative model at all? Is the state space poorly parametrized, i.e., too high dimensional? Otherwise the maximization in Eq. 10 could just be performed over all states, and not the generative model of the states. 3) Did the authors tune hyperparameters for the baselines as well for a fair comparison? 4) In Line 285 the authors simplified the setting of X=Z, is it correct to assume there is no generative model in this case?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are clear, although understanding why the surrogate for implicit errors is necessary should be elaborated. This was introduced in a too vague manner.

Summary

The paper is very good and interesting - and on a relatively novel topic that has so far recently seen very less attention in the AI community, and is only starting to see more attention with the recent questions on fairness and trust in AI models. The proposed GEESE algorithm is interesting, the foundation of the paper is mathematically sound, and this reviewer appreciates the idea of the authors to use the algorithm to build trust and confidence in traditional AI models.

Strengths

The contribution to the world of physics + AI combined is in itself very useful for the AI community. The major USP of the paper is its clear thought process on how in complex real-world engineering systems the error tolerance is often very low, and thereby, one can obviously not trust traditional optimisation techniques in ML. The idea of using physics-informed optimisation is appealing.

Weaknesses

Only issue with the paper is its title - it is very generic. This reviewer would suggest the authors to make the title more specific (e.g. include the mention of ML models in the title or something similar to show that it pertains to utility in ML applications, as is the thought with NeurIPS!).

Questions

N/A

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations have been clearly addressed in the conclusions section/discussions of the paper.

Summary

The article presents a new method of solving inverse problems using what they call is a grey box method. The introduce key concepts of using Generative methods to reduce the number of objective function invocations in an optimization problem. The presented results are very good and show a lot of promise for the techniques. Although there are a few points that are not addressed properly. They are mentioned in the sections below. Another good thing with the method is that you can continue doing Exploration in Parallel, although this is not discussed in the paper. But on this point, I have other concerns that are mentioned in the Questions section.

Strengths

The methodology is explained in detail. The appendices are used adequately to describe the required concepts and proofs in detail. The code is provided in the supplementary material, but I could execute it because not much guidance was provided on how to execute the code.

Weaknesses

The method of explanation is not correct. In lines 126 to 130 authors have explained the algorithm briefly, which is not explained correctly. Important details are missing which creates misunderstanding. A better way would have been to introduce a complete sketch of the algorithm in the form of an algorithm that could be presented here. Then each part of the algorithm could be explained later, for example, Hybrid Neural Surrogate Error Models and Twin State Selection. Another way could be do first embed these fundamental concepts for the reader and then explain the algorithm in one go. Personally, I will recommend the first approach because that is iterative and reinforcing. Following are some language-related issues. - Line 83 A grey-box setting is thus more suitable in practice, where ones do not [Correction] A grey-box setting is thus more suitable in practice, where one does not - Line 141: followed by the twin state selection for charactering the probability [Correction] followed by the twin state selection for characterizing the probability - Line 257 for evaluatation are explained [Correction] for evaluation are explained - Line 307 general, hile outperforming other methods [Correction] general, while outperforming other methods

Questions

Line 207, if you are not training the generator then there it can be sampled from an appropriate distribution. How random weights are going to help you when they are not being trained?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The authors have mentioned the limitations of complex systems with a large number of states. They have mentioned that they will work on this problem in the future. They will look into how they can solve the problem of extended training time and data requirements.