Neural Conditional Probability for Uncertainty Quantification

Summary

This paper proposes Neural Conditional Probability (NCP), a novel operator-theoretic approach for learning conditional probability distributions. Extensive theoretical results are provided to support the optimization consistency and statistical accuracy of NCP. NCP can be used to extract conditional density and compute statistical measures such as conditional mean, variance, moments and CDF once it is trained. Experiments on a collection of conditional density estimation datasets are conducted to highlight the efficacy of NCP.

Strengths

- This paper is mathematically solid and well-organized. - This paper focuses on a fundamental problem of learning conditional distribution in statistical learning and introduces an effective and simplistic approach that outperforms baselines with more complex architectures.

Weaknesses

- The proposed NCP method is not clearly motivated or introduced. In Line 49-50, the authors mention that NCP does not belong to any of the four aforementioned approaches. But how is NCP in contrast with them and in what aspects does NCP make improvements? I believe adding some intuitive explanations accompanying theoretical analysis would help improve the readability. - Some key concepts or methods are not clearly explained, which makes it hard to understand the contributions of this work. For example, why is learning *conditional expectation operator* considered useful? Are there any baseline methods that also learn expectation operators?

Questions

Please see Weaknesses.

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of their work.

Summary

The authors propose a method (Neural Conditional Probability, NCP) for learning a conditional distribution P(Y | X) from a finite sample from a distribution. The method is based on following observations: (1) it is sufficient to learn the conditional expectation operator E_{Y | X}[f](x) = E[f(Y) | X = x]; (2) the conditional expectation operator can be written as (an infinite) SVD decomposition which could be truncated at some point, so the problem reduced to learning the finite number of functions in the SVD decomposition; (3) the joint distribution density can be written using the functions from the SVD decomposition of the conditional expectation operator, which gives an optimisation objective for fitting the functions from the SVD decomposition using the sample from a joint distribution. The authors provide an extensive theoretical analysis of the proposed method as well as a simulation study on a few synthetic datasets.

Strengths

+ An interesting, novel and theoretically well-motivated method addressing an important problem of conditional distribution estimation + The method uses a fairly simple neural network (MLP) but achieves the competitive to the methods using much more complex architectures + Thorough theoretical analysis on statistical properties of the proposed estimator

Weaknesses

- Limited experiments restricted to synthetic data making it difficult to judge the potential applicability of this method - It would be nice to have a short summary on the main properties of operators, their SVD decompositions, etc. I could generally follow the presentation without major problems, but having a such an operators summary would have made it easier to read the paper

Questions

- I am wondering about the choice of the specific loss function in Eq. (6). Could, for example, the log-likelihood potentially be used here? If so, what are the advantages of using the Eq. (6) instead of log-likelihood? - The loss function in Eq. (9) is roughly speaking a regularisation term enforcing that the singular functions are orthonormal, is it correct? Could it be possible to build the neural nets with such properties by construction rather than enforce it by regularisation? - What do you think about the scalability of the model to more complex datasets than in Section 6? For example, conditional image generation. Do you expect issues applying NCP in such cases?

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

4

Limitations

The limitations are sufficiently addressed (NeurIPS Paper Checklist)

Summary

The paper proposes Neural Conditional Probability, a novel operator-theoretic approach to learning conditional probability distributions by learning parameters of the truncated SVD of the conditional expectation operator with a neural network. The authors provide a rigorous mathematical derivation and argue for statistical guarantees of their method. The empirical evaluations require major improvements to an otherwise solid paper. **As a general note:** I do not consider myself an expert on the theoretical aspects of learning theory. My background is in Bayesian deep learning and simulation-based Bayesian inference. As such, my confidence regarding sections 3 and 5 is rather low, and my review shall mainly consult on the remaining sections that focus on presentation, embedding into other literature, and empirical evaluations.

Strengths

- The introduction is excellent, with a high degree of accessibility for the broader NeurIPS community and sound motivation of the proposed method. - The method seems mathematically rigorous, well-motivated, and sound. - The authors compare their method against a high number of competing algorithms in the numerical experiments.

Weaknesses

## Major - The Related Work section does a good job of acknowledging related works that aim to learn conditional distributions. However, it utterly fails to embed the current paper into this research landscape. I recommend the authors elaborate on the precise similarities and differences between the referenced papers and their methods in the rebuttal period. - The empirical evaluations are limited to low-dimensional toy problems. This is a stark contrast to the introduction of the method, where the authors repeatedly list the curse of dimensionality as a drawback of other methods. While I acknowledge that the paper is situated in the area of operator learning and ML theory, the quality standard of NeurIPS is not met by the authors’ experiments. This weak evaluation does not do the remainder of the paper justice and I strongly recommend the authors overhaul the experiments to feature high-dimensional tasks that cannot be solved with other state-of-the-art methods. This constitutes a major revision, and this is the main reason why I cannot recommend acceptance to NeurIPS 2024. ## Minor - The empirical evaluation is missing some important information for real-world applications: What are the approximate wall-clock times for (1) training and (2) inference of the competing methods? Further, the authors mention the large required training set size, which might also influence the practically expected training duration in real-world tasks. - Please fix the citations throughout the manuscript: Most citations are ‘text citations’ even if their embedding in the sentence warrants parenthesized citations (Author, 1976). - This is just a personal preference, no need to address it: The ‘paper organization’ paragraph at the end of the introduction does not add value and the space could be used more efficiently elsewhere in the manuscript. - The first sentence in the conclusion is incomplete.

Questions

- As per your answer to checklist item 5 (Open access to data and code), I would like to request access to the full and reproducible code of the empirical evaluations. - Since you want to compare your method with state-of-the-art conditional density estimation methods: Why don't you benchmark against conditional flow matching?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

- The authors mention limitations throughout the manuscript, which I appreciate. However, I would recommend adding a dedicated **Limitations** section in the conclusion to give a compact overview for readers who don’t engage with the entire paper in detail. - The performance of NCP in high dimensions might be a limitation, but the authors do not study this crucial setting.

Code now available

Hello reviewers, There has been some discussion of wanting to examine the code for this paper. The authors (following the official guidelines) have sent me a link to a repo containing the code. I have checked, to the best of my ability, that this is anonymous, so I am passing on the link to you here: https://anonymous.4open.science/r/NCP/README.md

Acknowledgement to the reviewer

We would like to thank the reviewer for their comments, which inspired us to make additional steps and improve our work. We are happy that our rebuttal was helpful, and we commit to incorporate it in the revised manuscript.