Summary
- The study proposes a method to learn the posterior distribution in infinite-dimensional Bayesian linear inverse problems using amortized conditional Score-based Diffusion Models (SDMs). This extends conditional SMDs into the infinite-dimensional function space setting, as existing conditional SDMs have previously only dealt with finite-dimensional vector spaces (noting also that _unconditional_ SDMs have recently been extended to infinite dimensional vector spaces by Pidstrigach et. al). This leads the way for applications in, for example PDEs, where the unknown parameters to be estimated take the form of functions.
- The key technique underlying their approach is to define the _conditional_ score in an infinite dimensional setting, extending the method of Pidstrigach et. al, who defined the _unconditional_ score in the infinite dimensional setting.
- Using their definition of the conditional score in infinite dimensions, this allows them to avoid having to solve a potentially expensive proximal optimization step, as was done by Pidstrigach et. al.
- The authors then provide a comprehensive theoretical analysis of the use of their conditional score in SDMs and show:
- How this newly defined score is used as a reverse drift of the diffusion process, which leads to a generative model that samples from the correct target conditional distribution under certain conditions.
- That as long as you start from the invariant distribution of the diffusion process, the reverse SDE converges to the target distribution exponentially fast
- By explicitly computing the expected square norm of the conditional score, they shows that a uniform in time estimate is not always true for the conditional score. This leads them to provide a set of conditions to be satisfied to ensure a uniform in time estimate for a general class of prior measures.
- That the conditional score can be estimated via a conditional denoising score matching objective in infinite dimensions.
- That the conditional denoising estimator is a consistent estimator of the conditional score in infinite dimensions.
- That unlike the unconditional score, for noiseless observations the conditional score blows up as T->0
- They present a small toy experiment that validates their approach by demonstrating the applicability of their method in approximating non-Gaussian multi-modal distributions.
Strengths
I will address strengths and weaknesses across the four dimensions (Originality, Quality, Clarity, Significance) below.
Weaknesses
**Originality:**
- Are the tasks or methods new?
- Yes, the authors make a novel contribution to the quickly-growing diffusion model literature, by showing how to extend conditional SDMs into the infinite-dimensional vector space, thus opening the door to a wider range of applications (PDEs, etc)
- Is the work a novel combination of well-known techniques? Is it clear how this work differs from previous contributions?
- The authors are very clear about:
1. How they are starting with the recently-proposed framework of Pidstrigach et al
2. The specific point where they deviate from, and then extend, Pidstrigach's work (Definition 2, eq. 8)
- Further, they provide a comprehensive analysis of the use of their defn 2 in SDMs in sections 4 and 5.
- However I should state that I am not adequately familiar with the mathematical techniques deployed in this paper to check their technical claims for correctness or novelty, so please differ to another reviewer with more expertise in this area.
- Is related work adequately cited?
- Yes.
**Quality:**
- Is the submission technically sound?
- It appears to be, although as previously stated I am not sufficiently versed in their mathematical techniques to be 100% sure.
- Are claims well supported (e.g., by theoretical analysis or experimental results)?
- Theoretically yes, experimentally no. I realize this is a theory paper and don't necessarily expect full-scale experiments on PDEs, but in Section 6 I expected to see at least 2 obvious baselines which were foreshadowed in the body of the text but not experimentally validated on the toy experiments. These are:
1. How does their method compare to the crude, discretization-based approach discussed in lines 56 - 61:
> A straightforward solution may be to discretize the infinite-dimensional input and output function spaces into finite-dimensional vectors, and apply SDMs to learn the posterior. Yet theoretical studies of current DMs suggest that performance guarantees do not generalize well on increasing dimensions [7, 9, 33]. This is precisely why Stuart’s guiding principle to study a Bayesian inverse problem for functions— “avoid discretization until the last possible moment” [41] — is more than ever critical to the use of SDMs.
A primary motivation for their method is "Stuart's Principle", which says to avoid discretization until the last possible moment. I would have liked to see experimentally why this principle is so important on the simple toy examples they have provided.
2. How does their method compare to the approach proposed by Pidstrigach for conditional sampling, which uses a proximal optimization step? This is a second primary motivation for their approach (and appears in the abstract) - their method is more performant because it avoids solving an optimization problem at each timestep. As far as I can tell it has not been experimentally validated that their approach is faster or more performant than the baseline method of Pidstrigach. Evidence showing their approach either gets better samples, or get samples of the same quality but more efficiently, is needed, given that a primary motivation for their approach is that the baseline method of Pidstrigach may be too computationally costly because of their use of proximal optimization.
- Are the methods used appropriate?
- Yes
- Is this a complete piece of work or work in progress?
- Yes, modulo the missing baselines discussed above. I believe such baselines are needed to consider this a complete piece of work, given how "Stuart's Principle" and "avoid proximal optimization" play a key role in the storyline and motivation for their technique.
- Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
- Yes. In particular I found their observation, that unlike the unconditional score, for noiseless observations the conditional score blows up as t->0, particularly interesting. However in the conclusion I would have liked to read more about the authors' reflections on the strengths/weaknessness/future directions of their approach, both from a technical standpoint and from the standpoint of potential downstream applications of this work.
**Clarity:**
- Is the submission clearly written?
- I found all writing up until section 4 relatively easy to follow. I began to get lost around sections 4 and 5 and couldn't follow the math, but I attribute this largely to not being comfortable with stochastic differential equations. I could still follow the high-level plot in these sections, but will defer to other reviews to evaluate the technical claims.
- Is it well organized?
- Yes
- Does it adequately inform the reader?
- Yes
**Significance:**
- Are the results important?
- Yes, although I would have liked the authors to motivate the applications of their approach more thoroughly. I believe they only listed PDEs as an example for why you would want to use this approach but surely there are more applications than just PDEs, no?
- Are others (researchers or practitioners) likely to use the ideas or build on them?
- Yes
- Does the submission address a difficult task in a better way than previous work?
- Theoretically it appears so, but baselines are needed to experimentally validate these claims.
- Does it advance the state of the art in a demonstrable way?
- Theoretically it appears so, but baselines are needed to experimentally validate these claims.
- Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
- Yes, they build on the work of Pidstrigach in a novel way.
Questions
- What are some other applications of this work besides PDEs? Can you include a few of them in the introduction so the reader doesn't think applications of your work is limited only to PDEs?
- In the conclusion the authors say their method "is able to perform conditional sampling directly on infinite dimensional Hilbert spaces." Don't Gaussian processes also allow you to perform conditional sampling directly on infinite dimensional Hilbert spaces? What are the differences between your approach and Gaussian processes? When would I want to use one vs. the other? Could a GP be used as a baseline in your Figure 1? If so it would be very interesting to see how it compares.
- The authors also say "We also show that the conditional score can have a singular behavior at small times when the observations are noiseless, in contrast with the unconditional score under similar hypotheses.". Is this just a limiting phenomenon of theoretical interest, or do you expect this to cause difficulties in practice? Are there any applications where we should expect observations to be noiseless or does that assumption never hold in practice?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
- There does not appear to be a "Limitations and Broader Impacts" statement in this work.