Continuous-Time Functional Diffusion Processes

Summary

In this work the authors propose an extension to the diffusion models framework for function spaces. In particular, given a Hilbert space $H$, they introduce a noising process based on a Wiener process on $H$. Depending on some assumptions on the type of noise, they then show the existence of the time-reversal process. Then they derive an ELBO via an extension of the Girsanov theorem. Later, akin to the the classical Shannon-Nyquist sampling theorem, they show that for a subclass of functions and grids for which the function is uniquely determined by its values on this grid. Eventually, the process is discretised at the evaluation points, yielding finite dimensional noising and denoising processes. They apply their model to model the CelebA dataset, and show that with relatively few parameters their are able to obtain low FID.

Strengths

- I found it quite easy to follow the submission, especially Section 2 which I think is well fleshed out. - The proposed methodology is rigorous and sound. - Theorem 2 is nice and potentially useful as it gives sufficient conditions to recover the original function.

Weaknesses

- Perhaps the main weakness is the limited experimental setups and lack of ablation studies. As such it is hard to really understand the practical benefits of this specific approach. - In particular, one advantage of working with functional data is to be able to handle data at discretised at arbitrary resolutions or on varying and irregular grid, yet this work practically focuses on the CelebA dataset. It is interesting to see the parameter efficiency by representing as the data as function, but indeed there is no hope to beat method that assume this fixed grid setting and can rely on specialised architectures (e.g. U-net). It may be of interest to tackle tasks such as upscaling, or data with missing values, irregular time-series etc.

Questions

- line 98: Why does $R$ has to be diagonal? - line 105: Is it correct to interpret the 'trace-class' as the following: we have $\mathrm{Tr}(R) = \sum_i \lambda_i = Var(W_t)$ that is the trace of the operator is given by the arithmetic series of its eigenvalue which itself is the variance of the $R-Wiener$ process, so this can be seen as a finite variance constraint? Which seems pretty natural. - Equation 3: How can we define a density $\rho_t$ since there is no 'Lebesgue-like' (translation invariant) measure on infinite dimensional Hilbert space? Is it w.r.t. a Gaussian measure? - Table 1: It is nice to see that FDP can be parameter efficient. Is the reason for not trying larger architecture computational? If so what's the bottleneck? Would $\inf$ - DIFF with 1M parameter be performing as well? - line 333: The MLP architecture seems very deep, have you tried something perhaps a bit shallower yet wider (e.g. 512 neurons)? - Section 6: What noise $R$ was used for this experiment? From a Karhunen-Loeve perspective would make sense to use the correlation of the data process see [Angus et al 2022]. Spectral Diffusion Processes, Phillips, Angus and Seror, Thomas and Hutchinson, Michael and De Bortoli, Valentin and Doucet, Arnaud and Mathieu, Emile, 2022

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

- Something that has not been explored, yet can be left , is conditional sampling which I believe is often of interest e.g. looking at the neural processes literature.

Summary

This work proposes *functional diffusion processes (FDPs)*, which generalizes the SDE-based continuous-time framework for diffusion models for data living in Hilbert spaces. The work builds on the theory of infinite-dimensional SDEs and their time reversals, as well as an application of the infinite-dimensional Girsanov theorem to derive an ELBO objective. A practical implementation, via discretization and implicit neural representations, is proposed and evaluated on the Celeb-A dataset.

Strengths

- The paper is generally clear and well-written throughout. - This work adds to the growing literature on function-space diffusion models (a quite active area) and will likely be of significant interest to the machine learning community. The main theoretical portion of this work places continuous-time, SDE-based functional diffusion models on solid theoretical grounds. - All theoretical claims throughout the paper are precisely stated and rigorously justified. To the best of my knowledge, the proofs of the claims (contained in the appendix) are correct.

Weaknesses

- The experiments (section 6) are the weakest part of the paper. - The proposed methodology obtains significantly worse FID scores than standard (Euclidean) diffusion models. However, note that this is achieved with far fewer parameters. - It was also somewhat unclear how the "complexity" column in Table 1 was derived, as this seems like a highly subjective measurement - Also in Table 1, it was unclear why FID scores were being compared against FID-CLIP scores (for the model of Bond-Taylor & Willcocks, 2023). - It was unclear if the parameter counts in Table 1 also included the parameters for the INRs, or merely the parameters for the score network. - The relationship between this work and the various other concurrent function-space diffusion model works could be better clarified. #### Minor comments - The notation on line 169 was somewhat unclear to me on a first read, particularly what $\hat{\mathbb{Q}}_0, \hat{\mathbb{P}}_0$ represent - There may be a small typo on line 169: should it read $d \hat{\mathbb{P}}^{\chi_T} = d \hat{\mathbb{P}}^{\rho_T} \frac{d \chi_T}{d \rho_T}$?

Questions

- The work restricts its attention to classes of functions which can be reconstructed exactly via finite sets of samples (Sect. 3). If my understanding is correct, the observation points may vary across functions. Does the proposed practical implementation allow for this? If so, how would you expect performance of the model to change if one were to e.g. train on one discretization and sample on another, such as an image super-resolution task?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The limitations of the work are adequately addressed in the paper.

Summary

The paper introduces a novel continuous-time diffusion-based generative model on function space. Unlike previous works in this area, which primarily focused on discrete-time formulations, the authors concentrate on stochastic differential equations (SDE) in their approach. The paper begins by defining the forward process, also known as the noising process, as an stochastic differential equations (SDE) on function spaces. The paper subsequently demonstrates the existence of corresponding backward processes under certain conditions, similar to the formulation of finite-dimensional score-based generative models. Notably, two constraints in the forward process are required to guarantee the existence of the reverse diffusion process, which distinguishes the current work from its finite-dimensional counterparts. Firstly, the perturbation noises should be cylindrical Wiener processes, including $R$-Wiener process with a trace-class covariance operator $R$. Secondly, an operator $\mathcal{A}$ can be present in the drift term of the forward process, which generates a strongly continuous semigroup; the semigroup will serve as a contraction map. The paper highlights the novelty of its results by emphasizing that previous approaches have not fully explored the limits of discretization. Therefore, the provided proof offers theoretical support for the utilization of various types of integrators, unlike previous approaches. Furthermore, the authors introduce an evidence lower bound (ELBO) on the reverse diffusion process, leveraging an extension of the Girsanov theorem. This result provides a theoretical foundation for optimal guarantees, in contrast to previous approaches that relied on heuristics to minimize score matching objectives on function space. The paper also extends the sampling theorem, enabling the reconstruction of the original function from its evaluations at a countable set of points. This extension justifies the training of function-space models with finite observations. Consequently, the authors exploit the sampling theorem to employ implicit neural representations for the model parameterizations, unlike previous methods that predominantly relied on neural operators. Finally, the paper demonstrates the effectiveness of the proposed method on various image generation benchmark datasets. In the experiment, they use Fourier basis functions inversely proportional to the square of the basis index for the perturbation noise.

Strengths

I believe that the paper's theoretical results will make significant contributions to the machine learning community for several reasons. First, the existence of the reverse process ensures the optimality of common model parameterizations (Markovian), specifically those that model only the drift term of the reverse process, including score modeling. Secondly, the SDE formulation sheds light on the utilization of various integration methods for sample generation, such as DDIM or exponential integrators. Furthermore, the derivation of the evidence lower bound (ELBO) using Girsanov results in Equations 12 and 13. In these equations, the covariance operator $R$ is presented in the norm of the $R^{1/2}(\mathcal{H})$ space. The inclusion of this additional $R$ helps address the problem encountered when defining the Kullback-Leibler (KL) divergence for previous generative models based on discrete-time diffusion in function space. Lastly, the sampling theorem introduces intriguing open questions regarding the parameterizations of score-based generative models in function space. This opens up the possibility of employing new network architectures beyond neural operators.

Weaknesses

Most of the content in the paper is good. However, some improvements are needed in the experiment. In particular, the characteristic of the function-spaced model is its resolution invariance. However, the analysis of this aspect is missing in the paper's results. I believe that some analysis regarding the resolution-invariant property of the learned models should be included, even if it's just a small amount. In such analysis, I don't think it is necessary to strictly rely on an implicit neural representation.

Questions

Unlike neural operators, implicit neural implementation is powerful, but achieving discretization invariance is challenging. How do you think is the best way to approach this?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

Please refer to the comments provided in the weaknesses section.

Summary

The authors propose a model approach called Functional Diffusion Processes (FDPs) that generalizes score-based diffusion model to infinite-dimensional function space. The authors derive the reverse time dynamics and sampling theorems to find a subset of functions on a countable set of samples without losing information. The authors demonstrate their approach on a multi-layered perception.

Strengths

1. The theoretical parts of the paper appears to be well structured. However, the paper itself does not appear to be self contained and has many references to the appendices to understand. For example, the paper makes many references to the assumptions being made, but they are never mentioned in the main paper and are only mentioned in the appendix.

Weaknesses

2. The authors claim their approach is to create a new breed of generative models in function space, which do not require specialized network architectures, and that can work with any kind of continuous data. This itself is an interesting idea. However, after reading the introduction and looking at the experimental results. I cannot see what applications this may benefit and how much of an improvement it has to have a purely functional domain. I believe this could be better motivated. 3. This may be because I didn't understand the experiment in Section 6 and does not appear to be self contained. I can understand that there is a page limitation, however, I believe that the experiment section is a bit compressed and difficult to understand. Having said that, the authors do have additional experiments in the appendix which appears to be much better structured.

Questions

The theoretical parts of the paper is quite dense and I cannot say I have fully understood the paper to come up with any meaningful questions.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The authors introduce a framework for infinite dimensional diffusion models in Hilbert spaces, including deriving a reverse process and novel ELBO loss objective. The authors introduce a novel score model network architecture.

Strengths

- This is a timely paper with a lot of interest in the community. There are many concurrent works also looking into similar topics e.g. [1] (I do not expect detailed comparison given they were only public recently). - Although performance is not great compared to standard diffusion methods in terms of generative modelling, the results are better than Neural Processes and similar point wise methods (with the exception of [2], concurrent so cannot criticise for worse performance) - The use of implicit optimization based encoder, g, in equation 16, is interesting. I have some concerns detailed below but nevertheless it is interesting. - Theoretical results seem correct but have not been checked thoroughly [1] Infinite-Dimensional Diffusion Models for Function Spaces, Pidstrigach et al, 2023 [2] Infinite resolution diffusion with subsampled mollified states., Bond-Taylor et al, 2023

Weaknesses

- The implicitly defined INR encoder, g, detailed in equation 16, considers gradient descent in an inner layer. Will this not be quite slow and hence generation / sampling would also be quite slow? - Empirical performance / FID scores are quite bad in comparison to other standard methods. Although interesting, is there an application where this method would be more useful than standard methods? - Functional diffusions could be interesting for dimension agnostic diffusions, super resolution, irregular data. There are limited experiments looking into these other applications. - Complexity in table 1 is quite subjective. I would argue parameter could does not matter if it can be ran on a single GPU in reasonable wall clock time. The implicit layer, g, with gradient descent for equation 16 is sequential so I imagine this slows down generation significantly. - Calling other concurrent / prior work heuristic without details. "However, these works do not formally prove the existence of a backward process and their score approximation and score matching optimization objective is heuristic". Does not [1] prove existence in Theorem 1? (I do not expect detailed comparison given they were only public recently, but given the authors claim this conccurrent work is heuristic I would be interested to know more). [1] Infinite-Dimensional Diffusion Models for Function Spaces, Pidstrigach, 2023

Questions

- The implicitly defined INR encoder, g, detailed in equation 16, considers gradient descent in an inner layer. How many steps are taken? Is the gradient through the score network taken through these gradient descent steps? Is this not quite slow? Have the authors considered other point-wise encoders?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have a brief paragraph on limitations regarding perform, more challenging experiments and comparisons to other approaches like NFO, but could be further discussed. See questions and weaknesses.

response

Thanks for the clarifications! > We explored deeper MLP architectures and found no substantial benefits Would you have any idea why? Can the architecture of Infty-Diff be used here?