Generative Fractional Diffusion Models

Dear Reviewer kZkj, We sincerely apologize for the formatting issues in our previous response and the missing reference. Here is the corrected version, along with additional answers to the questions we couldn't cover previously due to the character limit: [1] Yang Song and Jascha Sohl-Dickstein and Diederik P Kingma and Abhishek Kumar and Stefano Ermon and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations, International Conference on Learning Representations, 2021. >Why for do we see improvement with the augmented dynamics? Because of the correlation structure? Yes, our hypothesis is that the known part of the score function is guiding the data generating process towards the direction of the true data distribution, since the starting distribution of the augmenting processes is known and all processes are driven by the same random path realization of Brownian motion. In the forward process $\mathbf{Y}_t$ does not depend on $\mathbf{X}_t$, but in the reverse model $\mathbf{Y}_t$ does depend on $\mathbf{X}_t$, since it depends in every entry on the score model evaluation. Assume that $\mathbf{Y}_t$ does not leave its true trajectory when we simulate the reverse time model, in this case the trajectory of $\mathbf{Y}_t$ might serve as a corrector for the trajectory of $\mathbf{X}_t$. >With $K>>1$ the performance deteriorates. Does it means true fractional dynamics does not work well? What is happening? We believe that the most important factor for chosing the optimal number of augmenting processes is the dataset. In Table 1 of the attached Rebuttal PDF, we observe that on MNIST, $K=4,5$ works well in the super-diffusive regime, while on CIFAR10, the performance degrades in the same regime. In Figure 2 of the attached rebuttal PDF, we observe that for the same FVP dynamics, there are different well-performing regimes for the number of processes. (K=3,4,5 on MNIST and K=1,2 on CIFAR10). >How does $K$ impact the approximation of true fBM (you cite a reference on Line 184, can you give something quantitative?) The integrated $L_2$ error over time $[0,T]$ for different choices of $K$ is visualized in Figure 7 of Daems et. al[2]. The higher the value of $K$, the better the approximation. However, after a certain number of augmenting processes, the error saturates, depending on $H$. [2] Rembert Daems, Manfred Opper, Guillaume Crevecoeur, Tolga Birdal. Variational Inference for SDEs Driven by Fractional Noise. The Twelfth International Conference on Learning, 2024. > Table 1 and effect of $K$ for $H=1/2$: Not much difference, until FID deteriorates, where then increases. If the sample quality heavily deteriorates and that the model produces random out of distribution samples, then it is not a very interesting situation and the metric loses meaning. Moreover no discussion on why choosing too high of a eventually leads to bad samples. Numerical instabilities? Training loss? Difficulty learning the augmented process We agree with the reviewers point that a low-quality image with increased pixel-wise diversity is not a favorable situation. In Table 1 of the attached rebuttal PDF, we observe that the pixel-wise diversity increases with a higher number of augmenting processes on MNIST and CIFAR10. However, the quality seems to depend on the Hurst index and the dataset, as we observe on MNIST for larger $K$: | K | H=0.9 | H=0.7 |H=0.5 |H=0.1| |---|-------|-------|------|-----| | 4 | 1.22 | 0.86 |1.86 |6.25 | | 5 | 2.17 | 1.36 |4.89 |9.57 | while on CIFAR10 we observe a clear degradation in quality: | K | H=0.9 | H=0.7 |H=0.5 |H=0.1| |---|-------|-------|------|-----| | 4 | 29.72 | 8.45 |8.85 |5.02 | | 5 | 69.06 | 35.91 |96.54 |7.38 | Our hypothesis here is that $K$ controls the pixel-wise diversity, while the optimal number of augmenting processes for quality depends on the dataset and the dynamics (FVE or FVP).

>The effect of varying the hyperparameters does not seem very consistent. Will the observed remarks for $(K,H)$ hold with different datasets? Will this scale? It would also be nice to have multiple runs with the associated standard deviation to assess statistical significance here. We agree with the reviewer's point that our previous presentation of our findings was not very clear. We hope to adress this issue in our attached rebuttal PDF: * in Table 1 we observe that the super-diffusive regime $(H>0.5)$ yields the best performance on MNIST and CIFAR10 in terms of quality and an increasing number of augmenting processes yields higher pixel-wise diversity. * in Figure 1 we observe that the quality evolution w.r.t. the number of augmenting processes evolves similar across dynamics and dataset. We also agree with the reviewer that it would be favorable to report the associated standard deviation across different run. However, this is beyond our computational resources. One training on CIFAR with 2 GPUs (Ampere A100 40 GB RAM) takes roughly 50 hours, making it unfeasible to train multiple times. Following the evaluation setup of Song et. al[1] we trained for 1 million iterations on CIFAR10 and evaluated one checkpoint every 100k iterations, to ensure that the result is not affected by overfitting. This ensures to some extend that the FIDs we report on CIFAR10 are not a lucky pick. Nevertheless, to adress the reviewers concerns we report in Figure 3c) the average FID and the associated standard deviation for a certain number of sampling steps over three rounds of sampling. Please note that to report one of these scores for $NFE=1000$, we need to sample $50k$ images on a single GPU (Ampere A100 with 40 GB RAM), which takes 16 hours. >In the abstract, limitations of diffusion models are mentioned, in particular (i) [...]. For (i), there is no mention of the number of diffusion steps/number of function evaluation in the experiments We thank the reviewer for pointing out that we did not mention the number of function evaluations (NFE). To address the reviewer's point, we compare the performance of the super-diffusive regime to the purely Brownian-driven dynamics in Table 1 of the attached rebuttal PDF. Evaluating the performance for different numbers of NFE of sampling steps in Figure 3a), we observe that the super-diffusive regime of MA-fBM saturates at $500$ NFE on a lower level than the purely Brownian driven dynamics. >(ii) mode-collapse on imbalanced data, which are not addressed in the experiments and discussed properly.[...] For (ii), maybe try a run on CIFAR10_LT. Unfortunately, this was not feasible given the short time period from review release to rebuttal. Based on the reviewer's suggestion, we will consider experiments on CIFAR10_LT in our future work. >Line 259: 'we evaluate GFDM on [..] test distribution coverage': this seems not to be done anywhere, as Vendi score does not relate to coverage in this sense? Our evaluation of negative log likelihood (NLL) in Table 1 and Table 2 was intendet to measure test distribution coverage, since NLL estimates the negative log-likelihood of test data under the learned density. >The diffusion is run on $[\epsilon, T]$ but the value of $\epsilon$ is nowhere to be found it seems, and no experiment to see its effect on GFDM's performance? Thanks for noting. We defined $\epsilon$ in the Appendix (lines 730-731), but the definition should, of course, be in the main part of the paper. We used throughout all experiments $\epsilon=1e-5$ for training and $\epsilon=1e-3$ for sampling according to Song et. al[1]. >Line 32-34: why would BM's lack of control over sampled trajectories matter? The correlation structure is independent of the model, depending on the data it could also increase mixing time or actually make it harder to learn the score. Is it because the augmented process makes it possible for the model to incorporate better correlation structure? What are the basis of these claims? The Hurst index $H$ in our framework provides control over the diffusion processes trajectory in terms of roughness vs. mild randomness. Our results indicate that the more regular and correlated trajectories are preferred, which is not possible to achieve with BM. As we pointed out in our answer above, a smoother trajectory might decrease the discretization error from continuous time to discrete time steps.

>Line 83-84: $\hat f$ seems not to be properly introduced Thanks for the very detailed review. Since we have $\mathbf{f}(\mathbf{x},\cdot):[0,T]\to\mathbb{R}^{D}$ for fixed $\mathbf{x}\in\mathbb{R}^{D}$ we intended to apply the definition for $\overline{g}$ as well to $\mathbf{\overline{f}}(\mathbf{x},t):=\mathbf{f}(\mathbf{x}_t,T-t)$ in line 80-81. In order make it more clear that the notation applies to any process $\mathbf{P}$ and any function $f$ we replace line 80-81 by: "Whenever $\mathbf{P}=(\mathbf{P}_t) _{t\in[0,T]}$ is a stochastic process and $f$ is a function on $[0,T]$, we write $\mathbf{\overline{P}} _{t}=\mathbf{P} _{T-t}$ for the reverse time process and $\overline{f}(t)=f(T-t)$ for the reverse time function." >Line 168-169, eq (9): typo in the denominator of the last fraction? Thanks for finding this typo. We cut of the lower incomplete gamma function in the second part of $\mathbf{b}_{k}$. We replaced it by the correct term: $\mathbf{b} _{k} := \frac{T}{\gamma^{H+1/2} _{k}}P(H+1/2,\gamma _{k}T) - \frac{H+1/2}{\gamma _{k}^{H+3/2}}P(H+3/2,\gamma _{k}T)$ >Line 210: $\Sigma_t$ seems not to be properly introduced Thanks for worrying about the details. In this case we already defined it in line 195 - 198: "The missing components in the conditional covariance matrix $\Sigma_t$ of the augmented forward process are the conditional marginal variance of the forward process and the conditional marginal correlation between the forward process and the augmenting processes." >Line 291: Did not you mean super-diffusion here with $H = 0.9 > 1/2$? idem in caption of Figure 2, with $H=0.7$? Yes, thanks for noting. This is a typo and should be super-diffusion.

Dear Reviewer HxVk, We sincerely apologize for the the missing references: [1] Yang Song and Jascha Sohl-Dickstein and Diederik P Kingma and Abhishek Kumar and Stefano Ermon and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations, International Conference on Learning Representations, 2021. [2] Aaron Lou and Stefano Ermon. Reflected Diffusion Models, International Conference on Machine Learning, 2023. We hope that we have sufficiently addressed the weaknesses and questions raised by the reviewer. With best regards, The Authors

We appreciate the reviewer's comments and feedback. However, we believe the strong criticism is excessively harsh and unjustified. The absence of a summary or acknowledgment of our method's strengths raises concerns about the reviewer's full understanding of our work. Below, we address the specific comments and questions they posed: > thin numerical experiments. We now provide a set of additional experiments and report the computational considerations in the attached rebuttal PDF. We have included all of these in a further revision. We have also included a motivation for exploiting fractional Brownian motion (fBM) instead of Brownian motion (BM): Having control over the diffusion trajectories are crucial for obtaining a trade-off between generation quality and diversity. Moreover, there is no true support for why the noise increments should be independent. Our framework provides two hyperparameters, the number of processes and the Hurst index to have control over diffusion processes and characterize the nature of trajectories, \emph{e.g.} roughness vs. mild randomness. Our results indicate that in both aspects, quality and diversity, the diffusion models driven by Brownian motion lead to suboptimal results and the best outcome are obtained when $K\geq 2$ and $H>0.5$. This indicates that more regular and correlated trajectories are preferred while a single process is usually not sufficient to capture the entire complexity. > ... the experiments should focus on exactly showing the marginal improvement while controlling everything else, In fact, we exactly do this. We have two parameters, the number of Markov processes ($K$) and the Hurst index ($H$), that differentiate our method from a standard diffusion model. When $K=1$ and $H=0.5$, our method closely recovers the original model driven by Brownian motion. Our evaluations cover a range of $K$ and $H$ values, clearly demonstrating the improvements from our contributions. We have to admit that we suspect the reviewer might have overlooked some of the important parts in our paper. > The paper does not have strong methodology contribution, in additional to the replacement of fractional BM We kindly yet strongly disagree that *our method does not have a strong methodology contribution*. Replacing BM with fBM in diffusion models is definitely a non-trivial task due to the non-Markovian nature of fBM. To achieve this, we have made significant contributions, including deriving a memory and computation-efficient reverse process, proving the optimality of the score model, and providing explicit formulas for the marginals of the conditional forward process. Besides, strength of a contribution is a subjective matter and the reviewer does not mention on which aspects our contribution is not strong. > focus more on understanding the structure of image data rather than the noise Regrettably, this is another subjective comment. We agree that there are various ways to improve diffusion models. In this work, we specifically focuses on the driving noise, which, as shown by our experiments, has a significant impact in quality and diversity. Our work should be evaluated within this context. We find it surprising that the reviewer recommends a different, vague research direction instead of providing constructive feedback on our work. In the light of these and the feedback of other reviewers, we kindly ask the reviewer to reconsider their initial assessment.

As authors, we find it difficult to comprehend the perspective of this reviewer, and we believe that further engagement in what appears to be a non-scientific discussion would not be productive. We have brought this matter to the attention of the Area Chair and respectfully request that the other reviewers form their judgments independently of this review, which we believe to be biased.