Unifying GANs and Score-Based Diffusion as Generative Particle Models

Summary

In this paper, the authors consider the task of generative modeling as one of particle flow, and provide a unifying theory for score- and diffusion based generator-free models, and generator-based GANs. They propose the particle-flow models that encompassed diffusion and score-based methods such as NCSN or EDMs and subsequently like the approach to incorporate GAN losses to result in flows based on the GAN discriminators. The authors subsequently propose two novel training algorithms. One based on training GAN generators with the score of the data, and its own push-forward distribution score, and another, based on discriminator-based flow. While the performance is not state-of-the-art, given the current interest in diffusion based models, a unifying theory that brings to light connections between GANs and flow based models is certainly interesting. I am convinced that these results could be used to leverage insights learned in the past decade of GAN training to potentially improve the performance of score-based models.

Strengths

- The proposed formulation is well presented and easy to understand. The overall concept on unifying score-based and GAN generator based modeling is clearly presented. - The contributions of the paper are generally made clear, presented as definitions and/or finding. - The limitations of the proposed method in achieving state-of-the-art performance are addresses head-on, which, in my opinion, is better, as it manages expectations on what the paper hopes to achieve — a clearer focus on theory, and not on showing SOTA.

Weaknesses

- The training algorithm for Score GANs and discriminator flow could be better clarified. While the supplemental sections do address some of the questions I has, the algorithms / experimentation sections leave a good amount to be desired when it comes to understanding how training progresses (please see the __Questions__ section below for detailed concerns on experiments). - While most formulations are covered, I feel that a few additional comments could be made in the main text on how sampling/gridding schemes (of the SDE/PDE) serves to affect this formulation. For example, the authors could talk about the possible links between adjusted/unadjusted Langevin dynamics, and the NCSN/EDM formulations of PMs. does having $\sigma^{\prime}(t)\sigma(t)$ affect the discretization in any way. Across the framework, the authors stop at formulating the particle flow PDE, but maybe including comments on how discretizing this equation affects the understand could help. Could we drawn some connections with generator training (which, in some sense, is a “way to solve” to PM PDE) and how one discretized these SDE in diffusion/score-based models. Is this direction something the authors have considered? - The transition from equation (7) to (8) is treated a bit too casually, in my opinion. While (7) is an SDE, Karras et al., present a discussion on how the discretization could be split into a Probability flow ODE (PF-ODE), and a stochastic component. The formulation in this paper entirely ignores the stochastic component $\mathrm{d}W_t$, and places itself within the PF-ODE framework. Do the authors feel that this limit the scope of the proposed framework? For example, could they comment on what the implications of having a generator-based framework which solves a variant of (7), including an exploratory noise term? Is it implicitly present in GAN training?

Questions

Beyond the concerns remained in terms of the ODE formulation, I have a few additional concerns regarding the experimental formulation: - I find it a bit hard to understand the overall picture in the training algorithms. I think the authors could add a bit more clarity on how both the score network for the generator in Score GANs, or the discriminator in discriminator flows are trained/used. First, in score GANs, if I understand correctly, the score of the generator would be required at each training step $t$, to approximate the score of a sampling iteration index $t$, correct? The generator, as the authors also remark, is essentially distilling the entire sampling process of a score-based model, to a single generator sampling step. If so, it is not clear how Score GANs are trained. From Algorithm 1, I understand that $s_{\phi}^{\rho}$ is trained for K steps on a DMS loss, when the iterations on $t$ are replaced with noise levels $\sigma$, starting with $x=g_{\theta}(z)$. Is this $g_{\theta_t}(z)$? Does this mean that there are K steps of update per generator update. How large would K need to be to estimate the generator score accurately? Have the authors considered ablation experiments on this, at least say, on Gaussian data? I suspect the need for larger K as the data complexity increases. Minor comment: Maybe mention what $p_{\sigma}$ is, in the main document as well (a one line summary of Lines L714-L720 might suffice). - I have similar concerns with the discriminator training.The presented explanation is not too clear. If my understand is correct, the explanations provided in the appendix indicate that the discriminator, provided with batches of data, considers random $t$-step Langevin sampling, and is trained such that the discriminator mimics this sampling behavior using a chosen loss? What step 6 in Algorithm 3 actually simplifies to, is unclear. It might help to add some intuition on what this loss amounts to minimizing, in the GAN context, (assuming the WGAN-GP loss, as the authors do, as per Table 4). Again, would this be an alternating scheme? Since $f_{\phi}$ is part of the sampler in lines 3-5, Algo. 3, would we need to reevaluate these samples after ever M steps of update on the discriminator? Is there an interplay between N and the number we would need for M? The authors explore this to a small degree in B.3, but I feel that the ablation experiments on these parameters could be a lot more exhaustive, or at least discussed in more detail. - Did the authors also explore various embedding schemes for the time index in discriminator flows, and how different would the proposed discriminator be, from the formulation of Denoising Diffusion GANs (Xiao et al., ICLR 2022)? While the embedding is described in L707-711, it is not clear what Table 4 implies, regarding "Fourier" time embedding, or how frequency scale and embedding size matter. Lastly, while the time-efficiency analysis in B.3 does answer a few concerns I had on this regard, I’m not entirely convinced as to what the discriminator is acutely achieving, in comparison to a 2nd order Heun solver. I get that the performance of the discriminator flow is one par with second order methods, but can we say more about this? Would it be feasible to explore training the discriminator with samples generated via second order discretization? - __Minor__: (Figure 5, caption) … neural of function evaluations … — I’m not familiar with the original definition of the measure, but NFE should either be neural function evaluations or number of function evaluations. Equation (14), on the right of the first equation, I do not see an occurrence of $z^{\prime}$. Should the summation be over $z^{\prime}$? Overall, while I find that experimentation is lacking in some regards, such as ablation experiments, given that the paper is more theoretically aligned, it might at least help to provide insights or intuitive comments for variables/hyper-parameters where experiments are not included. I also think the authors would benefit from making the algorithms a bit clearer to understand, in terms of how the models are trained. [1] Tackling the Generative Learning Trilemma with Denoising Diffusion GANs, Xiao et al., ICLR 2022

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

The paper is clear about its limitations on being an analytical exploration of GANs and diffusion-based approaches, and not an experiments one.

Summary

The paper points out commonalities between diffusion and GAN models by examining the time evolution of the sample particles. For diffusion, this evolution happens at inference time by following the theory-given flow that transports the initial noise distribution to match the data distribution (in practice this flow is memorized by a neural network in the training stage). For GANs, it happens during training: at any given training step, the current generator induces positions for the particles, and they “slide down” the discriminator surface (which likewise jointly evolves to respond to the new positions). The paper formulates these modalities into a common framework of (interacting) particle models. Based on this insight, the paper presents two practical algorithms, showing that either the discriminator or the generator can be removed and replaced with other diffusion-inspired mechanisms, while maintaining some GAN-like properties. The Score-GAN aims to follow the dynamics of a particle model in its training iterations by replacing the discriminator with an on-the-fly learned score that is combined with a pre-trained score of the noised data (as such, the model is not really a fully independent generative method). The Discriminator Flow drops the generator, and trains a time-varying discriminator, such that sliding initially white noise particles along its gradient brings them closer to the data.

Strengths

I am not extensively familiar with the literature in this line of work, so I don’t have strong opinions about the magnitude of the novel insights in this paper. Personally I found it quite interesting and thought provoking, and as far as I can tell, theoretically sound. The authors have put a good amount of effort into validating the theory and ensuing algorithms in practice. While the results are not high-quality by modern standards, the two models are somewhat intriguing, and could inspire further research. The combination of scores and reintroduced freedom for the network to decide on the paths, shape of the latent-to-image mapping, etc., might open up some interesting avenues, such as the speculated distillation use case. The paper is polished and pleasant enough to read. However, there might be some miscellaneous errors in how the formulas are written (see Questions below), so going through to double check the rest of them once more could be helpful.

Weaknesses

As acknowledged by the authors as well, the models are not really contenders for the state of the art at the moment. It’s possible (but by no means clear) that they could be significantly improved by a deep dive into engineering the design choices, but at the moment they mostly serve as a reasonable basic validation of the theory. The Score-GAN approach seems to be limited in that it requires a pre-trained score function for the data. This seems to rule out the most common usage scenario of straight-out generation. Of course, one could argue that training that model is the first stage of the method, but at that point one could use it in standard diffusion. The actual algorithm seems to then essentially be distillation into a single-step method.

Questions

How do latent interpolations (in z-space) with the proposed algorithms behave? Do you find a similar smooth morphing behavior as with standard GANs? Algorithm 1 is, ultimately, a somewhat loose implementation of the theory: the estimated score is not trained to completion, expectation is evaluated stochastically, etc. I assume you also use something like Adam for training the generator with the update on Line 5 (?) Do you have any insight on whether the theory can accommodate some of these practicalities, or does it come down to intuition and empirics in the end? Are things like mode collapse a consequence of inexact adherence to theory? Are these typos or such? - Eq. 14, should it be z’ in a couple of places instead of z? - Algorithm 1, you have x ~ p_data but then “overwrite” it with x = g(z), in two places – if I understood correctly, you shouldn’t be sampling from data distribution at all here? - Algorithm 1, line 5, again confusing wrt. line 4: x or x^sigma isn’t even used, alternatively it seems like you should instead have a noise addition on line 5 (like in appendix) - Algorithm 2 and Eq. 20, are there some gradient symbols missing around f? Isn’t it the gradient of c(f()) that you follow?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper acknowledges reasonably well various shortcomings of the methods, and the fact that the results are far from state of the art. There is a discussion of the broader impacts in the appendix.

Summary

The authors interpret GANs through the lens of particle optimization and show the links to score-based diffusion models. Based on these theoretical results, they propose to either 1) do particle learning (without a generator) based on the flow of a discriminator, or 2) learn a generator by minimizing the score of the generator from the real data score.

Strengths

They show interesting links between score-matching and particle-based GANs. They test some interesting ideas. They try different solvers for the particle-based GANs.

Weaknesses

Particle GANs are not very novel; people don't use them frequently because its hard to optimize in high-dimensional space; this is not really new, but as far as I know, its rarely done, so this could still be interesting if it was really investigated deeply which they do not do (e.g., they don't verify the effect of different initializations, trying different optimizers, etc.). I know many papers study post-training flow/Langevin methods (e.g., Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling), but this is slightly different from particle optimization. The score-based GANs might be novel, but I am not 100% certain. Results are much worse for the disc-flow method compared to regular GANs and Diffusion and also the best GAN-Hybrid methods (e.g., Tackling the Generative Learning Trilemma with Denoising Diffusion GANs). It is well known that optimizing directly in the particle-space is difficult in high dimensions (I sadly do not remember those prior works, but I remember reading this), so this is not surprising. Results are much worse for the ScoreGAN method compared to regular GANs and Diffusion. Results are very limited, with only MNIST and CelebA and 2 baseline methods. Maybe the proposed approaches are worse because the architectures are not that good, or maybe minimizing the score-function loss is not enough; more in-depth analyses using different optimizers, initializations for the particles, and architectures on slightly bigger datasets would add a lot. In the related work, I would consider adding papers that combine GANs with diffusion or score-based models. (These are some I could come up with, but a few more exist) Diffusion non-score-based: - Diffusion-GAN: Training GANs with Diffusion (2022) - Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2022) Score-based: - Adversarial score matching and improved sampling for image generation (2020) Overall: Weak results, limited novelty, limited experimentation to try to make these approach perform better than their current performance (worse than every baseline methods).

Questions

line 21: 'challenge the conventional view that particle and adversarial generative models are opposed to each other': the Discriminator can be used to learn particles (learn x_fake directly); this is well known as far as I know. 'Score GANs': I would encourage trying to find another name that is a bit more explicit about your method; it's a bit too generic, and other papers have used the same name. I mentioned some things like extended analyses in the 'Weakness' section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The only limitation mentioned is that the generator-less method is slower. They don't mention many limitations. They say the results may be poor because they are too novel and have yet to be fine-tuned. I don't consider those enough.

Summary

The paper proposes a framework that unifies particle-based deep generative models, such as gradient flows and score-based diffusion models, and adversarial generative models, such as GANs, by framing generator training as a generalization of particle models. The authors suggest that a generator is an optional addition to any such generative model and show how it is possible to train a generator with score-based gradients which replace adversarial training and to train a GAN without a generator, using only the discriminator to synthesize samples. The paper presents two new models, a Score GAN and a Discriminator Flow, as proofs of concept to illustrate the viability of these ideas.

Strengths

- Overall, the paper is well-organized and well-written. - This paper investigates the two most prominent classes of generative models in the field, i.e., GANs and diffusion models, and provides a unified framework that explains their similarities. This work is valuable as it has the potential to inspire future research efforts to unify these two classes of generative models in addressing their respective problems. - This paper has a relatively strong theoretical foundation and meets the standards of the conference.

Weaknesses

- Some mathematical symbols (e.g., $\nabla_W$) require further explanation to enhance the readability of this paper for a wider range of readers. - The experiments in this paper are relatively weak, with a limited number of datasets and tests conducted. Furthermore, according to Table 3, the proposed methods (Discriminator Flow and Score GAN) fail to surpass baseline methods (EDM and GAN).

Questions

- In line 29, what does it mean that the generator has the role of a smoothing operator? - What is the time cost of training Score GAN and Discriminator Flows? Do they have a greater training overhead than Score-based Model and GAN?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

No, the authors have not adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper proposes a framework, i.e., a generalization of particle models, that unifies particle-based generative models like score-based diffusion and gradient flows with adversarial generative models like GANs, where generator training is framed as interacting Particle Models. Based on this perspective, the paper introduces two new hybrid models Score GANs (using score model to train generator) and Discriminator Flows (as prooof of concept. Experiments on MNIST and CelebA confirm the viability of the new framework.

Strengths

+ The findings from mathematical derivation seem to be interesting. + The writings are mostly clear and sound, well-supported by mathematical derivations and experiments.

Weaknesses

+ The claim that "Discriminator Flows have the expected advantage of converging faster to the target distribution as compared to the state-of-the-art diffusion model EDM, and thus have higher efficiency during inference." may be overstated, as s Figure 5 in the Appendix demonstrates that EDM (Heun solver) exhibits comparable efficiency. Which solver is employed for EDM in Figure 1?

Questions

+ Why both $x \sim p_{data}$ and $x = g_θ(z)$ exists in Algorithm 1?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors talked about the limitations of models proposed, and I believe these models are more of proof of concept. The authors also discussed social impact in Appendix.

Summary

This work presents a unified generative particle model framework that explains generative adversarial networks (GANs) and score-based diffusion models. This model is characterized by an evolution equation with a functional ${h_{\rho}}_t$ that captures the transitioning over the time index $t$. This work explains that the generation/inference of score-based diffusion models and the training of generators for GANs follow this model. These particle models allow the decoupling of gradient flow in the score-based models and the generator in the GANs. This led to the two new models: (1). Scored-based GANs with generators that trained with score-based gradients instead of using adversarial learning. (2). GANs that train without a generator. Only the discriminator synthesizes samples.

Strengths

The paper is well-organized. It also cited relevant works in a way that makes it easy to understand the main point. The work is also important because it offers a new perspective into the workings of two main classes of generative AI models. This opens the door to understanding and ultimately making beneficial adjustments to these models.

Weaknesses

Some of the underlying assumptions and claims are not clear. The particle model for score-based models represents the generation/inference of the models. The interacting particle model for generator-based models is the training of the generator. It is not clear if the functional $h_{\rho}$ in both cases are interchangeable. Also, it is not clear how you arrived at Finding 6. Please provide more details on the equivalence of the interacting particle model of GANs and the diffusion models.

Questions

(1). The following phrase is not clear: Line 51 on Page 2: "In prior work, ...". It is not clear if this is referring to a specific work or prior works. (2). It is not obvious that the modeling of the generation/inference with score-based models is interchangeable with training generators. Please can you provide more information on this? (3). Please provide more details on the equivalence of the interacting particle model of GANs and the diffusion models as stated in Finding 6. (4). Is it possible to come up with a hybrid structure that combines the two main techniques under the unified PM model? If yes, do you think this could improve the performance?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The author(s) addressed the limitation of their work.

Further Elements of Discussion for Reviewer 6P1y

The small character limit prevented us from completely answering the reviewer's many and detailed questions. We would like to offer additional elements of discussion in this message to start the discussion phase. ## Training of Score GANs The reviewer's interpretation is correct. This score network depends on the chosen noise level $\sigma$, like in EDM, but that we sample at each training step $t$ instead of scheduling it, unlike EDM. Indeed, $g_{\theta}(z)$ in Algorithm 1 corresponds to $g_{\theta_t}(z)$. The entirety of Algorithm 1 describes a single training iteration $t$. There are K steps of score updates per generator update, similarly to discriminators in GANs. Accordingly, $K$ is an important parameter in Score GANs. First of all, like in GANs, the tuning of $K$ heavily depends on the ratio $r = \frac{\lambda}{\eta}$ between the learning rates of the score network and the generator (Jelassi et al., 2022). A higher ratio may allow us to decrease the necessary number of steps $K$. In our experiments on image data, we use $r \geq 2$ (Appendix, Table 7). However, to gain more intuition empirically, we performed a set of experiments on MNIST with $\lambda$ s.t. $r = 1$ by making the number of steps $K$ vary from $1$ to $10$. We obtained similar results across this range of values for $K$, close to the values reported in Table 3. This indicates that even low values of $K$ can provide a sufficient approximation of the score of the generated distribution. To observe this qualitatively, we plotted in the document attached to our global response, Figure 3, the evolution of the score training loss in-between generator updates for $K=100$. At the beginning of training, $s_{\phi}^{\rho}$ needs around 20 updates to converge. However, after a low number of generator updates, it is already close to the optimum before its first update. This confirms that the continuous update of the score of the generated distribution $s_{\phi}^{\rho}$, like a discriminator, coupled with a learning rate ratio $r > 1$, makes a low number of score updates $K$ between generator updates sufficient. S. Jelassi et al. Dissecting adaptive methods in GANs. arXiv, 2022. ## Further Information on Discriminator Flows For time embeddings, we followed the implementation of EDM (Karras et al., 2023), which embeds time / noise levels using either Fourier random features (Vaswani et al., 2017) or positional encodings (Tancik et al., 2020). It is possible to train and infer the model using a second-order solver to solve Eq. (20) (step 2 of Algorithm 2). We experimented with this during the rebuttal period but did not observe a significant enough improvement in terms of NFE. Furthermore, we draw the reviewer's attention to our global response, where we explain that we must retract our claim that Discriminator Flows are more efficient than first-order EDM. A. Vaswani et al. Attention is all you need. NIPS 2017.\ M. Tancik et al. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS 2020 ## Discretization Studying how discretizing the considered continuous phenomena could affect our formulations is an interesting perspective for future work. We initiate a discussion on this topic below. Choosing the best discretization method for diffusion models is challenging (Karras et al., 2023): it depends on the chosen $\sigma(t)$, its efficiency is assessed w.r.t. NFE instead of the number of discretization steps like in numerical methods, and the final purpose of discretization (generating realistic data) differs from its initial one (approximating a solution to an SDE). Hence, standard approaches like EDM rely on empirical discretization grids and custom solvers, tailored to the generation task. Our framework, by identifying the true probability flow PDE of Eq. (9), may help diffusion models cope with discretization errors through the score of the generated distribution (as explained in line 109). The previous discussion, however, only holds for score-based diffusion models, for which the probability path is known in advance. For other particle models, this is not possible, and studying the convergence properties of their discretizations, like Arbel et al. (2019) do for MMD, is non-trivial. Adding a generator, as suggested, is an alternative way to solve the underlying particle model PDE. By generalizing the parallel between Wasserstein and Stein gradient flows in Section 3.3, generators can be seen in our framework as a preconditioning over the particle model PDE via the linear operator $\mathcal{A}_{\theta_t}(z)$ in Eq. (13). A well-chosen architecture, adapted to the particle flow $\nabla h$, may speedup and simplify the dynamics towards the data distribution. ## Typos We will correct typos in the next version. The first part of Eq. (14) should indeed read as: $[\mathcal{A}\_{\theta\_t}(z)] (V) \triangleq \mathbb{E}\_{z' \sim p\_z} \left[k\_{g\_{\theta\_t}}(z, z') V(g\_{\theta\_t}(z'))\right]$.

Response to Rebuttal.

Thank you for responding to my questions. Your response clarifies most of the issues I raised. Consequently, I will be upgrading my rating.

Response to rebuttal

Thank you for the thoughtful responses. I would would be happy with accepting the paper.