Adversarial Schrödinger Bridge Matching

Summary

This paper proposes Discrete-time iterative Markovian Fitting, which is an efficient alternative that greatly reduces NFE from IMF (Shi et al. 2023). This was possible due to the tractability of Brownian bridges for any subintervals, so discretization of IMF does not hurt the performance of if the model can be trained on it. For training a generative model, the authors adopted a time-dependent GAN called Denoising Diffusion Generative Adversarial Models (DD-GAN). Compared to DSBM, the proposed ASBM shows significantly low computational costs with good FID scores.

Strengths

Overall, I think this paper is well written and quality of presenting idea and theoretical statements are very good. This work also contains certain contributions for probabilistic generative models. - This work broadens our understanding of IMF, reference measures, and the application of Brownian bridges. - The results verified that GANs hold as a good nonlinear probabilistic generative model training method. - Theoretical statements were clear and completely proved.

Weaknesses

- I strongly believe the performance gains are mostly from applying GAN architecture, since D-IMF and IMF indicates essentially the same learning scheme. Therefore, it is indeed an improvement from DSBM, but novelty of this work as a distinct SB study might not be significant from the perspective of diffusion models. - This model does not always yield state-of-the-art scores for syntactic datasets (see Table 1&2). - While I understand this work for some extent, I do not clearly understand the motivation why do we need a sampling-based approach when we have sampling-free method such as SF^2M and LightSB [1] models Typo - L213: Denoising Denoising GANs [1] @inproceedings{ korotin2024light, title={Light Schr\"odinger Bridge}, author={Alexander Korotin and Nikita Gushchin and Evgeny Burnaev}, booktitle={The Twelfth International Conference on Learning Representations}, year={2024}, url={https://openreview.net/forum?id=WhZoCLRWYJ} }

Questions

How do the authors suspect that the FID score is very low for ASBM? Following the arguments of [1], could the authors report CMMD scores or other metrics for results in Fig. 3? [1] https://arxiv.org/pdf/2401.09603

Rating

6

Confidence

1

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations

Summary

This paper proposes the Discrete Iterative Markovian Fitting (D-IMF) method. Specifically, this work introduces discretized reciprocal properties and Markovian processes, showing that the optimal plan matches the solution of the static Schrödinger Bridge (SB) problem. Additionally, this work demonstrate that the D-IMF procedure theoretically converges to the (static) SB solution. For the implementation of the Markovian projection, the authors employ a GAN structure (DD-GAN).

Strengths

- The paper is theoretically fruitful. This work extends the reciprocal and Markovian properties into discrete sense, and also derive that the D-IMF method converges to static SB solution. This theoretical result is non-trivial and theoretically significant. - Integrating these concepts into an adversarial framework is an innovative idea, adding value to the research.

Weaknesses

- The algorithm seems to be ineffective. It requires two discriminator and two generator networks, which seems to require significant computational resource as well as significant computational time. It would be beneficial to compare the computational burden (both training time and gpu occupancy) with that of DSBM. Moreover, since it employs two adversarial networks, it would be hard to stabilize. It would be valuable to show its (empirical sense) stability through various ablation studies. - The paper lacks of empirical studies. The practical experiment is only conducted on Male-Female data. Moreover, the paper lacks ablation studies. For example, the important hyperparameter such as the initial distribution setup (simple distribution produce vs minibatch matching), the number of function evaluations (NFE, DDGAN steps), and the number of phase (K) are not explored. - The FID for Male-to-Female translation is reported, but not for the reverse direction. Since bidirectional learning is being conducted, it would be valuable to report results for both directions. Moreover, there is no metric provided to measure if the images are transformed into ones that are perceptually close. It would be useful to compare c-FID or Wasserstein distance with DSBM. Minor comments: There is a typo in Line 224: the term $q(x_{t_n}|x_{t_{n-1}})$.

Questions

- Can the authors provide a comparison of the computational burden between your method and DSBM, particularly in terms of time and resource usage? - Is the time discretization handled as like DDPM style or Uniformly? - Did you use any pretrained model (e.g. pretrained DSBM) on I2I task?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation is adequately discussed in the paper.

Summary

The paper extends the recently proposed Schrödinger Bridge method, DSBM or IMF, to a discrete-time setup. This extension is non-trivial, requiring appropriate notation for discrete reciprocal and Markovian projections. By leveraging the structure of Brownian bridges used in the original IMF, the paper achieves efficient sampling and inference processes. The discrete-time formulation also establishes a natural connection to other generative model classes, such as GANs. Experiments were conducted on low-dimensional Gaussians and unsupervised image-to-image translation.

Strengths

- The discrete-time construction of IMF is an elegant piece derived from the DSBM, which heavily relies on Brownian bridges. The analytic solutions for Gaussians may also be of interest to readers from other domains. - The connection to DD-GANs is both straightforward and clever. It's exciting to see how different classes of generative models are starting to overlap and combine, leveraging the strengths of each approach.

Weaknesses

- I enjoyed reading the majority of the paper—except, perhaps oddly, the paper title. Over 85% of the main technical contributions (Sec 3) stem from the construction of the discrete IMF formulation. The concept of "adversarial" appears only as one (of many plausible) implementation of Eq (15). I feel like the "discrete" aspect isn't highlighted enough, or at all, in the title. This is, of course, just a personal comment rather than a criticism. - In L220, the non-saturating GAN loss should be provided in the main paper rather than being postponed to the Appendix.

Questions

- The notions of $T$ and $M$ are slightly confusing. For example, In L73 and L76, both represent Markovian processes. Is there a typo?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations were addressed in Appendix A.

Summary

In this paper, the authors study the Schrodinger Bridge problem and propose an improvement over the Diffusion Schrodinger Bridge Matching (DSBM) methodology [1, 2]. In particular, they propose a discrete counterpart formulation of DSBM. Instead of relying on stochastic processes, they rely on a discrete-time version of the Markovian and reciprocal projections which are the key components of DSBM. By doing so, they have a better control on the number of steps needed in DSBM. They next leverage advances in the field of diffusion models to reduce the number of stepsizes needed in order to solve the Schrodinger Bridge. In particular, they focus on larger jumps using the approach of Denoising Diffusion GAN (DD-GAN), introduced in [3]. The method is illustrated on translation tasks on the Celeba dataset. [1] Shi et al. (2023) -- Diffusion Schrodinger Bridge Matching [2] Peluchetti (2023) -- Diffusion Bridge Mixture Transports, Schrödinger Bridge Problems and Generative Modeling [2] Xiao et al. (2021) -- Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Strengths

* The paper is overall clearly written and easy to follow. It relies on the notions of Markovian and reciprocal projections that were already introduced in [1]. All the required notions are properly introduced. * I enjoy reading the full description of the DSBM algorithm in discrete time. I agree that this should allow for 1) easier spreading of DSBM-related methodology 2) more flexible algorithm (like the introduction of GAN jumps as proposed). * The experimental results seem to compare favorably compared to DSBM (more on that later). [1] Shi et al. (2023) -- Diffusion Schrodinger Bridge Matching

Weaknesses

* Part of the theory was already done in [1]. Especially the discrete Markovian projection was already identified as the minimizer of the Kullback-Leibler divergence in [1, Appendix E]. I think the claims of novelty of the paper regarding the introduction of the discrete scheme should be tamed down. * I am not clear regarding what is the purpose of the Gaussian study. I understand that the authors use it in their experimental section (Section 4.1) to study the convergence of their algorithm but I think Section 3.4 is distracting the reader from what I consider to be the main points of the paper which is 1) full discrete study of DSBM 2) GAN jumps. Section 3.4 could easily be considered in the Appendix which would leave more room for additional experiments. * The claim of exponential convergence should be tamed down. Reading the paper it seems like this is proven in the current paper (I am talking especially about the bold "exponential rate of convergence" in l.294 which can be misleading). To my understanding this is only experimentally shown. * There is something quite striking in Figure 3. (for example see the last row of (d) and (f)). While the quality of ASBM is better (by the way, one of the reason the FID score is so high for DSBM must be because of the remaining noise in the output samples, an easy way to remove this noise would be to consider a deterministic last step as a postprocessing as is routinely done in diffusion models, see [2] for instance), it seems that the diversity is much lower than DSBM. I would like to see a proper investigation of this phenomenon which is also reported in the adversarial distillation of diffusion models. According to [3], there is an incompressible trade-off between high-NFE/good diversity and low-NFE/low-diversity. [1] Shi et al. (2023) -- Diffusion Schrodinger Bridge Matching [2] Jolicoeur et al. (2020) -- Adversarial score matching and improved sampling for image generation [3] Dieleman (2024) -- The paradox of diffusion distillation

Questions

* I understand that the main goal of the author is the speed-up of DSBM-like algorithms. However, why choose DD-GAN? This doesn't seem to me like the obvious candidate for speeding up DSBM. Other techniques like distillation (adversarial distillation if one really wants to use GANs [1, 2]) or consistency models seem like potential better candidate. If the finetuning strategy is what the authors want to avoid, consistency models can be trained from scratch [3, 4]. * Some typos: * l.280 "Schrodigner" --> "Schrodinger" [1] Xu et al. (2023) --UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANs [2] Sauer et al. (2023) -- Adversarial Diffusion Distillation [3] Song et al. (2023) -- Consistency Models [4] Song et al. (2023) -- Improved Techniques for Training Consistency Models

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are discussed in the Appendix A. However, the authors are quick to dismiss these limitations. For example, regarding the adversarial training "While we mention this aspect, we do not treat it to a be a serious limitation." I think there are serious problems regarding adversarial training regarding the training (suffer from training instabilities) and the results themselves (less diversity). These issues are usually reported in the field of adversarial distillation of diffusion models, see [1] for instance. I think this is doing a disservice to the paper and the community by trying to minimise them instead of exposing them for what they are. [1] Dieleman (2024) -- The paradox of diffusion distillation

Thank you for the response

I appreciate authors for clarification and for the additional ablation studies. I would like to raise my score to 6.