Summary
The authors approach the reverse diffusion process in score-based generative modeling from the perspective of spontaneous symmetry breaking, which may explain why generation is minimally affected by "late initializations" (e.g. initializing x_t at t=T, but simulating the reverse process starting at some t < T). Motivated by this framework, the authors propose late-start scheme and show improved performance compared to some baselines.
Strengths
**Exciting direction of study.** The authors propose an exciting perspective on the diffusion model in terms of spontaneous symmetry breaking in physical systems.
**Initially promising empirical results.** The empirical results appear to be promising. (However, they do not presently provide much in the way of concrete proof that the symmetry breaking framework provides tangible improvements on generation speed or quality, see "Weaknesses").
Weaknesses
**Motivation.** The motivation of the work should be made more clear. If it is to improve generation speed, the model should be compared against fast generation models, such as Model Distillation, Consistency Models, and EDM. If it is to generate better samples, then the model should be compared against standard diffusion models such as [4] or [5].
**Empirical Results.** The empirical results are not very convincing. All models achieve FIDs that are far from state-of-the-art. It is not clear at this stage whether improvements come directly from the proposed approach, and whether the improvements will remain when models are tuned to be more competitive with respect to the state of the art.
**Claims.** The claim (up to 3x FID improvements) feels overstated. The authors do not compare against state-of-the-art fast samplers, such as Model Distillation [1], Consistency Models [2], and EDM [3].
**Soundness.** I have some concerns over the soundness of the claims in the paper.
- The existence of stable points is only shown for toy examples (e.g. Secs. 3.1 and 3.2). Even in these toy examples, it is not clear to me how the stable points will affect the result of the reverse diffusion process for most initial conditions of the reverse process. Showing the existence of stable points at the origin and the original data points implies only the existence of a finite set of stable points, which are essentially probability zero set of the entire space. Diffusion models are initialized from a standard normal, so x_T can be very far from the origin (or any of the stable points). So is it true that the spontaneous symmetry breaking model is relevant here, even in theory?
- While the second toy example is more realistic (Sec 3.2), and the authors claim that the assumptions required are mild, I do not believe they hold for any usual image distributions (e.g. MNIST, CIFAR10, ImageNet, CelebFaces, etc.).
- Finally, the authors connect the theory of symmetry breaking to diffusion modeling with a single piece of empirical evidence: The FID of samples drawn with late start diffusion is relatively stable (up to a critical) point in time t_c. I can see how this is necessary condition to show the existence of symmetry breaking. But how is it sufficient?
In summary, while the theory of spontaneous symmetry breaking is very interesting and thought-provoking, I am not yet convinced that it is presently a useful perspective in diffusion-based generative modeling.
**Minor edits.**
39: the theory weak and electromagnetic -> the theory of weak and electromagnetic
108: Hessian matrices (of what?)
[1] Progressive Distillation for Fast Sampling of Diffusion Models. https://arxiv.org/abs/2202.00512
[2] Consistency Models. https://arxiv.org/abs/2303.01469
[3] Elucidating the Design Space of Diffusion-Based Generative Models. https://arxiv.org/abs/2206.00364
[4] Denoising Diffusion Probabilistic Models. https://arxiv.org/abs/2006.11239
[5] Score-Based Generative Modeling through Stochastic Differential Equations.https://arxiv.org/abs/2011.13456
Questions
See "Weaknesses" section.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
See "Weaknesses" section.