Summary
The paper studies the convergence of consistency models theoretically. They provide bounds on the Wasserstein distance between target and estimated data distribution using such models for as most general assumptions as they can (specifically, for bounded density support and light tail density). They transfer this to total variation distance under smoothness assumption, giving convergence guarantee. Moreover, they also derive theoretically the benefit of an additional sampling step in the specific case of Ornstein-Uhlenbeck process. They provide advance in the theoretical understanding of empirical observations about consistency models which are (i) an additional step can significantly improve the quality of the sample and (ii) more steps only bring limited gains.
Strengths
The will of bringing theoretical claim about convergence of a method + analytical explanation of empirical observations is highly welcomed. It allows better understanding of flaws and successes of a method. Especially in the field of diffusion models, and specifically here consistency models which are SOTA methods for generative modeling. Consistency models aim to improve over classical diffusion, tackling the sampling efficiency. Bringing theory and explainability in convergence of such method is valuable.
- The paper base their claims on previous observations in the literature and try to improve in several ways.
- It is welcomed to see work that tries to provide theoretical claim with light assumptions, making them as most general as possible.
- Interpretability (in the case of multi-step sampling) is thrilling.
Weaknesses
As a general comment, I found the goal laudable. However, the paper is really hard to follow and lacks clarity in its presentation and writing. I will try to pack similar remarks in thematic groups here below.
**General confusion**
The introduced notations are overly complex and sometimes confused.
- Line 50-71 : The inline paragraphs bold titles are weird. Why not using clear separated paragraphs? (I know that you may lack space, I propose getting rid of useless repetition below).
- Score-based generative model notations are hard to follow.
- Is the switch between $P_t$ and $p_t$ needed?
- Operator $\mathcal{D}$ is not useful and can be simplified by just using the noise schedule kernel and the distribution defined line 143.
- Why $x_{t_i}$ and not just $x_i$? e.g. : line 165 where we have $x_{t_0} = x$, usually denoted $x_0$ in the literature. But on the other hand, you define a $\Delta \tau$ accounting for step between consecutive time, ... all of that can be simplified. I understood the paper but found it hard due to intertwined overly complex notations (at least understandable but making it hard to follow without making tons of back and forth in the text to check each piece). The time subscript is sometimes omitted, confusing also the reader concerning what we are talking about.
- The paper lacks structure and the reader is too often referred to other part of the paper. The authors introduce a lot of preliminaries 'useful for later', 'used in theorem x'... leading to a lot of repetition in the main text (in the abstract, the contributions, the theorem in themselves,...). It would be better to have more structured blocks. E.g. : Contributions can be shortened, more summarized, leaving the reader to the proofs in the main text.
- The consistency function introduced line 167-168 is used on distribution line 211. Of course the author precise **after** that the operation is now on distributions, but this further confuse the notations and highlight again a general confusion.
- It would be nice to introduce Wasserstein distance and total variation somewhere. Even in Appendix, it's ok. Indeed, In theorem 2 you use $W_2$ and then make a sort of connection with the KL line 187 that pops from nowhere. It lacks proper explanation.
- Lines 246-255 repeats a lot with preceding text.
**Results**
Analytical results are interesting. However they stick to theory and it's hard to identify their contribution in practice. Could you add, even for small toy problem, experiments that illustrate your findings. Indeed, you derive bounds given several assumptions. Use those for a toy example. It is crucial to link theory to practice to motivate the worth of your work. Experiments might also help identify potential yet to study problems/questions. Your results are sometimes specific (e.g. you stick to VP and VE SDE in case study 2) which is normal for theoretical derivations, please extend the scope of your work a bit. Even if you show that your conclusions do not hold at all in other setup, it might be valuable as later study can try to understand why? Or in the opposite, if your claims extend in other setup, it might raise the question of generalization of your work.
**Found typos**
I list below typos I detected, just to correct them (I might have missed some).
- Uppercase in contributions (lines 76-92).
- Space after a point line 169 **. At a high level ...**.
- Line 262: ~insteading~ -> instead of.
- Where is Theorem 1? At the beginning of page 6 you directly have Theorem 2 but there are no Theorem 1. Keep separated counting between Assumptions, Lemmas and Theorems.
- Line 367: ~forawrd~ -> forward.
- Line 368: ~nosie~ -> noise.
- Line 377: Without loss of generality preferred to not clear *W.l.o.g.*.
- Line 450: ~samling~ -> sampling.
**Suggestions**
- Remove the references to theorems in your contributions. Focus only to the key message of each of them and the rest will follow in the main text.
- line 165: remove the 'as a function of t'.
- Remove parenthesis around 'ground truth' line 166.
- Putting ':' inside a sentence often make it long and hard to follow (can be more structured). E.g. : lines 170-172
- Combine Theorem 2 and 3. Can't we see theorem 2 as a specific case of 3 when $C \to 0$?
For me, the paper is not ready yet. The main causes being the presentation and lack of illustrations.