Phased Consistency Models

Thank you for your response!

Thank you for your response! Could you specify `which parts are not resolved properly`? We are willing for further reply and discussion. For the data, we already showed that our model without applying any adversarial loss consistently surpassing the CTM w/ GAN on CIFAR through multistep refinement. Additionally, we show that our model trained with only 3.5% training budget of the official CM achieved even superior performance.

We are willing for further discussion!

Besides, when comparing each baseline for fair, the principle design of CTM without GAN loss is even inferior to naive CM. The core difference of CM, CTM and PCM are listed as the following table. | Methods | timesteps sampled | adjacent timesteps | target timesteps | | ------- | ------------------------------------------------------ | ------------------ | --------------------------------------------------- | | CM | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$ | $t_{n}$ | $s:=t_1:=0$ | | CTM | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$ | $t_{n}$ | $s\sim\textrm{Uniform}(\{t_1, t_2, \dots, t_{n}\})$ | | PCM | $t_{n+1}\sim\textrm{Uniform}(\{s_m, \dots, s_{m+1}\})$ | $t_{n}$ | $s := s_{m}$ | In our paper, we have provided `thorough analysis and discussion with CTM`. The random choice design of target timestep make the design of CTM : - more complex (additionally target timestep embedding) - need more capacity ($\mathcal O (N^2)$ learning objectives) - and harder to train (inferior performance without GAN loss) Please let us know which part do you think is not convincing enough, we are willing for further discussion!

We are willing for further discussion! CTM metrics needs to generate much more samples than compared methods, which is unfair to compare directly.

**The main metrics listed in the Table 1 of original CTM paper is augmented with more advanced sampling techniques while all the compared methods do not use them**. CTM applies a pretrained image classification model to score the generated samples, and only retain the sample with higher confidence and regenerate new samples if the confidence of the classification model is low. That means, `It needs to generate much more samples than all the compared methods`. It is noting that FID will generally become smaller when the number of samples become larger.

We are willing for further discussion! We validate the design of PCM on much more baselines than all compared papers!

Except for the validation on CIFAR and ImageNet. We hope to emphasize that we validate the design of PCM on much more baselines for all compared methods. PCM (Ours): 1. `CIFAR-10`, 2. `ImageNet`, 3. `SD-v15`, 4. `SDXL`, 5.`Text-to-Video` Instaflow: SD-v15 SDXL-Lightning: SDXL CTM: CIFAR, ImageNet CM: CIFAR, ImageNet SDXL-Turbo: SDXL LCM: SD v1-5, SDXL AnimateLCM: Text-to-Video We sincerely hope any further chance for discussion! Looking forward to your reply!

A suspicion of potential plagiarism.

I respectfully request that the other reviewers and the Area Chair take note of the potential plagiarism concerns regarding this paper. I suspect that this paper may have plagiarized the work of [1] Zheng, Jianbin, et al. "Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping." arXiv preprint arXiv:2402.19159 (2024). https://arxiv.org/abs/2402.19159v2 In accordance with Harvard’s Plagiarism Policy(https://usingsources.fas.harvard.edu/what-constitutes-plagiarism-0), this paper may be engaging in ***Inadequate Paraphrase*** and ***Uncited Paraphrasing***. The reasons for my concerns are as follows: 1. In the manuscript submitted by the author, from lines L134 to L161, the author asserts that their proposed parameterization is one of their core contributions. However, the equations presented, specifically Eq. 2 and 3, are identical to Eqs. 19 and 20 in [1]. Moreover, [1] was the first to apply Eq. 19 to the task of latent consistency distillation in text to image diffusion models. The description L134 to L161 also significantly overlaps with the parameterization discussion in Sec 4.2 of [1]. 2. The objective function detailed in lines L62 to L178 of the manuscript are consistent with Eq. 29 in [1], and the content and explanation in Sec 4.2 “training” of [1] closely resemble the author’s manuscript. 3. The author frequently references [2] CTM, which conducted extensive experiments on CIFAR and ImageNet datasets during the experimental phase. However, the author’s manuscript does not mention these tasks in experiment at all. In their rebuttal, the author mentioned lacking sufficient computational resources to conduct experiments related to [2] CTM, which is particularly confusing to me. Furthermore, the tasks in the author’s manuscript are identical to those in [1]. And obviously, given the knowledge of [1], the innovation of this paper is significantly weakened both in task and experimental results. ***I am not able to make a definitive judgment at this stage. However, based on the current evidence, I believe this paper may be suspected of plagiarism.*** ------ Here we paste the definition of Uncited Paraphrasing and Inadequate Paraphrase here: 1. Inadequate Paraphrase: When you paraphrase, your task is to distill the source's ideas in your own words. It's not enough to change a few words here and there and leave the rest; instead, you must completely restate the ideas in the passage in your own words. If your own language is too close to the original, then you are plagiarizing, even if you do provide a citation. In order to make sure that you are using your own words, it's a good idea to put away the source material while you write your paraphrase of it. This way, you will force yourself to distill the point you think the author is making and articulate it in a new way. Once you have done this, you should look back at the original and make sure that you have represented the source’s ideas accurately and that you have not used the same words or sentence structure. If you do want to use some of the author's words for emphasis or clarity, you must put those words in quotation marks and provide a citation. 2. Uncited Paraphrasing: When you use your own language to describe someone else's idea, that idea still belongs to the author of the original material. Therefore, it's not enough to paraphrase the source material responsibly; you also need to cite the source, even if you have changed the wording significantly. As with quoting, when you paraphrase you are offering your reader a glimpse of someone else's work on your chosen topic, and you should also provide enough information for your reader to trace that work back to its original form. The rule of thumb here is simple: Whenever you use ideas that you did not think up yourself, you need to give credit to the source in which you found them, whether you quote directly from that material or provide a responsible paraphrase. [1] Zheng, Jianbin, et al. "Trajectory consistency distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping" arXiv preprint arXiv:2402.19159 (2024). https://arxiv.org/abs/2402.19159v2 [2] Kim, Dongjun, et al. "Consistency trajectory models: Learning probability flow ode trajectory of diffusion." arXiv preprint arXiv:2310.02279 (2023). https://arxiv.org/abs/2310.02279v3

Thank you for your further response!

Thank you for your reply! We understand your concern on the paper: Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping (TCD) [1]. **Parameterization** The parameterization of PCM was induced from the analysis form of the exact solution provided in the paper DPM-Solver [2]. All the characters used in our paper and first-order approximation proof follow DPM-Solver (including Eq. 2 - 3). We carefully checked the paper TCD and found it provided a similar discussion from DPM-Solver. It appeared on the arXiv earlier. We are willing to add citations of TCD in the relevant discussion. Both PCM and TCD cite DPM-Solver. Additionally, we hope to clarify that's just a initially basic part for our further discussion. Our further discussion on the parameterization including the analysis of $\boldsymbol \epsilon_{\theta} (\mathbf x_t, t) = \frac{\int_{\lambda_t}^{\lambda_s}e^{-\lambda} \boldsymbol \epsilon_{\boldsymbol \phi}(\mathbf x_{t_{\lambda}(\lambda)}, t_{\lambda}(\lambda))\mathrm d \lambda}{\int_{\lambda_t}^{\lambda_s} e^{-\lambda} \mathrm d\lambda} $ and relevant proofs including, `error bound`, `CFG`, `adversarial loss`, have no intersections with TCD. **Training objective** The objective design of PCM is `totally different` from TCD. For equation 29 in TCD, please note that the loss of TCD accepts an additional timestep `$t_m$` for training, which is equivalent to the `$s$` in the third row of following table. That is, they have to `randomly choose` the target timestep. While for PCM, we fixed the target timestep (i.e., $s:=s_{m}$) before training and `set them to pre-defined`. `This is exactly the core difference between PCM and TCD, CTM [3]`. Considering your concern, we are also willing to run additional comparison experiments with the official TCD and have a specific discussion on it in our paper. We list the loss design comparison in the following table | Methods | timesteps sampled | adjacent timesteps | target timesteps | | ------- | ------------------------------------------------------------ | ------------------ | ------------------------------------------------------------ | | CM | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$ | $t_{n}$ | $s:=t_1:=0$ (`Fixed`) | | CTM | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$ | $t_{n}$ | $s\sim\textrm{Uniform}(\{t_1, t_2, \dots, t_{n}\})$ (`Random`) | | TCD | $t_{n+1}\sim\textrm{Uniform}(\{t_1, t_2, \dots\})$ (Sampled from all possible timesteps) | $t_{n}$ | $s\sim\textrm{Uniform}(\{t_1, t_2, \dots, t_{n}\})$ (`Random`) | | PCM | $t_{n+1}\sim\textrm{Uniform}(\{s_m, \dots, s_{m+1}\})$ (Sampled within pre-defined phases) | $t_{n}$ | $s := s_{m}$ (`Fixed`, no target timestep embedding) | From a more high-level perspective, as the sentence "enabling `seamless transitions at any point` along the trajectory governed by the PF ODE" stated in TCD, CTM and TCD all try to learn arbitrary pairs. While our method PCM `only learns transition within each sub-phase`, which not only easy to train but also does not require any target timestep embeddings. **New Experiments** Our paper is originally for high-resolution text-conditioned image and video generation, where we try to tackle the limitations of LCM in text-to-image generation and text-to-video generation, as stated in our paper. We did not test on the CIFAR and ImageNet since they are for unconditional or class-conditional low-resolution generation. For your request, we have tried our best to run as many experiments as possible on the CIFAR and ImageNet. Following the official CM, the ImageNet experiments require training on 64 A100 GPUs for a week, which we can not afford in the limited rebuttal period. [1] Trajectory consistency distillation [2] Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps [3] Consistency trajectory models: Learning probability flow ode trajectory of diffusion

Sincerely thank you for raising the score!

We are grateful for your decision to raise the score. Your recognition and feedback mean a lot to us. Please feel free to reach out if there’s anything else we can assist with. Best regards, The Authors

Sincerely thank you for the very positive review!

Thank you for maintaining the very positive score. We sincerely appreciate the time and effort you invested in reviewing our paper. Your feedback has been exceptionally valuable, and we are deeply grateful for your support. We are always here should you have any further questions or need additional information. Best regards, The Authors