PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Accelerator

Summary

This paper proposes piecewise rectified flow (PeRFlow) for accelerating pre-trained diffusion models. To overcome the requirement of synthetic data generation in rectified flow, the authors propose to prepare the training data by dividing the entire ODE trajectory into multiple time windows. The sampling trajectories within each time window are then straightened by the reflow operation. The proposed method is adapted to multiple diffusion models with different parameterizations. Experiments on text-to-image (SD-v1.5, SD-v2.1, SDXL) and text-to-video (AnimateDiff) models demonstrate the effectiveness of the proposed method.

Strengths

* This paper addresses a major performance bottleneck in rectified flow, namely the synthetic data generation stage, which requires costly simulation with higher numerical errors. The proposed solution allows online simulation of ODE trajectories and thus more efficient training. * The proposed PeRFlow is extensively tested on multiple diffusion models for text-to-image and text-to-video generation, and the comparative results with the previous state-of-the-art few-step diffusion baselines are impressive. * Code is provided for both training and inference.

Weaknesses

* The contribution of this work is weakened by its similarity to Sequential Reflow [1], and the additional design to be compatible with different parameterization strategies is somewhat incremental. * The main motivation to improve training efficiency (line 47) is not reflected in the experiments. The authors should provide a more comprehensive comparison with rectified flow in terms of performance/training computation tradeoff. * The statement in line 55 that PeRFlow "has a lower numerical error than integrating the entire trajectories" should be more carefully validated, e.g. by quantitatively comparing straightness [2] or curvature [3] within each time window. The authors could also apply their method to the commonly used 2D checkerboard data to provide a more intuitive visualization of the learned probability path. * There is a lack of ablation studies for several design choices. The authors should analyze the sensitivity of PeRFlow's performance to the number of time windows and sampling steps (more densely). It is also unclear how adding "one extra step in $[t\_K,t\_{K-1}]$​" (line 223) contributes to the final results. --- 1. Yoon, et al. Sequential Flow Straightening for Generative Modeling. arXiv 2024. 2. Liu, et al. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ICLR 2023. 3. Lee, et al. Minimizing Trajectory Curvature of ODE-based Generative Models. ICML 2023.

Questions

* The proposed method seems to be applicable to model training in addition to acceleration. Have the authors considered training their models from scratch on CIFAR-10 or ImageNet? This would allow a direct comparison of the performance/efficiency tradeoff with a broader family of flow matching algorithms. * What does "one-step" mean in Figure 8 when the sampling step should be lower bounded by the number of time windows of 4?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes. The authors have discussed their limitations in Section 5.

Summary

This paper presents a novel approach to accelerating diffusion models by introducing the Piecewise Rectified Flow (PeRFlow). This method significantly enhances the efficiency of generating high-quality generative samples by dividing the flow trajectories of diffusion models into several time windows and straightening them using a reflow operation. Key contributions of the paper include: - Superior Performance in Few-Step Generation: PeRFlow reduces the number of inference steps required while maintaining or improving the quality of generative samples. - Fast Training and Transfer Ability: The models adapt quickly due to inherited parameters from pre-trained diffusion models, demonstrating good transferability across different models. - Universal Plug-and-Play Capability: PeRFlow models serve as accelerators compatible with various pre-trained diffusion models, facilitating seamless integration into existing workflows.

Strengths

Overall I find that the writing is clear, concise, and well-structured, making it easy for readers to follow the arguments and understand the key points. I like the idea of multi-step or piecewise generative models since it is natural to extend InstaFlow into a multi-step fashion, which offers flexibility between speed and quality.

Weaknesses

- I think the multi-step consistency model [1] should be discussed since it has a strong correlation with this paper. In the experiments section, you only compare PeRFlow with LCM and InstaFlow, both of which are relatively early works. There are plenty of distillation methods in this field that are worth mentioning and comparing, including HyperSD [2], CTM [3], and DMD [4]. - The most important hyper-parameter N, i.e., the number of segments, lacks analysis. How do you choose its value? What’s the relationship between the number of segments used in training and the number of sampling steps used in inference? - I like the idea of “plug-and-play” accelerator by extracting the delta weight to speed up other diffusion models. However, the implementation details and analysis in the paper are really limited with just a few demos. Besides, I think this is a general method that can be applied to any accelerated diffusion model, such as LCM? - The paper claims that “the computational cost is significantly reduced for each training iteration compared to InstaFlow”. However, do you have any quantitative evaluation, including the comparison with other methods? [1] Heek, Jonathan, Emiel Hoogeboom, and Tim Salimans. "Multistep consistency models." ICML 2024. [2] Ren, Yuxi, et al. "Hyper-sd: Trajectory segmented consistency model for efficient image synthesis." *arXiv preprint arXiv:2404.13686* (2024). [3] Kim, Dongjun, et al. "Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion.” NeuIPS 2023. [4] Yin, Tianwei, et al. "One-step diffusion with distribution matching distillation.” CVPR 2024.

Questions

See the weakness above.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

In this paper, the author proposes a new paradigm of sampling process (Piecewise Rectified Flow-PeRFlow) with reflow operation in the diffusion model, straightening the trajectories of the origin PF-ODEs and achieving a better performance in a few-step generation. Specifically, the PeRFlow divides the sampling process (ODE trajectories) into multiple time windows, then does reflow operation in each single time window to straighten the trajectories in each time window. Compared to the original diffusion model with reflow operation, it significantly reduced the synthesis time of training data for reflow, also narrowing the numerical errors of solving ODEs when generating the training data to get a higher-quality generated training dataset. Also, it only requires several inference steps to solve the ending point in each time window, achieving a diffusion model acceleration method with faster training convergence, more linear trajectories, better performance.

Strengths

* The paper is well organized and clearly structured, it is very easy to follow. * Many fully detailed mathematical formulas are derived, making it easier to understand the details of the proposed method. * The figure about the proposed method is well designed, the effect and rough structure of the proposed method can be understood at a glance without looking at the text description. * The paper used enough large dataset of images which contain rich images & texts and used enough SOTA acceleration methods to evaluate the proposed method.

Weaknesses

* The evaluation metrics for most generative models include FID and IS. And this paper only adopts the FID as the evaluation metric. Although this paper is aimed to accelerate the diffusion model with better performance, it is better if author can evaluate the diversity of generated images of the proposed method using the IS. In this case, people can know if such method will affect the diversity of generated images. * It would be better if author can directly indicate on the table that lower FID values are better. People who are not in the generative model filed are not familiar with FID.

Questions

Please refer to the weakness part.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

The paper introduces a new flow-based method designed to accelerate diffusion models by dividing the sampling process into several time windows. The sampling path within each time window is straightened by the reflow operator. This approach allows for fast training convergence and, transferability, compatibility with various pretrained diffusion model workflows.

Strengths

- The approach’s motivation is clear. Theoretical arguments support the proposal well. - The empirical results are promising, and better than other existing baselines.

Weaknesses

- When dividing the sampling process into several time windows, the error of the previous windows immensely affects the later ones, potentially increasing the cumulative error of the whole sampling process.

Questions

- Can PeRFlow's approach be generalized to other types of generative models beyond diffusion models such as GAN-based or VAE-based? - Can you provide more detailed insights into the parameterization techniques used and their impact on the training convergence and final model performance? - What are the observed benefits of using synchronized versus fixed CFG modes?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A