Summary
The authors propose FlowTurbo, a method to adapt pre-trained flow-based generative models for faster sampling. The method is based on the observation that the predicted velocity field is quite stable throughout the integration time and hence at each timestamp only small refinement from the previous step is required. Therefore, the authors propose to train a lightweight refinement network and use it at certain timestamps instead of the original velocity field predictor. In addition to the refinement network the paper also suggests to speed up the Heun's integration method by reusing the velocities predicted at previous corrector steps. All the sampling blocks are compiled for further speed up. As a result, FlowTurbo demonstrates great trade-off between speed and quality compared to prior work and achieves sota results on certain tasks.
Strengths
The problem of speeding up generative models is of high importance nowadays. While the majority of the methods aims for decreasing the number of denoising steps, the authors of FlowTurbo propose to make each step significantly cheaper. This is achieved at low additional training costs and could have high impact on the methods that explicitly utilise the iterative nature of modern diffusion- or flow-based generative models (such as [1, 2, 3], see references in Weaknesses). The method is well-grounded and carefully ablated. The experiments demonstrate the effectiveness of FlowTurbo compared to default sampling.
Weaknesses
1) **Prior work**: The paper does a good job with contextualizing it relative to the diffusion-based models. However, references to many flow-based methods are missing, which include:
- Models that leverage flow matching for generative process [2, 4, 5, 6, 7, 8, 9].
- And (which is more connected to the topic of the paper) the methods that try to speed up the sampling for flow-based models [10, 11, 12, 13].
2) **Method**: Despite the good results, it seems that it is not trivial how to choose the exact configuration of the sampling blocks, i.e. the order and the number of different steps. From the first sight the configurations presented in the paper are quite random. At least some intuition on how to setup those is required. Otherwise, if the configuration needs to be tuned for every other dataset, the method is not that straight-forward to apply.
3) **Evaluation**:
- The tables would be simpler to read if NFEs (number of function evaluations) were also included as a separate column.
- Is the step of the baseline method also compiled in tables 1a and 1b? If not, it would be nice to add this experiment, in addition to the ablations, to see the effect of other parts.
[1] Watson et. al, Novel view synthesis with diffusion models, ICLR 2023
[2] Davtyan et. al, Efficient Video Prediction via Sparsely Conditioned Flow Matching, ICCV 2023
[4] Hu et. al, Motion Flow Matching for Human Motion Synthesis and Editing, arxiv 2023
[5] Fischer et. al, Boosting Latent Diffusion with Flow Matching, arxiv 2023
[6] Davtyan et. al, Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation, AAAI 2024
[7] Davtyan et. al, Enabling Visual Composition and Animation in Unsupervised Video Generation, arxiv 2024
[8] Gui et. al, Depthfm: Fast monocular depth estimation with flow matching, arxiv 2024
[9] Hu et. al, Flow Matching for Conditional Text Generation in a Few Sampling Steps, arxiv 2024
[10] Shaul et. al, Bespoke Solvers for Generative Flow Models, arxiv 2023, ICLR 2024
[11] Lee et. al, Minimizing Trajectory Curvature of ODE-based Generative Models, ICML 2023
[12] Pooladian et. al, Multisample flow matching: Straightening flows with minibatch couplings, ICML 2023.
[13] Tong et. al, Improving and generalizing flow-based generative models with minibatch optimal transport, TMLR 2023.
[14] Lipman et. al, Flow matching for generative modeling, ICLR 2023.
Questions
1) In the paper the order of blocks $H_{N_H}P_{N_P}R_{N_R}$ is fixed, and only the number of the blocks $N_H$, $N_P$ and $N_R$ changes. What is the reason behind this? Have you tried other schemes, e.g. sequentially using multiple blocks of type $H_{N_H}P_{N_P}R_{N_R}$?
2) A rather nitpicky comment: Multiple times throughout the paper it is mentioned that the linear interpolation corresponds to the optimal transport from the noise distribution to the data distribution. It is not precisely correct, as the linear interpolation corresponds to the optimal transport from the noise distribution to the distribution concentrated around a particular data point [14]. Methods that attempt to steer flow-matching towards optimal transport between noise and data involve additional tricks [12, 13].
Some typos:
1) Line 137: "interpolation between $x_0$ and $\cancel{v}$ $\epsilon$".
2) Line 170: "simulate the $\cancel{x_t}$ $x_{t_i}$".
3) Eq. 15: $\cancel{x_i \leftarrow x_{i-1}}$ $x_{t_i} \leftarrow x_{t_{i - 1}}$