FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification

Summary

The paper presents FasterDiT, a method intended to speed up the training of Diffusion Transformers (DiT) without making changes to the architecture. By utilizing insights from the Probability Density Function (PDF) of Signal-to-Noise Ratio (SNR), FasterDiT improves training strategies and the effectiveness of supervision. The paper also includes a thorough set of experiments and a new supervision approach called FasterDiT.

Strengths

The paper introduces an innovative approach to training strategies by considering the Probability Density Function (PDF) of Signal-to-Noise Ratio (SNR), aiming to improve training efficiency. It includes a thorough empirical analysis with significative experimental results, providing robust evidence to support the findings. FasterDiT achieves significant acceleration in training Diffusion Transformers with a notable increase in training speed, making it a relevant and practical solution for enhancing the efficiency of training large-scale generative models.

Weaknesses

The paper would benefit from additional experiments to demonstrate the generalizability of the proposed approach. A more comprehensive comparison with existing methods for accelerating training of Diffusion Transformers would provide a broader context for evaluating the effectiveness of FasterDiT. Additionally, the scalability of FasterDiT to larger datasets or more complex tasks is not sufficiently discussed, potentially limiting its applicability. The absence of theoretical results and proofs in the paper limits the depth of understanding of the proposed method.

Questions

No questions.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

A more extensive discussion regarding the drawbacks of the proposed approach is needed.

Summary

This work aims at solving the slow training convergence of Diffusion Transformer, from the perspective of Signal-to-Noise Ratio (SNR). Different from other works, the authors formulate the probability density function (PDF) of SNR during training, and then leverage such SNR PDF to analyze the association between training performance and robustness across some common pipelines of DiT. The finding of the trade-off between method robustness and performance makes authors to propose Fast-DiT which significantly accelerate DiT's training convergence.

Strengths

1. The motivation is very clear. DiT suffers from slow convergence and this work successfully solved this issue without any changes in model architecture. 2. The formulation of SNR PDF is novel. With this analysis tool, authors can check the robustness and performance among various DiT training pipeline. 3. The emipirical experiments are sufficient and the final results compared with SOTA are convincing for me.

Weaknesses

1. Current Table 1 compares the CFG (classifier-free guidance) results. Please compare the Class-conditional results of Faster-DiT with DiT and SiT, under the same setting in Table 1 (or even more iterations of Faster DiT). Usually we need both CFG and Class-conditional results to check the training convergence. Currently Figure 1 right with small iterations is not convincing. 2. I am still unclear how you change the std of data without changing dataset. Please specify this point.

Questions

Listed in weakness.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors has discussed the limations of the best performance of FastDiT without sufficient GPUs.

Summary

This paper propose FasterDiT, a diffusion model training method that considers the data distribution in the definition of signal-to-noise ratio. It formulates the SNR in a new framework, estimates the PDF of the SNR, and then employs it to improve the training efficiency of DiT. Experimental results show that FasterDiT achieves relative FID but with 1/7 training time.

Strengths

1. The idea of re-formulating the SNR in the diffusion model is a very interesting and novel strategy for me. I believe this paper has a good technical contribution. 2. The experimental results of DiT show both good acceleration and generation performance.

Weaknesses

1. This paper has a bad quality in writing. There are some typos and the explanation to the SNR is not clear to me. I'm confused for many questions: (1) What does the std means in Line 109. Does it mean the std of the value of pixels in the images? (2) Why can we assume std^2 approximate K(I)/std^2 as a constant C(I)? (3) Authors mention "robustness" for many times in the paper for "training robustness", "data robustness". What does it mean in detail? (4) In Figure 6, the caption inside the figure is "multi-step balance", while the caption after the figure is multiple-step balance. Please use the same description. What does multi-step balance mean? Does that mean the new training strategy with SNR? The "single-step supervision" is also not a good choice here. It should be something like "directionality of velocity" or "single-step supervision (ours)" since there has already been single-step supervision in the traditional training method. 2. Does the proposed method generalize well to the diffusion models besides DiT, such as latent diffusion models? Please discuss on this. If so, experimental results should be provided for this since only experiments on DiT & ImageNet is not very convincing.

Questions

1. Does x_* in the paper indicate the x_0 (real images)? If so, I advise to replace it with x_0 to align with previous works. 2. Section 4: Improving DiT Training. The "." should be removed. 3. The space between line 140 and 141 are excessively reduced. In summary, I think this paper may have good technical contribution. But it's really difficult for me to understand the details of this paper.

Rating

6

Confidence

3

Soundness

3

Presentation

1

Contribution

3

Limitations

n/a

Summary

The paper focuses on accelerating the training process of Diffusion Transformers (DiT) without modifying their architecture. The authors identify two primary issues: inconsistent performance of certain training strategies across different datasets, and limited effectiveness of supervision at specific timesteps. Key contributions include: 1. Extended Definition of Signal-to-Noise Ratio (SNR) 2. Extensive experiments and empirical findings for SNR PDF 3. A new supervision method

Strengths

- FasterDiT achieves competitive results (2.30 FID on ImageNet 256 resolution at 1000k iterations) while being seven times faster in training compared to traditional DiT (2.27 FID). - The paper presents a large number of experiments to empirically validate the proposed method. The paper presents various discussions and insights of SNR PDF. - By generalizing the definition of SNR and analyzing various training strategies through SNR PDFs, the paper shows promising outcomes of faster training.

Weaknesses

- The experiments were conducted on 256-resolution ImageNet only. It would be interesting to validate the proposed method on larger resolutions (such as 512, 1024, etc.). Acceleration is more critical to those scenarios. - The figures could be improved. The rendered text is not easy to read.

Questions

see weakness

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

n/a

Summary

This paper presents observations on training strategies for Diffusion Transformers (DiT) using Signal-to-Noise Ratio (SNR) Probability Density Function (PDF) analysis. Through extensive experiments, the authors derive insights into training performance and robustness. Based on these findings, they propose a method to accelerate the training process of DiT without modifying the model architecture.

Strengths

1. The problem addressed is practical, focusing on the significant computational demands of training Diffusion Transformer models. 2. The authors propose a simple yet effective technique to improve the training efficiency of Diffusion Transformer models without modifying the architecture 3. The paper is well-structured with clear motivation, methodology, and results.

Weaknesses

1. The paper lacks experiments with higher resolutions such as 512x512 or 1024x1024, which could provide insights into the method's scalability for more complex image generation tasks. 2. The hyperparameter 'std' requires tuning to find the appropriate setting, which may introduce additional complexity in implementing the method across different datasets or tasks.

Questions

Why does adding cosine similarity loss for velocity direction supervision accelerate the convergence rate of the model?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The scalability of this method is not thoroughly evaluated, including its effectiveness for higher resolutions, other datasets, or different types of generative tasks (e.g., text-to-image, video generation).

comment

Thank you for the experiments. I hope the authors will add the high-res results to the final paper.