Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT

Summary

Lumina-Next is an improved version of Lumina-T2X, featuring a core architecture that employs a Flow-based Large Diffusion Transformer (Flag-DiT). Through empirical experiments and analysis, Lumina-Next introduces an enhanced Next-Dit architecture and develops a fast sampling algorithm, which boosts the model's generalization capability and improves both training and inference efficiency.

Strengths

This paper focuses on Lumina-T2X and provides empirical analysis and discussion on the training stability, inference efficiency, and extrapolation performance of this type of Flow-based Large Diffusion Transformer. Its advantages are as follows: - This paper uses 3D RoPE to replace 1D RoPE, enabling the perception of positional information in both spatial and temporal dimensions without the need for learnable [nextline] and [nextframe] identifiers. These improvements help stabilize training and benefit the extrapolation of visual sequence context. - Sandwich normalization is applied to the model structure to prevent the accumulation of errors due to increased model depth, thereby preserving model performance. - Different extrapolation schemes is ablated, and Frequency- and Time-Awareness Scaled RoPE was proposed to improve resolution extrapolation. - A time schedule suitable for flow-based diffusion models was proposed, improving the visual quality of fast sampling produced with 10-20 NFEs. Overall, this paper addresses some of the shortcomings of Lumina-T2X and provides insights for the development of large-scale flow-based diffusion transformers.

Weaknesses

- Whether identifiers such as [nextline] and [nextframe] are unnecessary when using 3D RoPE is not sufficiently discussed in the paper. - The paper lacks a comparison with stable-diffusion-3, which is also a flow-based diffusion model. - Is the proposed Time Schedule universally applicable at low NFEs (<10)? Figure 7 contains too few examples to reflect the actual effect. Are there any numerical comparisons? Overall, this paper leans towards empirical research and analysis, making it more akin to a technical report.

Questions

see weaknesses.

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Although the paper discusses limitations in the supplementary materials, I believe it could benefit from adding some discussion on whether the training data might introduce biases related to violence, pornography, race, etc.

Summary

The paper introduces Lumina-Next, an enhanced version of the Lumina-T2X model, which is a Flow-based Large Diffusion Transformer (Flag-DiT) aimed at transforming noise into various modalities like images and videos based on text instructions. Compared with Lumina-T2X, Lumina-Next introduces the following improvements: * A redesigned architecture named Next-DiT, featuring 3D Rotary Position Embedding (RoPE) and sandwich normalization to stabilize training and inference. * Frequency- and Time-Aware Scaled RoPE for better resolution extrapolation in text-to-image generation. * An optimized time discretization schedule combined with higher-order Ordinary Differential Equation (ODE) solvers for faster and high-quality generation. The model demonstrates stronger performance in text-to-image generation, resolution extrapolation, and multilingual capabilities in a zero-shot manner. It also shows versatility across different tasks such as visual recognition, multi-views, audio, music, and point cloud generation.

Strengths

- This paper, like SD3 and PixArt, explores image generation within the DiT architecture, and the codes and models have been open-sourced, which is beneficial to the community. - Strong functionality. Without additional training, Lumina-Next can easily generate higher-resolution images and produce images of decent quality in fewer inference steps.

Weaknesses

- The paper mainly demonstrates the advantages over comparative methods in terms of functionality but does not show comparisons with these state-of-the-art methods in terms of text-image alignment and image quality. - Although the paper's method is about DIT, it does not demonstrate the manifestation and properties of scaling laws in the generation domain.

Questions

na

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

The paper introduces a new multi-modal generation model, Lumina-Next, which extends the previous Lumina-T2X approach by several innovations: i) 3D rotary position embedding (RoPE) ii) extra normalization to stablize training iii) frequency- and time- aware scaled RoPE for training free resolution extrapolation iv) improved time scheduling and higher-order ODE solvers for reverse sampling. Lumina-Next aims to achieve faster, better, and more efficient generation on multiple text-to-X tasks, e.g. text-to-image/audio/point-cloud, etc.

Strengths

Topic: the paper studies multi-modal generation using a unified framework, which would be of great interest to the community. The paper also promises to release the source code to support future research Methodology: the proposed technical extensions for Lumina-T2X are technically sound. Experiment: Lumina-Next demonstrates promising qualitative results on text-to-image generation, training-free resolution extrapolation, and text-to-point-cloud generation. The paper also provides quantitative evaluation on text-to-audio/music generation.

Weaknesses

The reviewer appreciates the authors’ effort on improving text-to-X generation. However, as a technical submission for NeurIPS, the paper is fully convincing as it provides limited quantitative evaluation. As a representative reader, the reviewer would expect i) quantitative evaluations on all generation tasks ii) extra apples-to-apples evaluations on any resolution recognition, as for now, there is only one baseline, DeiT-Base. The comparison is also apples-to-oranges as the models have different architectures and were trained on different data. iii) comprehensive quantitative ablations on the proposed technical extensions, a) 3D RoPE, b) extra normalization, c) frequency- and time- aware scaled RoPE, d) improved time scheduling and higher-order ODE solvers for reverse sampling, etc

Questions

Please refer to the weaknesses

Rating

4

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations have been discussed in the Appendix

Summary

This paper introduces the next generation of Lumina-T2X, Lumina-Next, which offers improved architecture, a scaled dataset, optimized sampling techniques, and a more efficient context extrapolation strategy. The improved architecture shows faster convergence rates, while the optimized sampling technique enables high-quality text-to-image generation with fewer steps. Through visual results, the authors validate the improvements provided by refining Flag-DiT in Lumina-T2X.

Strengths

* Next-DiT brings excellent high-resolution extrapolation generation. * Compared to previous work, it provides better generation quality under a few-step generation setting.

Weaknesses

* There is a lack of quantitative experiment comparison results on some Text2Image benchmarks to demonstrate the superiority and effectiveness of the proposed method. Most of the comparisons are shown through visualization.

Questions

* Text-to-image benchmark result should be evaluated and reported

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

None

Summary

Lumina-T2X encounters challenges including training instability, slow inference, and extrapolation artifacts. This paper introduces Lumina-Next, which improves Lumina-T2X with improved architecture, scaled dataset, optimized sampling techniques, and better context extrapolation strategy. On the architecture side, they replace 1D RoPE with 3D RoPE to eliminate the inappropriate positional priors to model images and videos using the attention mechanism. They further removed all learnable identifiers in Flag-DiT, and introduced the sandwich normalization block in attention modules to control the activation magnitudes. On the context extrapolation side, the authors propose a Frequency-Aware Scaled RoPE, reducing content repetition during extrapolation, and a novel Time-Aware Scaled RoPE for diffusion transformers to generate high-resolution images with global consistency and local details. On the sample techniques side, the authors propose time schedules tailored for flow models to minimize discretization errors. They further combine the optimized schedules with higher-order ODE solvers, achieving high-quality text-to-image generation samples within 5 to 10 steps. With the improved architecture, context extrapolation, and fast sampling techniques, the authors show that Lumina-Next yields strong generation capabilities, such as generating high-quality images much larger than its training resolution and multilingual text-to-image generation. Furthermore, they extend Lumina-Next’s versatility to other modalities, such as multiviews, audio, music, and point clouds, with minimal modifications, achieving superior results across these diverse applications.

Strengths

- The paper is well written with extensive results, applications and details, which is valuable to the community. - The paper provides practical improvements over architecture, dataset, sampling techniques, and context extrapolation strategy. - The rethinking of existing model design is appreciated, especially showing a much smaller network can get better performance.

Weaknesses

- the changes over Lumina-T2X are a bit incremental, considering Lumina-T2X is just released a month before the submission. Some of the designs in Lumina-T2X itself such as learnable tokens are not essential and only exist in Lumina-T2X, so the improvement (removing it) is not applicable to other DiTs, so removing extra learnable tokens itself can hardly be considered innovation. - the 3D rope does not show advantage over native designs, such as normal 2D + temporal positional embedding. There is no application showing the usage of 3D RoPE, such as video generation. - the ImageNet experiments can hardly justify the improvements. The comparison between Lumina-Next and SiT/Flag-DiT is not clear, such as whether the model sizes are the same. - there is a lack of explanation why model size can be reduced from the Flag-DiT 5B/7B model significantly. - The adoption of newer and stronger text encoders is also not ablated, so the improvement over Lumia-T2X cannot fully be attributed to the proposed changes. - There is a lack of quadratic evaluation metrics such as TIFA and ImageReward scores for fair comparison across models.

Questions

With all those changes on controlling the activation magnitudes, what about simply using grad clip norm to reduce gradient norms?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

There are more interesting contents in the appendix, which can be moved to the main paper.

Request for Feedback on Rebuttal Points

Dear Reviewers, We would like to extend our appreciation for your time and valuable comments. We are eagerly looking forward to receiving your valuable feedback and comments on the points we addressed at the end of the rebuttal. Ensuring that the rebuttal aligns with your suggestions is of utmost importance. Thank you for your dedication to the review process. Sincerely, Authors