UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peaks

Summary

The paper presents UltraPixel, an innovative architecture for ultra-high-resolution image generation that tackles semantic planning, detail synthesis, and high resource demands. UltraPixel uses cascade diffusion models to generate images at multiple resolutions within a single model, efficiently guiding high-resolution generation with lower-resolution semantics. It features implicit neural representations for continuous upsampling and scale-aware normalization layers. Moreover, it requires less than a 3% increase for high-resolution outputs, boosting efficiency.

Strengths

1、The paper demonstrates impressive results, with generated high-resolution images exhibiting remarkable detail. The proposed method outperforms existing approaches in terms of speed and flexibility, supporting arbitrary resolution image generation with a single model. This represents a significant advancement in the field. 2、The authors present a clear and well-motivated approach. They provide compelling evidence (Figures 2 and 6) to support their argument that the absence of low-resolution (LR) guidance can lead to suboptimal generation results.

Weaknesses

1、 The manuscript's layout requires some refinement. For instance, Figure 4 extends beyond the page margins, and the text adjacent to Figure 9 appears overly condensed. 2、 Given that this is a text-to-image generation work, the paper would benefit from a more comprehensive set of visual results, including additional comparisons with state-of-the-art methods.

Questions

1. The discussion in Section 4.3 regarding the timesteps of LR guidance extraction is intriguing. It would be valuable to see final generated images using different timesteps for guidance, rather than just attention visualizations. 2. The authors' use of LR guidance bears similarities to recent diffusion-based image super-resolution methods. A comparative discussion of these approaches could provide valuable context. 3. Given the method's design, it should theoretically support even higher resolutions (e.g., 5K, 6K). Have the authors explored this possibility? 4. The visual results demonstrating the limitations mentioned in the paper could be included in the supplementary materials to provide a more comprehensive understanding of the method's constraints. 5. Will the code and pre-trained models be made publicly available to facilitate reproducibility and further research in this area？

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See Weaknesses.

Summary

This paper introduces UltraPixel, a method for generating high-quality ultra-high-resolution images. It utilizes the semantics-rich representations of lower-resolution images in a later denoising stage to guide the overall generation of highly detailed high-resolution images. The method incorporates implicit neural representations for continuous up-sampling and scale-aware normalization layers that are adaptable to various resolutions. The experimental results show that it has excellent ability in generating high-resolution images of different sizes.

Strengths

1. The introduction of implicit neural representations for continuous up-sampling and scale-aware normalization layers adaptable to various resolutions is a creative solution that addresses a challenge in the scalability of image generation models. 2. The methodology is well-articulated, with a clear explanation of how the model manages to generate high-quality images while maintaining computational efficiency. 3. The ablation experiments are thoroughly conducted, systematically revealing the contribution of each component to the overall performance. 4. The paper proposes an innovative method for generating high-quality, ultra-high-resolution images efficiently, tackling a major challenge in image synthesis.

Weaknesses

1. The explanation of the implicit neural representation (INR) requires further clarity regarding its ability to enable continuous upscaling. Moreover, an in-depth analysis and dedicated ablation study of the Scale-Aware Normalization (SAN) feature would provide insights into its role in resolution adaptability. 2. To underscore the advantages of the proposed framework, the experiments should be expanded to include comparative analyses with Latent Diffusion Model (LDM)-based and pixel-based image synthesis methods, showcasing the superior performance of the framework in high-resolution image generation tasks.

Questions

1. Why is the perturbed version $𝑧_{1}$ preferred over the $𝑧_{0}$ from the low-resolution synthesis for guidance purposes? 2. Regarding Figure 4, could you clarify why additional learnable tokens are integrated with the guidance tokens for the self-attention mechanism, instead of solely relying on the guidance tokens? What unique function do these learnable tokens serve? 3. Can you outline the computational steps involved in the implicit neural representation? Is there a need for manually specifying positions? 4. What justifies the forms of Equations (3) and (4), which amalgamate terms with distinct physical interpretations? Is there an underlying principle that supports their direct summation, as it seems to go against intuitive reasoning? 5. In the context of line 255, the use of 𝑡=0.5 and 𝑡=0.05 is ambiguous. Are these intended to denote specific sampling stages within the low-resolution synthesis—fixed and terminal steps, respectively? Consequently, is 𝑡=1 encompassed within the scenario where 𝑡=0.5?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

1. The paper may not sufficiently address how well the model generalizes to datasets beyond the training distribution. It is crucial to understand if the model's performance degrades with different or less common image content. 2. There is a need for more rigorous testing of the model's robustness to various corruptions and perturbations that could be encountered in real-world applications.

Summary

This paper presents a method for Ultra-High-Resolution image generation from text prompts. The method is based on StableCascade. The original StableCascade can generate 1024x1024 images. This paper proposes another HR latent diffusion model that can utilize the guidance from 1024 x 1024 images and generate 4096 x 4096 images. Unlike previous methods that directly use the low-resolution output, the method chooses to use the features of the base model as guidance and proposes an implicit-based method to upsample the low-res guidance features.

Strengths

- The idea of guidance feature and implicit-based upsampling is simple but effective. - The paper reads well, and the presentation is clear. - The results are very impressive. - The proposed method only needs light-weight finetuning from StableCascade.

Weaknesses

- More validation and analysis are needed. In the comparison, a traditional image upsampler is used, but the traditional image upsampler is often smaller and also trained on much smaller datasets. For a fair comparison, it will be good to compare with the state-of-the-art generative image upsampler such as StableSR and Stable Diffusion Upscaler. - A comparison of this baseline is missing: instead of using guidance features, the HR latent model can directly use the LR images / latents from the base model. - It would be good to have visual results of the ablation on LR guidance timesteps. - Ablation on scale-aware normalization is missing.

Questions

- Is the base model frozen from StableCascade? - Is the implicit model jointly trained with the HR latent model?

Rating

4

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

It will be good to show some visual failure cases.

Response to ethic reviewer

**Q1 In line 176, the training data includes self-collected data. Clarifications are recommended regarding the copyright, privacy, and potential violent content within these datasets. Ensuring that all data collection adheres to relevant laws and ethical standards.** We utilized data from the Unsplash Full Dataset 1.2.2, which is publicly available at https://unsplash.com/data. This dataset is provided by Unsplash with the intent of enabling researchers and developers to explore and uncover meaningful insights from the data. It is important to highlight that the creators of the Unsplash dataset have explicitly stated that no user-level data is included. We formally requested and received approval from Unsplash to access their dataset for our research. To ensure quality, we filtered the data to include only images with a resolution of 1536x1536 pixels or higher. Unsplash's privacy policy ensures no personal information is collected or disclosed, and their robust licensing framework, including the Unsplash+ License, allows us to use the images legally and ethically without infringing on third-party rights. Given these comprehensive privacy practices and the legal protections afforded by the Unsplash+ License, we are confident that our use of the Unsplash dataset adheres to the highest ethical standards. We have taken all necessary precautions to protect privacy, comply with data protection regulations, and ensure that our research is conducted responsibly. **Q2 When the self-collected image dataset includes images of humans, it is recommended to discuss the potential fairness issues. Specifically, the paper should address whether there are significant fairness concerns and if any mitigation strategies have been applied.** Concerning fairness issues about human, we recognize that the inclusion of images of humans may raises potential biases, particularly related to gender, age, and cultural representation. For example, prompts like "policemen" often yield images of male officers, "a stylish woman with curly hair" typically generates images of very young women. To address these concerns, we plan to implement several mitigation strategies. Firstly, we will ensure that the dataset is diverse and representative across key demographic dimensions to reduce the risk of over- or under-representation of any group. Additionally, we will employ bias detection and evaluation techniques [1-5] during the model training process to identify and mitigate any emerging biases. These efforts are part of our ongoing commitment to developing a model that is as fair and inclusive as possible. [1] Unified detoxifying and debiasing in language generation via inference-time adaptive optimization [2] Multilingual text-to-image generation magnifies gender stereotypes and prompt engineering may not help you. [3] The male ceo and the female assistant: Probing gender biases in text-to-image models through paired stereotype test [4] Imagereward: Learning and evaluating human preferences for text-to-image generation. [5] OpenBias: Open-set Bias Detection in Text-to-Image Generative Models **Q3 Potential Negative Social Impact: From the application perspective, there are concerns about biased outcomes resulting from text-to-image models due to imbalanced training data that reflects historical biases. The reduced availability of high-resolution images can exacerbate this issue, potentially leading to even more imbalanced training sets. It is recommended to include a discussion on the potentially more biased learning outcomes when adopting high-resolution text-to-image models and the potential mitigation methods.** Thank you for raising this important concern. We acknowledge that the limited availability of high-resolution images might contribute to a more imbalanced training set. Such imbalances could lead models to perpetuate or even amplify existing biases, particularly regarding gender, race, and cultural representation. For instance, prompts about horses often generate images of brown-haired horses, and prompts about astronauts frequently depict space suits with a USA flag. To address these issues, we plan to implement several mitigation strategies. First, we will curate a more balanced and representative dataset by actively sourcing high-resolution images from diverse backgrounds, ensuring a wide range of demographic and cultural contexts. Second, we will incorporate bias detection and correction techniques [6-8] during the model training process to identify and address any biases that may arise from the use of high-resolution data. [6] OpenBias: Open-set Bias Detection in Text-to-Image Generative Models [7] Diverse diffusion: Enhancing image diversity in text-to-image generation. [8] Mitigating stereotypical biases in text to image generative systems