PromptFix: You Prompt and We Fix the Photo

Summary

The paper introduces PromptFix, a novel framework that significantly enhances the capability of diffusion models in following human instructions for a wide range of image-processing tasks. The authors propose a comprehensive multi-modal dataset and further design a frequency-based diffusion model trained on this dataset. Experiments show competitive performance on various restoration tasks.

Strengths

1. The large-scale, instruction-following dataset that covers comprehensive image-processing tasks could be helpful in the field of low-level image processing. 2. Integrating a high-frequency guidance sampling method and an auxiliary prompting adapter shows reasonable problem-solving capability.

Weaknesses

1. While instructions are necessary for users, the types of degradation tasks (such as snow removal and low-light enhancement) are clearly defined. In other words, for images with the same type of degradation (such as foggy images), how to choose different instructions to achieve the best results remains to be clarified. Additionally, it should be considered whether users can use instructions other than the task-specific prompts provided by the authors. 2. The modules used are relatively common. The AuxiliaryPrompt, serving as an information cue derived from the image itself, has been utilized in the literature as referenced in [1,2]. And High-frequency Guidance is also a commonly used method [3,4], albeit in this paper, it is constrained in the form of a loss. Ref: [1] Lin J, Zhang Z, Wei Y, et al. Improving image restoration through removing degradations in textual representations. CVPR 2024. [2] Chiu M T, Zhou Y, Zhang L, et al. Brush2Prompt: Contextual Prompt Generator for Object Inpainting. CVPR 2024. [3] Miao Y, Deng J, Han J. WaveFace: Authentic Face Restoration with Efficient Frequency Recovery. CVPR 2024. [4] Zhao C, Cai W, Dong C, et al. Wavelet-based Fourier information interaction with frequency diffusion adjustment for underwater image restoration. CVPR 2024.

Questions

1. It is important to consider whether the comparative methods have been retrained or fine-tuned on the same dataset. If not, the fairness of the comparison is questionable. 2. In line 205 of the paper, "low-pass operators" might be incorrect; it should probably be "high-pass operators". 3. The test set was randomly selected, and its domain is similar to that of the training set. It is necessary to supplement the comparison with untrained standard datasets. 4. The paper lacks ablation experiments for auxiliary prompts. Additionally, the High-frequency Guidance method lacks numerical metrics and only presents visual results. 5. In Table 1, for the unified restoration task, both PromptIR and AirNet, despite not incorporating instructions, demonstrate good performance, with half of the metrics surpassing those of PromptFix. This seemingly limits the perceived superiority of the proposed method's performance. 6. In line 228, The MGIE[19] was published in ICLR 2024. It has been incorrectly marked in the references.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

As shown in the Weaknesses and Questions.

Summary

This paper introduces PromptFix, a unified model designed to intelligently interpret and execute customized human instructions across a variety of low-level image tasks. To address the issue of spatial information loss in stable diffusion, PromptFix introduces a high-frequency guidance sampling strategy. Additionally, to tackle the degradation adaptation problem, PromptFix incorporates an auxiliary prompt module, providing models with more descriptive text prompts to enhance controllability in image generation.

Strengths

1. The authors construct a comprehensive dataset tailored for low-level image processing tasks. 2. The proposed PromptFix presents a user interactive image processing method, exhibiting superior zero-shot capabilities in blind restoration and hybrid degradation tasks. 3. Both the visual and quantitative results demonstrate the effectiveness of PromptFix. 4. The paper is well-written, with the motivation, method, and experiments clearly explained.

Weaknesses

1. The authors claim that when user-input instructions are discarded, PromptFix occasionally performs text-to-image generation based on the auxiliary prompt rather than image processing tasks. It is recommended that authors provide examples of these failed cases and evaluate whether the generated images can preserve the layout and structure of the input images. 2. There is a concern that PromptFix might learn a mistaken shortcut by memorizing the degradation type of the input image. Therefore, authors are encouraged to demonstrate PromptFix's ability to correctly execute tasks, such as coloring a snowy image while preserving the snow. 3. Authors assert the superior generalization and zero-shot capabilities in blind restoration and combination tasks. Therefore, it is suggested that authors evaluate the model on out-of-distribution and real-world datasets to substantiate these claims.

Questions

The authors claim that the HGS strategy has the potential to introduce spatial information loss. However, the visual results in Figure 5 are insufficient to prove HGS's effectiveness in maintaining image fidelity. Quantitative results should be presented to demonstrate that the proposed HGS module is indispensable to PromptFix.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors claimed the limitations in the Appendix.

Summary

This paper employs prompts to perform low-level image restoration tasks using pretrained diffusion models. To facilitate this, a substantial paired dataset with image restoration instructions was collected. The proposed method relies on latent diffusion models, incorporating the input low-quality image as an additional input and using a VLM to generate auxiliary prompts, serving as another text condition for the network. During sampling, the paper introduces High-frequency Guidance Sampling, wherein the VAE decoder is optimized to better capture the high-frequency details of the input images.

Strengths

1. This paper studies an interesting problem, utilizing instructions to do low-level image restoration tasks. 2. The qualitative results look promising on various image restoration tasks.

Weaknesses

1. The dataset is collected by manually performing the degradation which has a distribution gap with the real-world low-quality images. How will the method perform for the real-world image degradation task, for example, motion blur captured by the phone camera? 2. Regarding the comparison, since the baselines are not trained on the collected dataset, it might be a bit unfair as they aren't aware of the low-level restoration tasks. It would be better to compare the methods under a similar setting.

Questions

1. Regarding the additional cross-attention layers, do you simply add another cross-attention layer after all the existing cross-attention layers and do the two cross-attention sequentially or parallel? And do you clone the weights as well? 2. In Alg 1, it seems that the decoder is updated during the testing time for each input? Since the loss is based on the input (low-quality image), will it affect the output image quality? 3. I'm also curious about how robust the method is for different instructions. For example, if we use different text prompts as instructions, how will the image quality change?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Summary

This paper addresses low-level image processing tasks using a unified, Diffusion-based method. The key idea is to construct a dataset comprising pairs of editing instructions and targets for a variety of tasks and fine-tune a pre-trained text-to-image Diffusion Model on this dataset. Further innovations include augmenting editing instructions with text descriptions from a Vision Language Model (VLM) and a training loss that penalizes difference in high-frequency image components to facilitate detail preservation. The paper presents qualitative and quantitative results on a selected set of low-level image processing tasks, where the proposed method outperforms instruction-based Diffusion baselines and sometimes other task-specific models.

Strengths

- The method addresses low-level image processing tasks in a unified framework. Specifically, the model is conditioned on text-based instructions that specify the tasks to solve. This formulation avoids training one model per task and allows knowledge (captured by model weights) to be shared among tasks. - Low-level image processing tasks often require precise alignment of input and output pixels, yet pre-trained Diffusion models exhibit distortion as a result of lossy VAE encoding. To this end, the proposed method borrows ideas from UNet and introduces skip connections to pass along image textures that might have lost due to encoding. It further introduces a loss term to facilitate the transfer of high-frequency details. This is a sensible design that might be applicable to problem domains with similar requirements on input-output alignment. - Strong experiment results. As the model naturally benefits from strong Diffusion priors, it performs well on tasks such as dehazing and low-light enhancement where the collection of paired training data is challenging.

Weaknesses

- Lack of novelty. Instruction-based image editing using Diffusion models dates back to InstructPix2Pix, which similarly utilizes text conditioning to unify arbitrary image processing tasks. Further, I don't think it is fair to sell the dataset as a main contribution of the paper. Instruction generation using GPT-4 is not new (Both InstructionPix2Pix and LLaVA did that). The pipeline for corrupting GT images also follows standard practice. The auxiliary prompts are new but are specific to the the proposed approach. The techniques for enhancing spatial alignment are also new and seem effective, but they alone cannot justify the broad claim of the paper. - Regarding the two proposed techniques (auxiliary prompts and loss on high-frequency components), there is no ablation study showing they are absolutely needed for the method to succeed. Qualitative and quantitative experiments are needed to show that auxiliary prompts can help in the case where input images are corrupted. Similarly, ablation experiments are needed to show how precisely the F(.) and S(.) terms in Equation 6 affects output quality. - Some model details are lacking. For example, I am not sure I fully understand how auxiliary prompts are incorporated into the Diffusion UNet. If the cross-attention layers are replicated, are they fine-turned during training? The paper leaves me with the impression that only the LoRA convolutions are learned.

Questions

- Are the experiments on low-light enhancement, dehazing, desnowing, etc. performed on simulated data? The artifacts look unrealistic to me. How does the model perform on real data? I am concerned about generalization on real data since the model is exclusively trained on simulated data. - The paper claims in the abstract that the method "achieves comparable inference efficiency" with the baselines. Please provide concrete numbers to prove this claim.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Limitations are discussed in A.2. The model performance might be sensitive to user-provided instructions. Additionally, passing along high-frequency textures from the input image inevitably copies some artifacts to the output.