Summary
This paper presents GenArtist, a system that utilizes MLLMs as agents for image generation and editing, especially for complex language descriptions. The key idea is to first use MLLM to decompose the generation task as an execution tree of various tools such as SDXL and LMD, and then utilize all the tools in a predefined tool library. Extensive experiments on T2I-CompBench and MagicBrush shows promising results.
Strengths
- The problem of unified image generation and editing over arbitrary language instructions is very relevant for both academia and industry. How to leverage the existing powerful LLMs such as GPT-4 and other tools is also very important for the deployment of current academic progress.
- The proposed method is quite simple and is reasonable. The paper is clearly written and the results look good to me.
Weaknesses
Overall, I think this paper has made some good contributions and shown some promising results. However, I hold some concerns as follows,
**Major concern.** My major concern is about the possible limitation in the decomposition tree. All images generated for the complex text is by first generating an initial image, and then utilizing editing tools to match the text description. This could be very limited if the images initially generated are not very decent. This is because even if the final edited image matches the language description better in semantics, the overall quality and layout may not be very natural in some cases. The editing trace may be easily detected and this may be attributed the weakness of the editing models. The editing operation in the paper is mostly editing local regions while keeping the original overall structure, and this could be the problem. For example, the generated result with "hot dogs" in Figure 1 is not natural or of decent quality to me when compared to results generated by models like Playground. The image is blurred and the hot dogs look a little fake.
**Human-aligned evaluation.** The current evaluation is mainly conducted on T2I-CompBench and MagicBrush. However, it has been well known that traditional metrics such as DINO scores are not aligned with humans. This could cause a result that even if the image quality is not as good as another single-expert result, the score may still be better. However, this might be misaligned with human preference in some cases. Can authors conduct more evaluations on more advanced benchmarks like DreamBench++ [1]?
Minor suggestion: I think the paper organization could be improved by trying to put each full paragraph on one page or two. Try your best not to split one section or subsection into two pages. For example, Section 4.1 and 4.3 are all put under the bottom of one page with very few contents, this could cause a sense of reading fragmentation.
[1] DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation.
Questions
As stated in weakness, I am curious about the results of modern human-aligned benchmarks, such as DreamBench++. I would be happy to see more human-aligned evaluation, if possible.
I am looking forward to the authors' response.
Limitations
Yes, the authors have discussed the limitations. However, I would suggest the authors discuss the potential limitation brought by the initial generated results and editing artifacts.