Summary
The paper explores and proposes an interleaved generation framework based on prompting large-language models (LLMs) and pre-trained text-to-image (T2I) models. Additionally, it attempts to introduce an automated evaluation scheme based on LLMs for assessing Entity Consistency and Style Consistency.
Strengths
This paper presents a interleaved generation framework based on prompting large-language models (LLMs) and pre-trained text-to-image (T2I) models, which includes user query composition, text generation, adding global context.
The paper also explore to present an LLM-based evaluation strategy for assessing interleaved content generation in two aspects: Entity Consistency Evaluation and Style Consistency Evaluation.
Weaknesses
1. novelty. The major concern with this paper is its excessive reliance on the APIs of existing models, GPT-4 and SDXL. The method appears more like a prompt engineering approach, devoid of the need for fine-tuning models, analyzing network structures, or delving into training strategies. It lacks the provision of novel insights, and from my perspective, this paper resembles more of a technical report than an academic contribution.
2. Entity Consistency and Style Consistency. Indeed, Entity Consistency and Style Consistency are two crucial challenges in Storytelling generation. However, this work attempts to address these challenges merely using simple prompt description constraints, which appears to be a weak approach. Recently, some works[1] [2] [3] based on personalized LORA seem to offer a more reasonable solution for tackling consistency issues.
3. GPT4-version for evaluation. The proposed evaluation method appears to be quite rough and non-standardized. It seems to involve directly inputting the generated content into GPT-4 and asking whether it is reasonable or not, which is overly simplistic and engineering-oriented. Furthermore, the authors have not provided evidence for the validity of this method and have not offered new insights. At the very least, evaluations based on relevant personalized methods [1] [2] [3][4] would be necessary.
[1] Gu, Yuchao, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao et al. "Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models." arXiv preprint arXiv:2305.18292 (2023).
[2] Chen, Xi, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. "AnyDoor: Zero-shot Object-level Image Customization." arXiv preprint arXiv:2307.09481 (2023).
[3] Guo, Yuwei, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. "Animatediff: Animate your personalized text-to-image diffusion models without specific tuning." arXiv preprint arXiv:2307.04725 (2023).
[4] Ruiz, Nataniel, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. "Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510. 2023.