Summary
This paper presents StoryGen, an auto-regressive image generator that leverages text and image conditioning. StoryGen incorporates a style transfer module integrated into the text-conditioning module, along with a visual context module. The authors also constructed a substantial dataset called StorySalon, comprising 2K storybooks and 30K text-image pairs.
Strengths
1. This paper constructs a new dataset StorySalon contains 2K storybooks and more than 30K well-aligned text-image pairs. The authors have invested significant effort into filtering the data, making it a valuable resource for advancing the field of story visualization.
2. The paper is well-written and easy to follow.
Weaknesses
1. The illustration does not align with the description provided. In line 135, the authors state that "StoryGen generates the current frame $\mathcal{I}_k$ by conditioning on both the current text description $\mathcal{T}_k$ and the previous frame $\mathcal{I}_{k-1}$, as illustrated in Figure 2." However, the left figure of Figure 2 shows the image conditioned on more than one previous image, which contradicts the mentioned conditioning approach.
2. The improvement of Human feedback appears to be trivial, as indicated in Table 2. The 0.19 FID score gap could potentially be attributed to different training seeds, which raises doubts about the significance of the reported improvement. (I do not agree with the statement that 200 stories are too small since the model is trained using 2k stories overall. It appears to be sufficient for human alignment and does not require an extensive amount of data.)
3. The FID score lacks precision in the test set, particularly with only 100 storylines. It is recommended that the authors expand the test set by including more stories to provide a more accurate evaluation.
4. The baselines SDM and Prompt-SDM are too weak. It is suggested that the authors compare StoryGen with finetuned or LoRA-finetuned SDM models using the same training settings to establish a more robust baseline for comparison.
5. The auto-regressive generation approach employed by StoryGen has already been proposed by AR-LDM. Consequently, the architecture design itself lacks novelty.
6. StoryGen is only conditioned on one previous image and does not utilize the corresponding caption of the previous image. In the depicted cases of Figure 1 and Figure 4, there is only one main recurring character. If multiple characters were present, StoryGen may struggle to ground the characters in the previous images. Furthermore, if a character does not exist in the previous image, StoryGen may face difficulties in maintaining consistency between frames.
7. The language understanding capacity of StoryGen appears to be weak. For instance, in the second case of Figure 4, the rabbit appears small in the fourth frame, whereas it should be as big as it is in the fifth frame. Additionally, in the sixth frame, the boy's hair does not become brighter as described in the caption. Moreover, in the seventh frame, multiple other boys are depicted with the same yellow hair, which contradicts the previous story setting. This limitation may stem from StoryGen solely relying on the only one previous frame $\mathcal{I}_{k-1}$ and not incorporating previous captions into its generation process.
Questions
1. StoryGen generates the current frame $\mathcal{I}_k$ by conditioning on current text description $\mathcal{T}_k$ and the previous frame $\mathcal{I}_{k-1}$. I am wondering why the authors do not use all historical frames $\mathcal{I}_0, \cdots, \mathcal{I}_{k-1}$ in this process.
2. StoryGen introduces a style transfer module $\phi_{text}$ and a single-frame pre-training stage. However, since the training set is constructed from multiple sources (as mentioned by the authors in Line 244), the style is inconsistent. Moreover, many LoRA style transfer plugins not only tune the cross-attn module but also the self-attn modules. Why not follow this setting and add LoRA layers in both self- and cross-attn layers? This way, StoryGen can also benefit from other style transfer LoRA developed by the community.
3. In Line 186, the authors mention randomly dropping some words in the text with a certain probability following BERT. I have doubts about this technique because there is no masked language modeling task in StoryGen, and such a technique may not be helpful.
4. I question whether the authors truly need to replace descriptive captions with story narrative text. The story in the storybook and the story generated by LLMs are both in the form of story narrative text, rather than descriptive captions. Using story narrative text to train the model may lead to implicit alignment with human performance.
Rating
3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
1. The author should provide examples of the constructed dataset showcasing different visual styles and character appearances.
2. The StorySalon dataset consists of 2K storybooks and over 30K well-aligned text-image pairs, which is smaller compared to datasets such as FlintstonesSV (24K stories and 123K image-caption pairs), PororoSV (14K stories and 74K image-caption pairs), and VIST (27K stories and 136K image-caption pairs). Despite the authors' claim that StoryGen can perform open-ended story generation, it remains unclear whether StoryGen can generate stories involving more complex scenarios with unusual entities.
3. Apart from human feedback, the authors have not conducted any other ablation studies to evaluate the effectiveness of their proposed techniques, such as word dropout and curriculum learning.
4. There are concerns regarding the legality of using web-crawled e-books. The authors should provide additional information about the sources of the e-books and clarify whether proper copyright guidelines were followed.