Open-ended visual storytelling presents a formidable challenge for current text-to-image models, which frequently struggle to preserve both narrative coherence and consistent character depictions across generated sequences. To address this, we introduce StoryCrafter, a multi-character diffusion model that leverages a novel instance-level cross-attention module with supervised fine-tuning to ensure precise text-character alignment and consistent multi-character interactions throughout the narrative. Further, we propose Direct-Diffusion Group Relative Policy Optimization (D2GRPO), a novel RLHF stage that optimizes denoising strategies using automated story-aligned rewards, selecting the best candidate frames from a generated group. We evaluate our approach through human assessments and vision language model (VLM) scoring, measuring text-to-image alignment, style and character consistency, and fine-grained detail quality. Experiments on three benchmarks demonstrate that StoryCrafter outperforms existing methods, achieving 7% improvements in storytelling consistency and 10% in character accuracy, while outperforming baselines in both human and VLM evaluations.
Paper
Full text
StoryCrafter: Instance-Aligned Multi-Character Storytelling with Diffusion Policy Learning
Semantic Scholar · Computer Science · 2025
Abstract
Open-ended visual storytelling presents a formidable challenge for current text-to-image models, which frequently struggle to preserve both narrative coherence and consistent character depictions across generated sequences. To address this, we introduce StoryCrafter, a multi-character diffusion model that leverages a novel instance-level cross-attention module with supervised fine-tuning to ensure precise text-character alignment and consistent multi-character interactions throughout the narrative. Further, we propose Direct-Diffusion Group Relative Policy Optimization (D2GRPO), a novel RLHF stage that optimizes denoising strategies using automated story-aligned rewards, selecting the best candidate frames from a generated group. We evaluate our approach through human assessments and vision language model (VLM) scoring, measuring text-to-image alignment, style and character consistency, and fine-grained detail quality. Experiments on three benchmarks demonstrate that StoryCrafter outperforms existing methods, achieving 7% improvements in storytelling consistency and 10% in character accuracy, while outperforming baselines in both human and VLM evaluations.