Summary
The authors propose a tunning-free pipeline called RIVAL(Real-world Image Variation by Alignment) for generating diverse and high-quality variations of real-world images.
In previous works, some models also generate images with novel concepts and styles but require additional training stages and data, and others directly incorporated images as the input condition results in suboptimal visual quality and content diversity.
To tackle this issue, the authors suggest a pipeline that maintains the style and semantic content of the reference without additional training stages and data.
The proposed method comprises the inverted latent chain alignment(step-wise latent normalization) and cross-image self-attention injection.
The authors determine whether aligning the inverted latent chain and injecting the modified self-attention key and value by adopting $t_{align}$ and $t_{early}$.
Unlike the other papers that use the cross-image self-attention injection method with an additional forward to calculate the cross-attention mask, the suggested approach only needs a single forward.
The authors show that RIVAL enhances performance by applying the alignment process to other text-to-image tasks through experiments and ablation studies.
Strengths
- The authors claimed a novel plug-and-play pipeline to generate high-quality, real-world image variations without additional optimization through an inverted latent chain alignment and cross-image self-attention injection.
- The proposed pipeline enhances the performance of text-to-image models, such as text-driven image generation with real-image conditions and example-based inpainting. The pipeline generates images fast and flexibly with diverse text prompts adopting a new self-attention injection approach with a single forward pass.
- Compared to CFG, the advantage of the proposed pipeline is its ability to avoid misalignment between latents in two chains and its direct utilization. The authors decouple the two inference chains and rescale noise prediction during denoising inference. Furthermore, they formulate noise prediction as an adaptive normalization.
- The quantitative results outperform the other pipelines provided in the paper except for one metric, and the qualitative results show that RIVAL generates images with diverse text inputs while preserving the reference style.
Weaknesses
The experiments need to be more extensive.
- The authors proposed a plug-and-play pipeline, but there are no quantitative results compared with the other plug-and-play pipelines mentioned in the paper. RIVAL used a similar scheme as MasaCtrl[2] and Plug-and-Play[3] except the latents alignment. The authors employed similar schemes to the pipeline, so comparing the quantitative results with the others would be beneficial.
- In the appendix, the proposed method allows fast and flexible text-to-image generation, but there are no results about how fast it is. It would be reasonable to compare quantitatively with other plug-and-play pipelines.
The authors need to validate the ability of the pipeline to apply to other diffusion-based generation tasks.
- The authors show the qualitative results of self-example image inpainting in Fig. 6, but the results lack experiments. It would be hard to explain the ability to extend RIVAL’s framework to other image editing tasks.
(Minor) The notation of $t_{align}$ and $t_{early}$ is confusing.
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In *CVPR*, pages 10684–10695, 2022.[2] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. *arXiv preprint arXiv:2304.08465*, 2023.[3] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. *arXiv preprint arXiv:2211.12572*, 2022.
Questions
- In Table 1, I wonder if using specific $t_{align}$ and $t_{early}$ values for each metric would lead to better quantitative results.
- In Fig. 6, the inpainting results show a different masked area. Can you show me the same results by overlaying the masked area at the source image like SD[1] in the paper?
- Can you provide examples of other tasks to which RIVAL can be applied?
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolutionimage synthesis with latent diffusion models. In *CVPR*, pages 10684–10695, 2022.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The paper needs more extensive experiments and analysis of the limitations of the proposed method.