Summary
The paper attempts to integrate various large models, including LLM (LLaMA), LVLM (Bard), a Diffusion Model (Stable Diffusion), and an Editing Model (Blended Diffusion), to establish a finetuning and inference pipeline aimed at enhancing the capability of compositional text-to-image generation. The proposed pipeline is structured in three distinct stages: LVLM-based Evaluation, Diffusion Model Finetuning, and LVLM-guided Editing. While the paper provides examples of the pipeline’s application, it relies solely on qualitative evaluation methods.
Strengths
The concept of integrating several sophisticated models into a single pipeline that potentially complement each other's functionalities is commendable and could inspire more research in the field.
Weaknesses
1. The premise of amalgamating various complex models is appealing, but the justification for such a pipeline, along with comprehensive evaluations of its efficacy, is lacking. The current results do not substantiate the effectiveness of the pipeline. Particularly, the necessity for both stage 2 (Diffusion Model Finetuning) and stage 3 (LVLM-guided Editing), which seemingly aim for the same goal, is not clear. There is a lack of clarity about the contributions of each stage. An in-depth ablation study would be valuable to discern the individual and collective impact of these stages.
2. The paper’s evaluation approach is insufficiently rigorous. The success cases are limited in number and are repeatedly used, without the support of robust quantitative analysis. Examples only contains simples cases, e.g., “three black dogs”, “a white cat and a black dog”. The scenarios presented could potentially be addressed by Blended Diffusion Editing alone, raising questions about the added value of the other stages.
3. The computational demands of the pipeline, particularly the LVLM-guided iterative refinement in stage 3, are not discussed. An analysis of the computational costs and the typical number of iterations required for image generation would be beneficial.
4. The quality of writing and the clarity of figures require improvement to enhance the paper's comprehensibility and professional presentation.
Questions
1. There is a typographical error in Section 3.3 (LVLM-based Question Answering) involving the missing parenthesis around the $Q_i, A_i$ pairs.
2. There seems to be a mismatch between LLM (LLaMA) and LVLM (Bard) in the experiments, potentially leading to discrepancies in the answers and consequently, a lack of reliability in the accuracy measures. Would it not be more consistent to just use Bard, as it supports both textual and visual modalities?
3. Equation 7 mentions a term $\tilde{Q_i}$ which lacks a definition. Could you provide a clarification for this term?
4. In Figure 5, there is an inconsistency with the input text depicted in the first panel. The input text should be “three black dogs”?
5. References to Blended Diffusion and SAM appear to be missing when these terms are introduced in the paper.
6. The examples in Figure 4, such as the fifth case, seem to indicate that stage 2 is ineffective. This observation supports my concerns regarding the relative impact of stage 2.
Recommendation:
Given the aforementioned concerns, particularly the lack of robust evaluation and the unclear contribution of each pipeline stage, I recommend that the paper be revised before consideration for publication. The authors should address the weaknesses and questions detailed above, ensuring that the pipeline's design and efficacy are convincingly demonstrated through comprehensive experiments and quantitative assessments.
Rating
3: reject, not good enough
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.