Summary
InstructEdit, the system introduced in this paper, enables text-guided image editing. Leveraging a language model (ChatGPT) and, optionally, an image captioner (BLIP2), it extracts the object in question and rewrites the instruction into an edited caption. Subsequently, it employs the Grounded Segment Anything model to segment the intended region. Finally, the mask and edited caption are inputted into DiffEdit to produce the edited image. A small-scale qualitative user study (involving 26 reviewers and 20 examples) indicates a preference for InstructEdit over competing methods (MDP-$\epsilon_t$, Instruct-Pix2Pix, and DiffEdit) in 57.0% - 67.5% of cases. The paper further explores InstructEdit's applicability to NeRF model editing and acknowledges several of its limitations.
Strengths
**Originality**: The paper introduces a novel integration of ChatGPT, BLIP2, Grounded SAM, and DiffEdit for enhanced text-guided image editing. Its originality is marked by this unique combination, although the approach itself may be considered somewhat straightforward.
**Quality**: The paper addresses the issue of ambiguous edit instructions through the use of masks, building on a foundation laid by prior research. The methodology is technically sound, showcasing notable advancements over standard methods, particularly in images featuring complex or multiple objects. However, the scope of the experiments could benefit from expansion (details in the weaknesses section).
**Clarity**: The paper is well-organized and generally comprehensible. Nonetheless, certain figures might cause confusion and would benefit from further clarification (details in the questions section).
**Significance**: The paper's most significant limitation lies in its impact. The integration of existing models, while practical, offers limited theoretical or empirical advancement, leading to a potentially modest contribution to the field.
Weaknesses
The paper lacks a comprehensive theoretical and empirical analysis of its proposed system. Critically:
1. Reliance on pre-built models may hinder further innovation. Although the paper acknowledges certain limitations, such as suboptimal outputs from the ChatGPT + BLIP2 combination, it offers no clear strategy for remediation. This lack of guidance leaves users without a roadmap for improvement when issues arise.
2. The individual contributions of each component to the overall system are unclear. Questions like the accuracy of ChatGPT in extracting the object of interest, and how this impacts the final image edit quality, need addressing. At the bare minimum, the paper should build a test set to quantitatively evaluate this (even with subjective measures). Moreover, exploring alternative component configurations, like substituting ChatGPT with Chat LLaMa 2, could show potential trade-offs in accuracy and response time.
3. "Blended Latent Diffusion" is a relevant methodology that could be worth comparison. It could also potentially serve as an alternative to the DiffEdit component in the proposed framework.
4. The system's overall latency is unclear. It may be significantly slower than approaches like Instruct-Pix2Pix, raising questions about the justification for potential trade-offs between speed and accuracy.
5. Incorporating a segmenter may inadvertently constrain the system's editing capabilities, especially for commands like "add a hat" or "turn into Van Gogh style." Understanding these limitations is crucial for assessing the system's versatility.
Questions
In addition to the previously mentioned concerns, the presentation raises several minor issues:
6. Figure 1, the subplot for "Change the buildings to palaces" -- it exhibits noticeable artifacts in the sky. This suggests potential deficiencies in either the masking or blending processes. Clarification on the cause of these artifacts and proposed solutions would enhance the paper's rigor.
7. The examples presented in Figure 4 may not effectively showcase the strengths of BLIP2. The depicted fox appears relatively good at lower encoding ratios, while higher ratios introduce significant artifacts regardless of BLIP2's involvement. A more illustrative example might better demonstrate BLIP2's efficacy. This reinforces the need for a more precise quantification of each component's contribution to the system's overall performance.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.