Summary
The paper addresses the task of instruction-based image editing, with a specific focus on instructions that add new objects to the image.
To tackle this task, the authors note that it is much easier to remove an object from an image (e.g., using inpainting) than it is to add one, and so they can create a dataset of paired images with and without an object by using existing models to remove objects. To create editing instructions that align with the paired data, the authors leverage a VLM and an LLM. They further propose a set of quality filtering steps to ensure their dataset contains higher-quality examples. This dataset is used to train an instruction-based model (based on InstructPix2Pix) which outperforms existing baselines on object-adding benchmarks across a wide range of metrics.
Finally, the authors also demonstrate that their model can be further fine-tuned on general editing data (from e.g., MagicBrush) leading it to outperform InstructPix2Pix on general editing tasks.
Strengths
The paper’s approach is reasonable, and it appears to outperform a diverse set of baselines on multiple datasets and across many metrics. In this sense, the comparisons shown in the paper are fairly comprehensive.
There is also significant detail provided in the paper and the supplementary which should help facilitate reproducibility, and the authors also promise to release their dataset which could help future research on this task.
Finally, the paper was relatively clear and easy to follow.
Weaknesses
My primary concerns are twofold:
First, the core idea of learning to add objects by constructing synthetic data using inpainting already exists in prior published work, including work which was published in conference proceedings before the ICLR deadline (e.g., ObjectDrop, Winter et al, ECCV 2024). I think the authors should, at the very least, tone down their focus on presenting this as a core contribution, and instead highlight other aspects of the work such as the importance of the dataset and its construction.
Second, the authors repeatedly highlight advantages of their data construction approach, such as the fact that their target images are real and not synthetic, but I was unable to find any experiments which support these claims. Considering that the dataset and its construction are the paper’s core contribution, more should be done to ablate different aspects of its creation.
More specifically: the authors show that, as a whole, using their data during training is advantageous. They also ablate their use of VLM+LLM combo to create the editing instructions (in place of simple ‘add <class>’ type of prompts). However, I would have also liked to see experiments that demonstrate that: (1) The extensive data filtering the authors conduct is beneficial (at the very least, filtered data vs. unfiltered data, though if the authors have the compute for it, then more in-depth tests on specific filters would be interesting). (2) There is real advantage to using real images as targets, when compared with simply generating a larger set of synthetic examples.
The above concern is amplified by the fact that IP2P finetuned on MagicBrush data outperforms the proposed data construction method on most metrics, which signals that some aspects highlighted by the authors (e.g., fully realistic target images) are not as crucial.
Additional, more minor concerns:
a) I am unsure why the original IP2P is referred to as the state of the art on this task when MagicBrush (NeurIPS 2023) have a model (that the authors also use!) which outperforms it. The authors should add comparisons to this model in the qualitative figures, and I’m not sure I understand the reasoning behind arbitrarily splitting it to a different comparison in the quantitative results. The user study experiment should likely also be done against MagicBrush and not against the original IP2P.
b) The images in the paper show noticeable degradation of quality. The images lose a lot of high frequency detail and faces seem distorted (see e.g., lines in the carpet or the hair of Tobey Maguire in Fig 1.). Is this mostly a result of the VAE? Can you report fidelity metrics (e.g., L2) on the background for just VAE reconstruction?
Nits (did not affect my score but should be fixed):
a) In line 400: CLIP is not by Ruiz et al, and they are not remotely the first to use CLIP-T to evaluate image-text alignment, not even in the context of diffusion models or even in the context of model customization or image editing. This line should probably just cite Radford et al.
b) When you cite as “Name et al., (Name et al, 2020)” please use \citet{} to cite as “Name et al. (2020),”.
Questions
Please see the questions in the weakness section.
The main thing I’d like to know is what parts of your dataset construction approach are actually important. In other words, I'd like to see an ablation on the main contributions proposed by the paper.