CLIPAway: Harmonizing Focused Embeddings for Removing Objects via Diffusion Models

Summary

This work addresses the object removal problem with a simple and effective embedding arithmetic strategy. This idea is implemented by combining the object semantic understanding ability from alpha-clip and the generative ability from the text-to-image diffusion models. One good property is that the proposed method is a plug-and-play strategy that can be adopted in various Diffusion-based t2i frameworks. Compared to the previous GAN-based and Diffusion-based inpaint methods, this method demonstrates better results with suitable background and fewer artifacts.

Strengths

The simple and effective embedding arithmetic strategy is intuitive and suitable for the object removal problem. The designed technical framework of the alpha-clip plus t2i diffusion model is flexible and powerful since many powerful conditional image generative models are based on clip embeddings. It would be interesting to see how this technique can be applied to other t2i frameworks. Figure 5 demonstrates the superiority of the proposed method which performs a good trade-off between FID and CLIP Scores. Figure 6 also shows good inpaint results with fewer artifacts and blurry effects. The paper is easy to follow and the algorithm has been illustrated step-by-step. I highly recommend the authors release code and model for the following works. I believe it can benefit the related CV fields.

Weaknesses

It seems that the applicable image resolution is limited by the diffusion method. Comparisons on higher resolutions could demonstrate the boundary of this technique since one benefit LaMa-like methods is the applicability to high-resolution images. One popular baseline of “LaMa + SDinpaint” should be compared, please check https://github.com/Mikubill/sd-webui-controlnet/discussions/1597 More failure cases could be presented and analyzed to show the limitations. I suppose this method can be extended to other frameworks like SDXL, it would be great to perform these extensions to make this work more comprehensive.

Questions

“LaMa + SDinpaint” is required to be compared. Comparisons on higher resolutions could be demonstrated.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

More failure cases could be discussed.

Summary

The paper identifies a common limitation in recent diffusion model-based inpainting methods: unintended hallucinations of the removed object. To address the problem, the paper introduces CLIPAway, a plug-and-play module that does not rely on any specific training. Using vector arithmetic, it successfully obtains an embedding that predominantly focuses on the background. Through comparisons, it is demonstrated that the proposed method outperforms existing methods in object removal.

Strengths

- The paper identifies a well-known limitation in the current diffusion models and the motivation behind this paper is very strong. - The proposed method does not rely on synthetic data or any special annotations during its training, thus having a stronger ability to generalize and more robust performance compared to prior methods (Fig 8). - The idea of using AlphaCLIP encoder and doing vector subtraction is simple but effective. - Extensive results (Tab 1 and user study) are presented to support the conclusions of the paper, especially on whether the object is correctly removed.

Weaknesses

- Although the proposed model can effectively remove the object, the quality of the inpainted image is not stable. There are notable artifacts in Fig 1, such as the shadow in the 1st, 2nd rows on the right, and 3rd, 4th rows on the left. - The conclusions / observations in Fig 4 lacks enough evidence. E.g., in the bottom-left, it's hard to tell whether the foreground or the background is more prominent. Adding more visual results / quantitative comparisons may help to obtain a more solid/reasonable observation. - There have been a lot of evaluations on whether the object is correctly removed (in Tab 1), but should be more on assessing the visual quality, e.g., LPIPS.

Questions

- In lines 192-193, when setting up the baselines, the authors used empty prompt or "background" as input. In practice, providing an accurate prompt describing the background can also be useful in removing the object. E.g., when removing a laptop from a table, the prompt can be "an empty table". Does this work for the DM-based baselines?

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

Already addressed.

Summary

This paper proposes an approach that aims to tackle object removal in stable diffusion models. The paper utilizes AlphaCLIP embedding (that are trained with an additional alpha layer mask to enable incorporation of regions of interest), and techniques such as IP-Adapter that decouples the diffusion unet cross-attention mechanisms to accept features such as coming from image/text separately. The authors train using image prompts as inputs and controlling the alpha channel to correspond to background or foreground (inverse of background) focus to guide the attention mechanism. Eq 2 aims to calculate embedding that are orthogonal to the foreground embedding, resulting in embedding that are focusing on the background.

Strengths

- The method leverages existing work well, and the proposed method appears to indeed remove objects in a smoother way than competitors -It is sometimes difficult to judge from a few examples qualitatively, but it seems that the proposed method performs relatively better in most cases presented - ablation studies show the impact of background/foreground embedding - although in most cases it seems that background and foreground embedding do contain both. The orthogonal embedding do appear to include only information related to the background - although that does not exactly match the background of the image

Weaknesses

- As with the shadows example mentioned in the limitations, the method does not seem to remove objects based on much context - i.e. region of interest is a stronger control. This may lead to undesirable or unrealistic results - so perhaps a trade-off between strict ROI adherence and context could be employed. - The method can be plug and play after training - but the training process itself can make it specific to architectural choices - Although mentioned in the paper, this work appears to have a larger computational overhead that simpler inpainting methods, which perhaps can then be improved with simpler post-processing - the paper is focused on object removal in a background-consistent manner, which is a limited task, while some compared methods tackle the more general problem if image inpainting

Questions

please see above

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Summary

This paper addresses a commonly observed problem when using pre-trained diffusion models for object removal: in standard inpainting setups, these models often add similar objects in place of the ones to be removed instead of extending the background to the masked area. To address this problem, the authors use CLIPAway, a method that leverages region-focused embeddings from Alpha-CLIP, obtaining further disentangled background embeddings that can then be used to guide the diffusion model to perform high-quality object removal without foreground leakage.

Strengths

- This paper addresses a very common and important problem in practical applications of large-scale pretrained T2I diffusion models. - While very simple, the proposed method combines existing methods (Alpha-CLIP and T2I-Adapter/T2I diffusion models conditioned on CLIP embeddings) in a novel and very effective manner. The proposed method is demonstrated to work well qualitatively and outperform other approaches when combined with off-the-shelf inpainting methods in a quantiative evaluation. - The paper is written clearly and each part of the method is well-motivated and explained both in text and with qualitative examples.

Weaknesses

The main weakness with the paper is the simplicity of the method. Generally, a simple method that achieves the goal is often better than an over-involved complex method, but most of the parts of this method are relatively obvious (translating from one CLIP embedding space to another seems to primarily be an engineering decision as the pre-trained IP-Adapter does not directly consume Alpha-CLIP embeddings, but it would likely be very simple to just train an adapter on these embeddings; and orthogonalization is a very basic operation, even if it is curious that it works so well in this case) and aspects of transferability of this method, which are relevant for its further impact beyond the implementation presented in this paper (e.g., whether it is reliant on the exact two CLIP embedding spaces used, or whether they can be substituted with other CLIP spaces with similar properties), are not covered.

Questions

- Can the projection also be performed in the AlphaCLIP embedding space (or other CLIP embedding spaces) or does it only work in the adapted OpenCLIP embedding space? This would be especially interesting as it would speak to the generalizability of the proposed method, as being limited to specific CLIP embedding spaces might limit practical applications, such as applying this to a standard unCLIP model (e.g., Karlo https://github.com/kakaobrain/karlo, Stable unCLIP https://huggingface.co/docs/diffusers/en/api/pipelines/stable_unclip). - What is the time (wall clock/GPU-hours) required for training the MLP?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have adequately addressed the limitations and societal impact of their work.