Summary
The paper presents a method to invert input portrait images to latent diffusion model (LDM)'s embedding space and then use the same to create text-based identity preserving compositional portrait images. To enable image inversion, the authors create a novel active appearance model "Celeb Basis" (image-only identity PCA) from StyleGAN-2 style frontal images. Later, this "Celeb Basis", and a textual embedding are jointly used to train CLIP for identity preserving (ArcFace style loss) image synthesis. At test time, new images of the inverted target image (represented by the Celeb Basis) can be generated based on textual prompts.
The authors presents ablation studies wrt prompt and identity preservation (and detection) and compare against SOTA methods "An image is worth one word" (Textural Inversion), Dreambooth and Multi-concept customization of text-to-image diffusion (Custom diffusion) and achieve good results for prompt-based facial images.
Note, that while other methods are not specific for portrait images, the current Celeb Basis has been built for portrait style images only.
Strengths
While the idea of creating an active appearance model, Cootes et al. (Celeb basis) is quite established in the vision community; this paper attempts to "rig" (e.g. StyleRig, Tewari et al.) the LDM space to create new prompted facial image given target image (identity preservation only). This is the main contribution of the paper and is novel in that regard, while the idea of controlled prompt-based image synthesis via LDM has been explored before.
The celeb basis thus enables more control over image synthesis, where the semantic of "identity" preservation helps with a more _semantic_ control and results in better compositional results (two target images used together). These compositional results analysis (Fig 5, last two row) are mostly visual since I look at the overall image to argue if it looks more "believable".
The paper is generally easy to follow and extensive ablation studies justify the various modeling decisions made.
Weaknesses
- The range of "Celeb basis": Since the method used PCA and uses a limited set of image to create the basis, it's limited by the choice of images, which is further limited by the choice of named celebrities in the LDM model itself. For example, it's unclear how well the model will represent certain age, pose and expression ranges.
- Table 1 shows that while the users prefer the author generated images on Identity and Detect metics, Textual Inversion outperforms them, while that method is not specific to facial images as such. The reason argued is that Textual Inversion outfits. Given that Textual Inversion is a generic method, while the current method is meant for facial images, this result is surprising.
- Comparison against FastComposer: Since most results presented against a concurrent work "FastComposer" seem cherry picked, the presented comparison seems biased. For example, the result presented in Fig 1 (FastComposer) seem better than those presented in Supp (Figure 5). Moreover, the current method does not explicitly model expression control in their Celeb basis, and their Fig 6 comparison seems less principled.
Questions
- The data augmentation technique is essentially used color space augmentations. Did the authors consider any geometric augmentations? Since, the image inversion techniques in the StyleGAN space are quite mature (eg HyperStyle), adding augs that change the pose or expression would make the method further robust?
- How were the results shown in the paper selected? Was it the first image that the method created used or several prompt tries with subtle textual changes made to pick the results shown in the paper?
- Did the authors consider the use of multi-view images for identity reconstruction?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Novelty: The main novelty of the approach is introduction of celeb-basis, while the rest of the method seems straight forward. Since the results are only shown for a facial model, it scope limits the contribution of the work. Please rename the work from "Inserting Anybody" to "Inserting Anybody's Portrait". As such control over bodies, etc. is not part of the paper's contribution.
Expression control: While the basis does not model expressions in the 3DMM sense or provides a basis for expressions, the method seems to provide control expression control via prompts (supp. comparison against FastComposer). This seems to be an after thought and it's unclear how well the expressions are enabled by the method. In order to fully understand the expression control capabilities, please use a ground truth dataset (such as https://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html) and provide comparison against ground truth expression vs those generated by prompts. This would also help understand how well the expressions can be re-created.
Overall, I think this is an interesting paper and the results are generally good; the paper's novelty is limited since it enables only identity inversion, and later dependent on CLIP/LDM's capabilities to create pose, expression or semantic variations (e.g. an older V_1), etc. So, there's merit in the contribution. Besides the points/questions mentioned earlier, the authors should expand the limitations, move FastComposer results to the main paper.