Summary
The paper proposes a novel method for performing text-to-3D generation for specifically human (or humanoid) heads. These generated heads can be further edited and refined using more fine-grained detailed text prompts while still preserving the identity of the generated asset. This is accomplished with two main contributions: using a landmark-aware score distillation loss in order to ensure that the generated heads roughly align with a template head mesh (FLAME), and introducing a loss function which balances text-based editing with a new description with the original description used to generate the asset. The generated results fix a number of artifacts with existing text-to-3D generation methods specifically for heads. For example, the generated heads do not suffer from the multi-face problems, and the strong prior given by a parametric head model ensures that the generated results are geometrically correct. Additionally, the results can be edited while still maintaining the identity of the original generated results. This is shown in comparison to a number of baseline methods, which are outperformed.
AFTER REBUTTAL:
I have read the rebuttal and appreciate the detailed response. I appreciate the additional quantitative evaluations, as I believe this was a limitation of the method that has been addressed. However, I still think the quality is a bit limited, but this is the case for most text-to-3D methods. The additional comparisons weakness has been addressed as well. However, I don't see justification for the the more stable training - I would suggest removing this claim without verifying it significantly, while Fig 11 is nice, I am not sure this is enough to make an entire claim based on.
Strengths
In my opinion, the strengths of the paper are that:
1. The presentation of the paper is extremely high quality. The introduction and related work sections are comprehensive, motivate the problem well, and clearly delineate the contributions provided in the paper. The methods section is very clearly described and is simple to follow for those familiar with the field of generative 3D. I view this as very important because it is much easier to glean information from the paper, and future work in text-to-3D (potentially for avatar generation) would be much more likely to build off of this method.
2. The contributions are clearly stated, and to the best of my knowledge they are novel, and they seem like they could be relatively significant.
- The idea of integrating ControlNet together with a score distillation loss makes intuitive sense for gaining better control over generated 3D assets, and I have not, to the best of my knowledge, seen it proposed elsewhere. Maintaining control over the structures which are generated from text-to-3D, including ensuring they have higher quality and less noisy geometry, is a very important problem in text-to-3D generation and I view this solution as potentially having a large impact on those working in this field.
- While balancing losses between an editing prompt and original prompt seems simple, it does not seem to be used in other text-to-3D methods and seems like it could be useful for text-based editing of already fitted representations in other applications.
3. I find the ablation study to be high quality, and ablate all of the parts of the method which are relevant: introducing ControlNet into the SDS loss, the representation chosen and coarse-to-fine optimization, and the hyperparameters for balancing the editing losses. These are all clearly ablated, along with other smaller contributions such as textual inversion for generating the back of the head.
Weaknesses
In my opinion, there is only one main weakness with the paper, which is that the evaluations and comparisons to baseline methods do not seem complete.
1. There are no quantitative comparisons. While the user study is insightful, methods like DreamFusion proposed the CLIP-R metric. Is there a reason why this was not used in this paper? Does the metric do a poor job at quantifying what it is trying to measure? Additionally, some quantitative results for evaluating the editing quality would be useful: CLIP-R (or something like it, if CLIP-R is not useful) can be used to ensure that the identity is still preserved despite editing with an additional prompt (and perhaps also show that the rendered images are pushed closer to a desired fine-grained editing prompt).
2. The baseline methods are not evaluated entirely fairly. For example, I don’t see any of the baseline methods evaluated for editing quality. What happens if one of the generated examples (ex: those from Fig 4) go through a fine-grained text-based edit? How much will the identity be degraded or the edit not be done sufficiently? I think this is extremely important to show because without it, I don’t know what is the state-of-the-art that is currently being improved on and thus don’t know if the proposed method is even better.
3. It is mentioned that the optimization of the proposed method is “more stable” (L244-245) than baseline methods. However, this claim is not justified anywhere. This would be an extremely important contribution for text-to-3D (where optimization is notoriously brittle), so some quantification of this would be very impactful if it could be shown. For example, ranges of hyperparameters trained with which still lead to a good solution, or error bars on generated results, or failure cases.
Questions
I do not have additional questions brought up (see weaknesses for evaluations questions). The paper is exceptionally clear in describing the method and contributions.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The paper adequately addresses the limitations of the proposed method. The first limitation, which mentions that the results are “non-deformable” is insightful as this was a question which is natural to ask considering that the FLAME template is a deformable model for heads. Additionally, the generated results still are not completely photorealistic, such as those representations which have been fit from captured data. Some additional failure cases of the method would be interesting to see in order to understand better what the limitations are of the method and which piece would be the biggest bottleneck for someone who wanted to integrate this work into their application.