HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors

Summary

This paper presents a 3D human reconstruction system that takes a single RGB image and outputs 3D Gaussians, which can render the reconstructed 3D humans to any viewpoint. In contrast to existing 3DGS works, which requires per-instance optimization and cannot be generalized to unseen identities, the proposed system can be used for any unseen identities in zero-shot manner. To achieve this, the authors utilizes 2D mulit-view diffusion model and latent reconstruction transformer with human structure priors. A hierarchical loss, which incorporates human semantic information, is introduced. Experimental results demonstrate the powerful capability of the proposed work.

Strengths

Utilizing diffusion-based generative models to render unseen novel viewpoints is a reasonable choice considering the aiming task is a significantly ill-posed one. The latent reconstruction Transformer effectively incorporates the human structure prior with the generative model’s output. The hierarchical loss could be useful to enforce a consistency between 3D human geometry and appearance.

Weaknesses

1. Unclear writings Overall, technical details are not enough to fully understand the manuscript. 1-1. For example, Sec. 3.3 describes the video diffusion model. I can’t fully get how the ‘video’ diffusion model can be used for the ‘novel-view’ synthesizer. I understand that a video of an object can model different viewpoints when a camera is moving, but there should be a clear justification and reason why the authors chose this generative model for the novel-view synthesizer. In addition, it seems the video diffusion model does not take the target camera pose. Then which novel view is modeled with the video diffusion model given no camera pose? 1-2. Also, for the hierarchical loss, how can the authors access the target human image of certain body parts? Do the authors already have GT body part segmentation? If the authors simply use rendered body part segmentation from GT SMPL meshes, I don’t think that is a good choice as SMPL meshes only model naked bodies, while there could be hair and loose clothes in the image. 2. Lack of comparison to SOTA methods Please compare with more SOTA methods, such as ICON and ECON. 3. Lack of in-the-wild results As the proposed work aims to generalizable 3D human reconstruction, there should be more diverse generalized results. Only three results in Fig. 4 show the results on in-the-wild inputs, which have quite simple poses. Please report more results from input images with diverse **human poses**, identities, and camera pose.

Questions

Please see the weaknesses section.

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

Please see the weaknesses section.

Summary

This paper proposes a generalizable human rendering framework from single images. The proposed method relies on different priors and achieves state-of-the-art results on multiple datasets.

Strengths

- The topic of human modeling/rendering from partial observation is important and interesting. - Thanks to 3DGS, the proposed method achieves a good balance of rendering quality and speed.

Weaknesses

- Qualitative visual results on the rendered images are not satisfactory. It is acceptable that renderings of unseen views of the human can be relatively poorer. It is surprising to see renderings of exactly the input views are also in low quality (Figure 4). This is usually a sign of poor model capacity, which may be attributed to the proposed objective functions or the model itself. - It is not clear how the 'parts' of human are defined. Are they integer based index as in iuv? If so, how to obtain the ground truth body parts labels? Are they provided by the dataset or they have to be pre-computed by some off-the-shelf models. What if there are prediction errors in the 'ground truth' semantic labels? Is the proposed method robust to this kind of errors? I also have doubts on its effectiveness in terms of the overall rendering quality since there is no ablation study provided to show how it improves quantitative/qualitative results. - As presented in Table 2 (b), it seems that the proposed method is very sensitive to geometric prior (i.e. SMPL estimations). Since the model relies on PIXIE for SMPL prediction, what if there are errors in the predictions (most likely for challenging poses or complex clothes/background), how will the proposed method perform under those cases? This on the contrary shows advantages of geometric prior free methods such as LRM and HumanLRM. If there is a benchmark on human rendering with challenging poses, I am afraid those methods will perform better than the proposed one. - I wonder why there are not qualitative comparisons with SIFU but only quantitative metrics? - I'd suggest to change Figure 1 (b) to a different style. The dashes in the graph are misleading and confuse readers as they are in different scales.

Questions

Please see the weakness section.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations.

Summary

This paper proposes HumanSpat, a method that predicts 3D Gaussians from a single image of a human. The method comprises a 2D multi-view diffusion model and a latent reconstruction transformer that integrates human body prior.

Strengths

+ Well-designed generalizable model that incorporates human prior. The model design is intuitive and clear. The fusion of the human structure prior makes a lot of sense. The end-to-end design addresses major limitations in existing works, such as lacking of human prior makes the prediction missing limbs (as in Human lrm) and two stage approaches where the errors in the human pose prediction affects the clothed reconstruction. + Reconstruction time. Powered by 3DGS, the method has minimal reconstruction time and can render novel views at real-time speed.

Weaknesses

- Missing ablation on human prior. The paper should include ablation experiments where there is no human structure prior, as well as experiments where the human structure prior is estimated instead of GT. These experiments can provide insight on the benefit of the specific human structure prior fusion design.

Questions

In Table 1, how was the reconstruction time computed? PIFu is very lightweight and should not take 30 seconds per human.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are addressed.

Summary

- The paper, HumanSplat, focuses on photorealistic novel-view synthesis of humans from a single image. - The key idea is to use a multi-view synthesizer based on SV3D to hallucinate the other views + latent reconstruction transformer to predict the 3DGS. - Importantly, HumanSplat does not use optimization and is much faster than existing methods. - Baselines: Magic, HumanSGD, TeCH - Evaluations are done on Thuman2.0 and Twindom datasets. Metrics: PSNR, SSIM, LPIPS - The proposed methods consistently outperforms the existing methods.

Strengths

- The paper is well-written, organized and easy to follow. - The key idea of multi-view diffusion plus transformer to predict parameters of gaussians in the context of human avatar creation is novel. - An important technical contribution is the archictecture of the latent reconstruction transformer with geometry-aware interaction. - The experiments are done on multiple datasets along with informative ablative studies. - Table. 1 highlights the speed-up gains compared to existing methods. HumanSplat is almost as fast as LRM but utilizes an explict human prior and achieves much better performance.

Weaknesses

- Measuring 3D consistency of SV3D: The novel views generated by fine-tuned SV3D is important for predicting the gaussian parameters. An ablative study on the effectiveness of the novel synthesizer would be really helpful to understand the in-the-wild generalization of the proposed method. 3D consistency could be measure using stereo-based matching or local correspondences. - Qualitative results: Although, Table. 1 showcases better performance than HumanSGD on THuman2.0 dataset, notably in all metrics. The qualitative comparison between HumanSGD and HumanSplat is missing. HumanSGD results demonstrate much better modelling of human skin tone and overall 3D geometry. I understand the code is yet to be released, however, would it be possible to compare on similar stock images? - Minor: consider renaming "project-aware" -> "projection-aware". The writing at times is a little too non-technical, eg. L206-207, consider rephrasing such instances.

Questions

- Understanding the generalizability of novel-view synthesizer. Is it limited to full-body images? Any quantitative insights into 3D consistency. - Qualitative comparisons to HumanSGD on stock images.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes. The authors provide a discussion about utilizing 2D data and handling diverse clothing styles.

Reply to Rebuttal

I thank the authors for their rebuttal. The additional experiments as well as discussions are insightful. I'd like to raise my rating to borderline accept to acknowledge the efforts the authors put into the rebuttal. However, I still want to note on the poor quality of renderings. As a generalizable methods, it is understandable that unseen regions from the input image are poorly generated/rendered. But it is still not a good sign that the model replicates the input view poorly with blurry and unnatural details --- this may, again, indicate something is wrong with the methodology. The comparison results by HumanSGD, as provided in the rebuttal, however does a much better job replicating the input view. I'd like to acknowledge this paper's strength as well as note the weakness for AC's reference when making the final decision.