Creating realistic avatars from a single RGB image is an attractive yet challenging problem. Due to its ill-posed nature, recent works leverage powerful prior from 2D diffusion models pretrained on large datasets. Although 2D diffusion models demonstrate strong generalization capability, they cannot provide multi-view shape priors with guaranteed 3D consistency. We propose Human 3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion. Our key insight is that 2D multi-view diffusion and 3D reconstruction models provide complementary information for each other, and by coupling them in a tight manner, we can fully leverage the potential of both models. We introduce a novel image-conditioned generative 3D Gaussian Splats reconstruction model that leverages the priors from 2D multi-view diffusion models, and provides an explicit 3D representation, which further guides the 2D reverse sampling process to have better 3D consistency. Experiments show that our proposed framework outperforms state-of-the-art methods and enables the creation of realistic avatars from a single RGB image, achieving high-fidelity in both geometry and appearance. Extensive ablations also validate the efficacy of our design, (1) multi-view 2D priors conditioning in generative 3D reconstruction and (2) consistency refinement of sampling trajectory via the explicit 3D representation. Our code and models will be released on https://yuxuan-xue.com/human-3diffusion.
Paper
Similar papers
Peer review
Summary
The paper presents a pipeline for creating a 3D model of a full body avatar given a single input image. While the previous methods show a single image to 3D using a 2D diffusion model such as ImageDream, as a 2D model, they suffer from 2D inconsistencies. This paper combines the benefit of both the large-scale 2D pre-trained multiview image model as well as the 3D consistent generative model (here they used gaussian splatting as such a representation). Given a single input image, the method first creates 4 orthogonal views of the input using 2D ImageDream. While this creates some realistic views, those views might be inconsistent. To create more consistent views, the method renoises the image and pass it to 3D generative model based on a gaussian representation to create a 3D consistent version of the 2D output. The output from the 3D generative model is 3D consistent but the texture might not be as good as the 2D one. So the model finally redecodes the 3D rendering with a 2D model to create the final 3D-aware rendering. Both the 2D ImageDream and 3D gaussian-based generative models are trained jointly with a tightly coupled manner. To enable the tight coupling, the 3D generative model also takes a noisy image as well as a time step t, just like the 2D counterparts. The 3D generative model part is built on top of the U-Net decoder from LGM, which takes a 4-view image and produces gaussian splatting parameters. The model is trained on a combined dataset of 6000 human scans, which are converted to rendering images via Blender. The paper demonstrates convincing results on multiple challenging datasets (Sizer, IIT-Human, and GSO) with extensive numerical / qualitative comparisons.
Strengths
* Convincing results are demonstrated on challenging datasets of Sizer, IIT-Human, UBC Fashion and GSO (Google Scanned Objects) * While the paper mostly focuses on humans, the method itself is fairly generic and could also work on general objects (some results are shown) * extensive numerical and qualitative comparisons are provided and the proposed method performs better compared to the previous work * While many previous methods used mostly 2D generative priors, the paper shows how to use both 2D and 3D generative models to create a better result. * Paper is overall easy to follow and good amount of technical details are provided (but not all).
Weaknesses
* the paper provided most of the technical details but some key details are still missing such as the exact architecture of the noise conditioned gaussian based 3D generative model and the timing of how long it will take to process an image. * the overall pipeline seems fairly complicated to need to train both 2D and 3D generative models and that they rely on 3D scans, which are fairly limited for humans (only 6000). * the paper text has some English grammar errors and typos and could benefit from proofreading.
Questions
* How long will it take to process an image? * How is X^tgt_{t} exactly generated? In Fig2, it looks like the noisy images have some structured noise rather than a regular gaussian noise added. Why is that? * The visualization of the used views in Figs9 and 10 are inconsistent. For some subjects only front views are shown and other subjects only back views are shown rather randomly. Please make them consistent or show both. * The following papers are relevant for citation: Generative Novel View Synthesis with 3D-Aware Diffusion Models (ICCV2023) ReconFusion: 3D Reconstruction with Diffusion Priors (CVPR2024) ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image Typos * line225: typo, extra period after "reconstruction" * Figure 6: Visualization "of" intermediate sampling steps * line781: Follow[ing] [71]
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The paper discusses both the broader impact and the limitations. Some more limitations could be clarified: * Currently, limited materials are supported (e.g., shiny green dress on the right col of Fig13). * Fine-scale details such as stuffed toy furs (e.g., first image of Fig14) seems to have blobby 3D details likely from the 3DGS, which seem not just the problem of the low resolution generator.
Summary
This paper proposed an image-conditioned 3D-GS generation model for human 3D reconstruction. 2D diffusion models fall short in offering 3D consistency for multi-view shape priors. To address this, the authors introduce a method that combines the strengths of 2D multi-view diffusion and 3D reconstruction models to create avatars with explicit 3D consistency. They propose an image-conditioned 3D Gaussian Splats model that uses 2D diffusion priors to enhance the 3D reconstruction and guide a 2D reverse sampling process, resulting in geometry and appearance. It’s capable of joint training a 3D generative model and a 2D multi-view diffusion model end-to-end.
Strengths
1.The motivation and insight behind this paper are reasonable. I agree that only by obtaining 2D priors with good 3D consistency can one achieve high-quality 3D Gaussian reconstruction results. 2. An interesting idea of joint training 2D diffusion model and 3D Gaussian end-to-end. 3. The paper is well-written and easy to follow.
Weaknesses
1. The resolution supported by the model is too low. 2. By observing the qualitative results, I don't think the method has superior performance in geometry or texture; it seems to have a significant gap compared to existing methods that were not compared. However, the quantitative metrics are surprisingly good, which puzzles me. 3. Comparative experiments lack comparisons with many other works. I suggest adding comparative experiments with the following studies: 1.TeCH: Text-guided Reconstruction of Lifelike Clothed Humans（3DV 2024） 2. HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion（CVPR 2024） 3. Human-SGD: Single-Image 3D Human Digitization with Shape-Guided Diffusion（SIGGRAPH Asia 2023） 4. ECON: Explicit Clothed humans Optimized via Normal integration（CVPR 2023） 5. ICON: Implicit Clothed humans Obtained from Normals（CVPR 2022） 6. FOF: Learning Fourier Occupancy Field for Monocular Real-time Human Reconstruction（NIPS 2022）
Questions
1. How long is the training period? Is it possible to encounter the phenomenon of Gaussian overfitting before the diffusion model has converged? If it occurs, how is it resolved? 2. What is the inference speed like, and does it have an advantage compared to other methods? 3. The multi-view human generation results of your 2D Diffusion model should be displayed.
Rating
4
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
Limited reconstruction quality and lack of comparison results with state-of-the-art methods.
### **Q1: comparison with TeCH and FoF** We appreciate for the feedback. As recommended by the reviewer, we test additional baselines TeCH and FoF. We are happy to provide the additional results on FoF. We use the SMPL-X estimation from ECON to serve as body prior for FoF. We augment the quantitative evaluation tables as follows: Method|Published at|SMPL prior|CD(cm)|NC|F-score :-:|:-:|:-:|:-:|:-:|:-: *FoF*|NeurIPS2022|Yes|5.36|0.685|0.195 ECON|CVPR2023|Yes|3.52|0.749|0.278 Our|-|No|**1.35**|**1.38**|**1.31** It clearly shows that the FoF fails to reconstruct the clothed human accurately, and **further proves that our proposed approach achieves SOTA reconstruction performance quantitatively**. For TeCH, despite the protracted and costly inference process, we consistently obtain textured meshes with extremely noisy surfaces. For visual examples, we kindly ask reviewers to check the result Fig. 6a) in PuzzleAvatar[1] which is produced by the same author of TeCH. **We have also consulted the authors of TeCH and they confirmed us that TeCH indeed struggles with producing smooth surface in lots of cases**. For quantitative comparison, we run TeCH on the same 8 unseen subjects on CustomHuman and report the results below. Testing TeCH on more datasets is not possible as it takes 6h/image and can only run on A100 with 80GB memory. It can be seen in the table that the the normal consistency score is significantly lower than our method, which is consistent with the visual results. CustomHuman|Published at|SMPL prior|CD(cm)|NC|F-score :-:|:-:|:-:|:-:|:-:|:-: *TeCH*|3DV2024|Yes|3.37|0.64|0.31 Our|-|No|**1.03**|**0.85**|**0.66** [1] PuzzleAvatar: Assembling 3D Avatars from Personal Albums
### **Q2: your method has advantage in large loose skirts and children, but not compare frontal face views** We understand the concern that the reviewer has about the performance on the facial features reconstruction. We regret our previous rebuttal in not including more direct frontal views in the rebuttal PDF. We hope the following explanation will satisfactorily address the reviewer's concerns: Firstly, we wish to highlight that our user study, which included views from 45 degrees frontal right and front left, demonstrates a preference for our approach by **80.3%** of participants over SOTA baselines such as SiTH, SiFU, ICON, and ECON. Additionally, the side views provided in the rebuttal PDF illustrate that our method surpasses these baselines in facial appearance (facial color, hair, and helmets) and geometry (eyes, noses, and hairstyles). We appreciate the reviewer’s observation regarding superior facial detail in some baseline models, especially as depicted in Figure 5 of the SiTH paper. In response, we offer the following clarifications: 1) *Input Resolution*: Unlike SiTH which uses 1024x1024 as input, our approach operates at a lower 256x256 resolution due to model capacity and training cost considerations. We believe that employing a **higher-resolution multi-view diffusion model** capable of processing images at 512x512 would significantly enhance detailed facial regions. 2) *Underlying SMPL prior*: Unlike baselines which estimates SMPL to provide the body shape prior, our method doesn't rely on the SMPL template. Thus, **our approach has no additional information regarding the detailed face and hand geometry from SMPL**. However, we argue that estimating SMPL accurately from real-world images is still an open challenge. As illustrated in rebuttal PDF Fig.1, the inaccurate SMPL template brings disadvantage in representing loose clothing. 3) *Training data*: The aforementioned methods rely on ground-truth geometry (SDF) information, providing significant supervision on face geometry reconstruction. We rely only on RGB information, which is more flexible and allows using multi-view image and video datasets. In this paper, we propose a **general framework for monocular reconstruction** that can handle particularly challenging scenarios e.g. loose clothing, using flexible 3D-GS representation. Our framework elegantly combines 2D multi-view diffusion model with 3D-GS generation model. Reviewer iJQG highlights that our framework extends beyond human reconstruction, with the potential for generic object or 3D face reconstructions. We demonstrate that our framework obtains better overall human reconstruction but one could also apply our method to further improve face and hand reconstructions as well. We agree that accurate facial reconstruction is important for 3D human reconstruction, but we also want to emphasize that overall accuracy including clothing also is crucial for realistic avatar creation. **We hope reviewer could value our paper not only on the facial results but on the novelty and generality as well**. Given the reviewer's positive feedback acknowledging the **reasonableness of our motivation**, the **interesting nature of our idea**, the **quality of our presentation**, and the **demonstrated performance advantages in scenarios involving large loose skirts, children, and anime characters**, we want to understand if there are any additional concerns that might prevent the acceptance of our paper. We are committed to addressing any further issues to ensure our research meets the high standards expected for publication.
We are thankful for the feedback and the increased score of our submission. Regarding the limited resolution of 2D multi-view diffusion model, we wish to highlight that the pre-trained 2D multi-view approach is designed to **ensure robust generalization**, a key advancement over prior baselines as demonstrated in Fig.1 of our rebuttal PDF. Importantly, **our proposed framework is not bounded to any single model**; we initially utilized ImageDream[1] (256x256), the SOTA **pre-trained model available during our development phase**. This choice underscores our model's adaptability, not its limitation. As higher resolution models such as MVDiffusion++[2] and CAT3D[3] (both 512x512, not public yet) become available, our approach is fully capable of integrating these advancements, further enhancing its applicability and performance in future applications. [1] ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation\ [2] MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction\ [3] CAT3D: Create Anything in 3D with Multi-View Diffusion Models
Summary
The paper introduces a framework that combines 2D Multi-view Diffusion model and Gaussian Splatting to achieve the task of 3D clothed human body reconstruction from a single view. The focus of the paper is to deal with the 3D inconsistency present in 2D multi-view diffusion models.
Strengths
A novel framework that has reasonable motivations. The ablation studies support the various components introduced by the authors. The method appears to be robust to various different input data.
Weaknesses
Point-to-surface (P2S) metric, widely used in established works like PIFu, PIFuHD, ICON, and ECON, is not used in this paper. In terms of the resolution of the generated meshes (especially for facial features), the proposed model seems to perform poorer than what has been observed in SOTA like ECON or ICON. Can the authors provide a comparison of the proposed model with the SOTA methods but show only the geometry and not the texture?
Questions
Are all the SOTA methods trained with the same training set (with the same human subjects) as your model? What is the inference time required to generate each mesh, and how does that compare against the existing SOTA methods?
Rating
5
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
No potential negative societal impact. I hope the authors address the aforementioned concerns during the rebuttal.
### **Q1: additional P2S metric** We are happy that we address reviewer's concern. --- ### **Q2: facial features comparison in Rebuttal PDF** We understand the concern that the reviewer has about the performance on the facial features reconstruction. We regret our previous rebuttal in not including more direct frontal views in the rebuttal PDF. We hope the following explanation will satisfactorily address the reviewer's concerns: Firstly, we wish to highlight that our user study, which included views from 45 degrees frontal right and front left, demonstrates a preference for our approach by **80.3%** of participants over SOTA baselines such as SiTH, SiFU, ICON, and ECON. Additionally, the side views provided in the rebuttal PDF illustrate that our method surpasses these baselines in facial appearance (facial color, hair, and helmets) and geometry (eyes, noses, and hairstyles). We appreciate the reviewer’s observation regarding superior facial detail in some baseline models, especially as depicted in Figure 5 of the SiTH paper. In response, we offer the following clarifications: 1) *Input Resolution*: Unlike SiTH which uses 1024x1024 as input, our approach operates at a lower 256x256 resolution due to model capacity and training cost considerations. We believe that employing a **higher-resolution multi-view diffusion model** capable of processing images at 512x512 or even higher resolutions would significantly enhance the detail in the reconstructed facial regions. 2) *Underlying SMPL prior*: Unlike SiTH, SiFU, ICON, and ECON which estimates SMPL to provide the body shape prior, our method doesn't rely on the estimated SMPL template. Thus, **our approach has no additional information regarding the detailed face and hand geometry, which can be directly provided by SMPL**. However, we argue that estimating SMPL accurately from real-world images is still an open challenge. As illustrated in rebuttal PDF Fig.1, the inaccurate SMPL template can also bring disadvantage in representing loose clothing. 3) *Training data*: The aforementioned methods rely on ground-truth geometry (SDF) information, providing significant supervision on face geometry reconstruction. We rely only on RGB information, which is more flexible and allows using multi-view image and video datasets. We believe this analysis and the discussion reported here will be beneficial for future works that will embrace our original proposed novel idea, combining 3DGS generation with 2D diffusion models. We also thank the reviewer for acknowledging our results on complex cases are never addressed by other methods, such as loose clothing and children. We agree that accurate facial reconstruction is important for 3D human reconstruction, but we also want to emphasize that **overall accuracy including clothing also is crucial for realistic avatar creation**. In this paper, we propose a **general framework for monocular reconstruction** that can handle particularly challenging scenarios such as loose clothing, using the flexible 3D-GS representation. This framework elegantly combines 2D multi-view diffusion model with 3D-GS generation model. Reviewer iJQG highlights that our framework extends beyond human reconstruction, with the potential for generic object or 3D face reconstructions without altering the model's architecture. We demonstrate in this paper that our framework obtains better overall human reconstruction but one could also apply our method to further improve head, face, hair, and hand reconstructions as well. **We hope reviewer could value our paper not only on the facial results but on the novelty and generality as well**.
### **Q3: setting as SOTA baselines** As we provided the additional quantitative results of our approach trained on the same data as baseline works such as SiTH and SiFU, it clearly shows that our model design can outperforms baseline under the same number of seen subjects during training. As requested by the reviewer, we report the evaluation datasets separately as follows: |Sizer denoise|PSNR|SSIM|LPIPS|CD(cm)|S-to-P(cm)|P-to-S(cm)| |:-:|:-:|:-:|:-:|:-:|:-:|:-:| |SiTH|18.9|0.912|0.063|3.38|3.38|3.37| |SiFU|18.0|0.912|0.068|2.69|2.56|2.80| |Our (Thuman2.0 only)|20.54|0.916|0.060|1.52|1.63|1.41| |Our|21.3|0.928|0.047|1.06|1.05|1.07| |CAPE|PSNR|SSIM|LPIPS|CD(cm)|S-to-P(cm)|P-to-S(cm)| |:-:|:-:|:-:|:-:|:-:|:-:|:-:| |SiTH|22.2|0.908|0.082|3.76|3.75|3.76| |SiFU|22.0|0.907|0.085|3.72|3.70|3.73| |Our (Thuman2.0 only)|21.1|0.908|0.075|2.23|2.19|2.25| |Our|21.5|0.916|0.064|1.89|1.86|1.91| |CustomHuman|PSNR|SSIM|LPIPS|CD(cm)|S-to-P(cm)|P-to-S(cm)| |:-:|:-:|:-:|:-:|:-:|:-:|:-:| |SiTH|20.8|0.915|0.073|2.82|2.81|2.84| |SiFU|20.1|0.899|0.087|3.10|3.08|3.11| |Our (Thuman2.0 only)|21.61|0.909|0.069|2.08|2.08|2.09| |Our|22.3|0.926|0.048|1.03|1.05|1.02| |IIIT|PSNR|SSIM|LPIPS|CD(cm)|S-to-P(cm)|P-to-S(cm)| |:-:|:-:|:-:|:-:|:-:|:-:|:-:| |SiTH|22.7|0.906|0.077|3.31|3.54|3.08| |SiFU|22.6|0.899|0.087|4.24|4.32|4.17| |Our (Thuman2.0 only)|21.8|0.900|0.071|1.67|1.70|1.64| |Our|22.1|0.905|0.065|1.44|1.49|1.39| --- ### **Q4: Inference time** Sorry for the misunderstanding. We reported the comparison in the introduction section of general rebuttal, and we are happy to provide the details here: Our Human-3Diffusion is a diffusion-based feed-forward approach, avoiding the SMPL estimation and test-time optimization required by models like ICON, ECON, SiTH, and SiFU. This approach significantly boosts our model’s inference speed. We provide a runtime comparison on an Nvidia A100 GPU below, detailing the inference time from an RGB image to the final 3D representation: | |Our|SiTH|SiFU|ICON|ECON| |-|:-:|:-:|:-:|:-:|:-:| |Time(s)|22.6|106.2|48.9|60.5|45.3| |VRAM(GiB)|11.7|22.0|12.0|6.3|5.9| For mesh extraction, we use Gaussian Opacity Fields (11.5s, resolution=256) and TSDF-Fusion (14.8s, across 24 views, resolution=256). We will provide comprehensive details in the revised manuscript.
Reviewer's Follow-up Response to Rebuttal
Thank you for your clarifications, I have no further questions. I acknowledge the good quantitative performance and the limitations that you stated. Before the rebuttal, I was deliberating whether to downgrade or upgrade my initial score for your paper, but your rebuttal has helped me understand that the initial positive score is appropriate.
Summary
In this paper, the authors propose to create realistic avatar representations by coupling the 2D multi-view diffusion and 3D reconstruction models which complement each other. Specifically, the 3D Gaussian Splatting (3D-GS) reconstruction leverages the priors from 2D diffusion models and produces an explicit 3D representation. Meanwhile, the rendered images from 3D-GS representations further guide the reverse sampling of 2D diffusion models to improve 3D consistency. Experimental improvements on some examples are achieved to demonstrate the empirical effectiveness of the proposed method.
Strengths
+ How to use one single image to infer 3D structure is an important problem, which can affect many down-streamed applications. + The **Human 3Diffusion** method improves the existing methods in some scenarios. + The paper is well written with nice figures. I can follow it easily.
Weaknesses
+ In Figure 3, as for me, the outputs from Human 3Diffusion are similar to those of SiTH in terms of image quality. Actually, some examples in Figure 8 even show that SiTH can produce better results than the proposed method, such as the right hand is more reasonable on the 4th row. + Although the authors argue why they do not evaluate on the CAPE dataset, which is a standard testbed for previous methods, I think the Sizer and IIIT datasets are also not perfect. For example, as shown in Figure 7, some inputs contain the noisy ground part while the SiTH method produces cleaner background than Human 3Diffusion. I guess this also might yield worse metrics for SiTH in Table 1. So I suggest the authors additionally report the metrics on CAPE and CustomHuman datasets by following the evaluation settings of SiTH such that the readers can better perceive the empirical improvements from the proposed contribution. + Some wordings are a bit too strong. For example, in L198, the authors state that "guarantee the 3D consistency...". I understand that the developed 2D Multi-view Sampling could improve the 3D consistency, but it is hard to say it can fully address this problem.
Questions
+ In Table 4, why only using FID for evaluation? Could the authors provide the results with other metrics like PSNR, SSIM and LPIPS? + How many camera views and how many examples are used for evaluation? Will the authors release their exact evaluation settings in the future? + It seems that the results of Table 2, 3, 4 are performed under different settings. Could the authors align all ablated revisions with the settings used in Table 1? It can better help the readers to comprehend the importance of each key technical idea compared to previous baselines. I am looking forward to hearing about these points from the authors.
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors address the limitation and societal impact in the appendix.
We are happy that most of the reviewer's concerns have been addressed. --- ### **Q1: Different SiTH numbers between our table and SiTH paper** For the evaluation, we use the official inference pipeline from SiTH github repo to obtain the reconstruction and use the official alignment script to align with GT meshes before evaluation. The only difference is that our test images are rendered from **perspective cameras** while SiTH which uses **orthographic cameras**. Due to this difference, there is some pose offset from SMPL fitting results, leading to the gap. It is worth mentioning that estimating 3D SMPL on perspective Images can lead to less faithful 3D, as also discussed in CLIFF[1] and SPEC[2]. We also observe similar artifacts in the SMPL estimation, such as legs bending backwards, as shown in our **rebuttal pdf**. Moreover, we **rerun SiTH on orthographic renderings and reproduce similar numbers as reported in the SiTH paper**. We will clarify this in the experiment section. We really appreciate the great results produced by SiTH and SiTH's author for helping in figuring out the evaluation settings. Nevertheless, it is crucial to acknowledge that **nearly all real-world images are captured using perspective cameras**. Thus, we argue that our evaluation setting, designed to handle perspective images, offers a more accurate reflection of performance in real-world conditions. [1] CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation, ECCV2022\ [2] SPEC: Seeing People in the Wild with an Estimated Camera, ICCV2021
Thank you
I thank the authors for providing the detailed explanations. Based on the current response, I would retain my positive attitude.
Thank you for the clarifications and the rebuttal response. After reading the authors’ response and the comments of the other reviewers, I have the following concerns: 1. It's important to stress that a qualified paper needs to do a comprehensive comparison experiment with previous related works. I approve that it is difficult to make a detailed quantitative comparison with TeCH (high-quality geometry and texture), FOF (fast inference for high-quality geometry), and other works within a limited time. Still, I think it is unreasonable not to show the qualitative comparison results. 2. As stated by the reviewer uByW，"your model seems to perform poorer than SOTA baselines". Although the proposed approach has an advantage in large loose skirts, children, and anime characters as shown in the rebuttal PDF, the authors do not compare frontal face views. So, I think this paper does not achieve the sota performance, and I will keep my rating.
One more question
Thank the authors for offering a detailed response. I really appreciate it. Generally, the response addresses my most concerns. However, I am a bit confused why the scores of SiTH in **Q2: Evaluation on new Datasets like CAPE and CustomHuman** do not comply with Tab. 1 in SiTH's paper. Could the authors explain this a bit?
Reviewer's Response to Rebuttal
Q1: Ok, that answered my question. Q2: I looked at Rebuttal PDF (Figure 1). I appreciate the effort, but I actually asked to see facial features of the generated meshes. From what I can observe, I do believe that existing papers (e.g. SiTH, Fig. 5 from their paper) may be able to do this better. Nevertheless, Rebuttal PDF (Figure 1) does show the structural accuracy of your proposed method. Overall, I do not believe my concern here was well-addressed. Q3: The table you showed do somewhat address my concern, but I do not agree with the reasons you cited for not doing this in the first place. In particular, you could have picked other methods that do not use SMPL as baselines. I am also not sure why the evaluation datasets are combined into one during quantitative evaluation. Overall, my concern is partially addressed here. Q4: This is one of my more minor concerns, but I looked at your paper and did not find it. Please specify the line number and the table number. Overall, I find your rebuttal response to be a good effort although I feel it was mixed in terms of convincing me. Hence, I feel it is appropriate for me to retain my original score.
I read the entire rebuttals including all the reviews by other reviewers. The rebuttal sufficiently addressed most of my concerns and I did not find new concerns. I will keep my original rating. The figures (second Ninja turtle example) are still inconsistent though in terms of rendering poses in Figure 1 of the rebuttal PDF.
Thanks for addressing my concerns about comparison methods. However, I still think the qualitative results are not good and the limitation for the input resolution of the propose method is not a valid reason, but rather a shortcoming of the method. Considering the results of the qualitative comparison, I will raise the rating to borderline reject and maintain the negative attitude.
A rebuttal summary for AC and all reviewers
We would like to thank AC and all reviewers for the valuable feedback and active involvement in the discussion. We summarize the merits of our work and additional results for rebuttal and discussion below: ### **Brief summary of our contributions** - Novel method for 3D reconstruction. We introduce a **general framework for image-based 3D reconstruction**, which leverages the strong prior from 2D multi-view diffusion (MVD) models and 3D consistency of explicit 3D Gaussian splatting representation. At the core of our method is a **novel joint diffusion process** that synchronize 2D MVD and 3D generation model which **enhances performance of both models**. - SOTA human reconstruction: Our method achieves SOTA results in human reconstruction and **surpasses 10 baselines (most recent at ECCV2024)** (Fig.1 in rebuttal PDF, Fig.7-8 in paper, Q3 to reviewer RJbF). - Strong generalization: Our design leverages the 2D multi-view model effectively, enabling strong generalization to diverse unseen cases like **large loose clothing**, **animals**, **anime characters**, **Human-Object Interaction**, demonstrated in various figures across the paper and rebuttal (Figs. 5,14-16 in paper and Fig.1 in rebuttal PDF). ### **Brief summary of rebuttal stage** We are very happy to see that there is **no criticism of our method’s novelty and significance**. The concerns raised in the initial reviews focused on the quality of 3D reconstruction, inference time, and the need for more extensive quantitative comparisons with other baselines. We addressed in the rebuttal with following aspects: - We show more qualitative comparisons in the rebuttal PDF, clearly demonstrating that **our Human-3Diffusion outperforms all SOTA baselines**. Furthermore, we conducted a user study with 70 participants evaluating 40 randomly selected subjects. The study showed a strong preference for our method, **with 80.3% favoring it over others**. - We also reported runtime performance at ~22 seconds per image, **significantly faster than the baselines** such as ICON, ECON, and SiTH, which exceed at least 45 seconds due to SMPL estimation and surface optimization post-processes. - We expanded our comparison to include four additional baselines (with code available): ICON, ECON, FoF, and TeCH. Now, our Human-3Diffusion model is **compared against 10 recent SOTA methods**, and our method **consistently outperforming all baselines in 4 challenging datasets**. We are grateful that the additional data and comparisons have addressed most concerns. Reviewers KYnm, uByW, and iJQG maintain a positive attitude to accept our paper, with RJbF acknowledging our enhanced comparison and qualitative results, subsequently raising his/her score. We kindly request that all reviewers consider revising their scores upwards if they are satisfied with our revisions. Thank you again for all the constructive feedback and discussions. Respectfully,\ Authors
Decision
Accept (poster)