OccFusion: Rendering Occluded Humans with Generative Diffusion Priors

Summary

This paper presents OccFusion, a 3D human avatar creation system that combines 3D Gaussian Splatting (3DGS) and 2D diffusion models to effectively render occluded regions. To this end, three stages are designed. First, in the initialization stage, complete human masks (without occluder) are obtained using a pre-trained diffusion model. Then, in the optimization stage, 3DGS is optimized with score distillation sampling (SDS) to recover occluded regions. Finally, in the refinement stage, in-context inpainting is designed to improve the rendering quality.

Strengths

What the authors try to address (recovering 3D human avatar from human images with occlusions) is an important and unexplored problem. The authors’ design, which incorporates pre-trained diffusion models, is a reasonable choice.

Weaknesses

1. Overall, the rendering quality is bad. This is applicable even for visible areas. Fig. 5 and 6 shows that the results are quite blurry especially on the OcMotion dataset. 2. I don’t think the rightmost binary masks in Fig. 4 are consistent. All of them obviously have different pants silhouettes. Given these inconsistent masks, which masks did the authors use to optimize 3DGS? 3. Overall, paper writing should be polished. --- post rebuttal --- The authors addressed my concerns well. I'm raising my rating.

Questions

1. What makes the result look so blurry even for visible areas? 2. In Fig. 5, the second column and second row results do not have visible humans in the reference. 3. How much of the mask (M) and complete human mask (\hat{M}) are accurate? Also, how much of the optimization stage is robust to wrong M and \hat{M}? 4. Fig. 3 shows a pose-to-image generation example, while the actual use case is (image+pose+mask)-to-image as in Fig. 4. In the case of Fig. 4, the pose simplification of FIg. 3 is still necessary? 5. Fig. 5 shows that the leftmost column is an input. What do the authors mean by input? As far as I understand, 3DGS is optimized to replicate images of a single person in the training set, and there is no input to the 3DGS. Or the 3DGS takes images of any person and can render images of that person with novel poses and viewpoints?

Rating

5

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

The major limitation is the rendering quality. Most of the rendered images are quite blurry.

Summary

This paper introduces OccFusion, a method for rendering occluded humans. Similar to other 3DGS-based human rendering methods, OccFusion optimizes a set of 3D Gaussians to improve training and rendering speed. OccFusion proposes adopting generative diffusion priors to ensure complete and high-quality renderings to aid in reconstruction. OccFusion was evaluated on ZJU-MoCap and challenging OcMotion sequences, and it achieved state-of-the-art performance in rendering occluded humans.

Strengths

The 3-stage strategy is efficient in handling occluded humans. During the initialization stage, inpaint occluded human visibility masks into complete human occupancy masks. During the optimization stage, initialize a set of 3D Gaussians and optimize them by SDS in both the posed and canonical space. During the refinement stage, inpaint unobserved regions of the human with context from partial observations and renderings from the previous stage. This method outperforms the state-of-the-art in rendering humans from occluded videos.

Weaknesses

Lack of discussion about NeRF in the Wild, Ha-NeRF, Gaussian in the Wild and NeRF On-the-go.

Questions

It is better to include more video results and add diversities of datasets.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations.

Summary

This paper proposes a method for reconstructing gaussian-based human avatars from occluded captures. The gaussian avatar model is based on GauHuman, which is optimized in multiple stages, including using a diffusion based prior in the canonical space to recover the complete human. In the first stage, a consistent human silhouette is generated using "z-buffered" pose key points. Next, this silhouette is filled in with textures using SDS which are then further refined by a diffusion model (same as one used for SDS) using the phrase - "the same person standing in two different rooms". Finally, these inpainted images are used to refine the color of the gaussians to generate the final avatar. Both quantitative and qualitative results demonstrate the effectiveness of the proposed method over prior work. While each component of the model is not novel in itself, the overall method is, and it works well.

Strengths

1) The use of the diffusion model to generate the silhouette, inpaint it in the canonical space and use it as pseudo ground-truth is novel. 2) The architecture is well motivated, with the purpose of each component very intuitive and well ablated. 2) The evaluations seem thorough for the most part, apart from the ablations. 3) The paper is well written.

Weaknesses

1) In Figure 6, it would be nice to show results on both subjects for all experimental configurations (Exp A to E). In its current form it is incomplete.

Questions

1) I'm curious as to whether the authors tried inpainting in the deformed space as well? If yes, does it work better or worse. It will be great if the authors could include those results as well

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

OccFusion proposes an approach to model human bodies that fail under occlusion in monocular videos. The authors utilize 3D Gaussian splatting for efficient rendering and leverage pretrained off-the-shelf image diffusion models as 2D priors. Their approach involves a three-stage training process, sequentially refining estimates to achieve precise 3D human representations. They demonstrate superior performance compared to previous methods, enhancing one of the baselines to better handle occlusion scenarios.

Strengths

1. The paper is well-written and provides valuable engineering insights, particularly through the extensive use of various off-the-shelf models. The authors demonstrate proficient application of these methods to their tasks. 2. As for another strength, they enhance the GauHuman baseline to handle occlusion correctly, a significant contribution worth highlighting. 3. The ablation study is informative and visual, effectively supporting the narrative. 4. The qualitative study shows the method's superiority, with well-justified reasons in the text.

Weaknesses

1. Some important details are either omitted or mentioned too late in the text. Below are some examples: 1.1. While the extensive use of off-the-shelf models is well justified, it is only superficially discussed in the corresponding technical sections. For instance, the introduction does not mention the use of image generative models at the initialization stage to improve the masks. This is a core aspect of the work, and mentioning models like DreamFusion, Stable Diffusion, and ControlNet should be done when discussing the three stages, preferably in the introduction. 1.2. Figure 2 (method overview) is cumbersome and not very instructive. It does not clarify the three-stage training, and its understanding relies heavily on the detailed ablation study. I recommend reorganizing Figure 2 to include all off-the-shelf models used in the training pipeline. Even a plain list of inputs and outputs with the side models used at each stage would be more descriptive than the current Figure 2 and the introductory motivation. 2. Obscure training details 2.1 Training time: authors insist throughout the text that the training time of their method is 10 minutes. Assuming the monocular video is the only input, the result must depend on the length of the video, yet it is never discussed. 2.2 It is unclear why 10 minutes is enough for good performance. Would estimates improve if trained longer? 3. Results 3.1 In Fig.6 Exp.A authors demonstrate that OccGauHuman estimate has some bodyparts ripped apart. This should not happen if initial splats are distributed all over SMPL mesh and "densification and pruning of splats is disabled. (line 137)". How can authors explain this behavior? 3.2 In Table 2 GauHuman and OccGauHuman are trained for 10 and 2 minutes, correspondingly. This is unclear how estimates improve when OccGauHuman is trained for 10 minutes, as GauHuman. 3.3 Continuing the issue 3.2, if all experiments with OccGauHuman model are also taken with 2-min limit, this makes the quantitative comparison unfair. maybe it is simply undertrained. 4. (minor) OccGauHuman. In Sec.3 authors should say explicitly that this method they propose is not the part of the contribution but the way to build a better baseline and a starting point for their own method. 5. (minor) Throughout the paper authors say "gaussian" referring to the splat or blob. It is better to avoid such namedropping, since Gaussian is a very broad term that can imply many different things and is itself an adjective and does not fit in the sentence properly.

Questions

I would like authors to answer to the issues I raise in Weaknesses. If I find them constructive, I would raise the rating.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

-

Summary

This work presents a Gaussian-based approach to reconstruct 3D human poses from occluded monocular video. Building upon the success of previous methods like SDS, the paper suggests introducing a pre-trained diffusion prior to complete the occluded areas and further refine the missing appearance. The framework is evaluated on public datasets, including ZJU-MoCap and OcMotion, and shows improved efficiency and appearance quality.

Strengths

- This paper tackles a relatively new and practical problem of 3D human reconstruction from occluded monocular video. - The introduction section provides a clear motivation and background for the proposed method. - The authors conduct comprehensive experiments to validate the efficiency of each component of the framework.

Weaknesses

- The ablation study could be more extensive. The authors mention that "applying SDS on RGB images causes appearance inconsistency" in line 192. It would be helpful if the authors could provide some visualizations to support this claim. - If the optimization step can generate consistent appearances, the refinement stage could also introduce inconsistencies in the occluded areas, as there is no cross-frame constraint during the inpainting process.

Questions

- The supplementary videos lack method names, making it difficult to identify the final result of the proposed "OccFusion" method. It is unclear why some blur exists in the visible areas, as evident in cases like "ocmotion_0011_1" and "ocmotion_0011_2".

Rating

5

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

No negative societal impact.

Reply to Rebuttal

I thank authors for their instructive and well detailed answer. All of the issues I raised are well addressed.