DELIFFAS: Deformable Light Fields for Fast Avatar Synthesis

Summary

The paper introduces a method for real-time rendering of animatable full-body avatars. The main contribution is a hybrid mesh-lightfield representation: two surfaces are produced with an off-the-shelf motion-to-surface model, and normals of those surfaces are then used as input to two UNet, the output of those is fit into the lightfield MLP as features. Method is near real-time and runs at 30FPS on a GPU for 1K resolution. Quantitative and qualitative results over recent baselines demonstrate that the method is on par with state-of-the-art.

Strengths

- Paper is well-written and is easy to follow. - Proposed method seems technically sound, and the two-layer surface formulation fits really well with the lightfield-based neural rendering. - Apart from the quality, proposed formulation runs at 31fps for a 1K resolution image, and only requires 1 MLP evaluation per ray. - Quantitative and qualitative evaluation is thorough, the choice of baselines is reasonable (although there are some potential issues, please see below). Ablation study is present and seems to suggest that the proposed two-layer formulation is less sensitive to the quality of the mesh (which can be considered a contribution in itself).

Weaknesses

- There are not too many details on how the deformed mesh is obtained, but assuming it is only conditioned on the motion, it is unclear whether providing normals of the resulting mesh should be sufficient to reconstruct the full image: it would seem that the state of person's clothing cannot be fully described by the pose or sequence of poses. - On a similar note, it looks like the underlying mesh actually seems to have some measurable effect for the method performance: judging from numerical results in Table 2)? From Figure 6, it is unclear if the quality comparison is done for both models trained with perceptual loss or not? Yet, authors do not really provide any details on how to build the underlying motion-to-mesh model. - Claim "This allows perceptual supervision ... compared to previous approaches" is incorrect, e.g. [Habermann'15], [Bagautdinov'21], DVA [Remelli'22] should support full-image supervision with arbitrary perceptual losses. Generally, it is a bit unclear how much of the qualitative improvements are coming from the use of perceptual loss (which I would not consider a contribution of this work).

Questions

- Does the mesh model take as input anything apart from the pose? Would it be sufficient to capture non-pose-dependent deformations? - Do you expect the quality of the method to degrade on grazing angles? - Is the number of parameters between a single- and two- layered model the same? In the two-layer case, there are two UNets, which could also explain the boost in the performance? - Could you please confirm that the qualitative comparison (Figure 6) is done consistently for both single- and two-layer models trained with/without perceptual loss? Might be a good idea to be more explicit about this in Table 1 as well.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Authors did not really discuss limitations. Suggestions: - inability to capture realistic dynamics due to information-defficient inputs. - artifacts on grazing angles due to incoherent features between the two layers.

Summary

This work proposes a method for human avatar reconstruction from multiview video data. This work leverages deformable light fields to model the geometry and texture. Experiments show that this method outperforms the existing methods in terms of novel view and novel pose synthesis.

Strengths

1. The idea of using light field to improve computation efficiency is novel and well-motivated. It can overcome some of the drawbacks of NeRF, such as the heavy computation cost for point sampling. 2. Using mesh as geometry representation is also reasonable as it is naturally compatible with conventional rendering engines, which could open up many application possibilities.

Weaknesses

1. Although the framework is novel, the idea of using differentiable rasterization for animatable human avatar creation has already been explored in prior work[1], please explain the difference between this work and [1]. 2. There may exist a risk of overfitting because this work only experiments on one dataset, it could be better to add more experiments on the commonly used benchmark dataset such as ZJU-Mocap. 3. Efficiency is one of the most important contributions of this work, the authors could report more numerical results to verify its rendering speed, and it’s better to compare it with [1]. 4. The setting for ablation study (line 269) is confusing, I don’t understand why other variants are not trained with perceptual loss. I think this loss term should not be skipped except for studying the necessity of optimization objectives. [1] UV Volumes for Real-time Rendering of Editable Free-view Human Performance

Questions

The training data are multiview videos, is the proposed method possible to be trained on monocular videos?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Addressed.

Summary

The paper introduces DELIFFAS, an innovative method for generating controllable and photorealistic digital human avatars in real-time. This system utilizes a deformable two-surface representation to parameterize a surface light field, deforming two surfaces according to a deformable mesh model, allowing the light field to be driven by skeletal motion. This approach significantly increases inference speed, enabling the entire image to be rendered during training, unlike previous methods. This faster rendering time also allows for full-image level perceptual supervision, in contrast to earlier approaches, which could only supervise individual pixels or small patches due to slow runtime.

Strengths

Key strengths of the paper: - The proposed method DELIFFAS can generate controllable and photorealistic human avatars in real-time. - The use of a deformable two-surface representation to parameterize a surface light field improves both accuracy and efficiency of appearance synthesis. - The efficient design of the neural architecture and light field representation allows for integration into the graphics pipeline, resulting in real-time performance. - Due to the high inference speed, the system can render the entire image during training and employ full-image level perceptual supervision.

Weaknesses

Some potential weaknesses of the paper: - Limitations in Pose Variety: It is unclear how well the model would perform with extremely unusual or rare poses, or if it would be able to accurately generate and maintain details under these conditions. - Dependency on Underlying Mesh Quality: While the authors have demonstrated that their method can tolerate coarser geometry, the accuracy and realism of the output may still depend on the quality of the underlying mesh. - Robustness to Complex Clothing: While the authors mention that their method can handle "challenging loose garment types," there may be limitations when dealing with more complex or unusual clothing, especially if such examples are not well represented in the training data. This can be often the case where common clothing could exhibit highly glossy or specular lighting effects such as silk.

Questions

- How does the DELIFFAS model perform when exposed to extreme poses or movements not well-represented in the training data? Are there specific types of movement or pose it struggles to accurately represent? - Could you elaborate on the computational resources required for real-time implementation of the DELIFFAS model? A100 is a quite heavy GPU configuration, what happens if tested on other GPU configurations for the performance? - How would the system handle more complex or unusual clothing types, especially those not well-represented in the training data? - The method seems to rely heavily on the quality of the underlying mesh model.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Discussed above, in the weakness section.

Summary

This paper aims to learn digital human body avatars from multi-view videos. To achieve both photorealism and fast inference speed, the authors introduce a new representation based on a surface light field. The surface light field is attached to a drivable human mesh model, and is conditioned on skeleton motion. Such a representation can be easily defined on texture atlas, and efficiently rendered with standard rasterization pipelines. Overall, the proposed representation allows perceptual supervision and real-time rendering, resulting in superior performance over existing baselines.

Strengths

1. The proposed method enables real-time rendering of full-body human avatars with a resolution of 1K. This is a nice property for many down-stream interactive applications. 2. The proposed representation further allows perceptual supervision on the full image, which is crucial for learning high-quality appearance details like cloth wrinkles. 3. All important implementation details are adequately discussed in the main paper and the supplemental document. For example, in line 179-180, the authors describe how they handle corner cases where a ray only intersects with the outer surface.

Weaknesses

1. Although the proposed representation is based on a light-field defined by two deformable surfaces and demonstrates superior performance over the single-surface baseline Sec.4.3, it is still tightly bounded to the underlying mesh model. According to Line 149, the shell bounded by two surfaces has a thickness of only 3 cm, which is relatively small. This means that the underlying mesh model should be a close approximation of the real avatar surface. Therefore, I am suspicuous whether other human body templates (like SMPL) can be used in the proposed method, although the authors claim in Line 122 that "any other deformable surface representation could be used as well". 2. Although the proposed two-surface representation is nice for modeling dynamic texture, the geometric deformation is not modeled. In Line 118, the authors mention that they assume "a skeletal motion-depdent deformable mesh surface is given". This can be obtained for the task of novel view synthesis, as mesh tracking is a key step in data preprocessing. However, this is very challenging for novel pose synthesis, especially for loose garments like a long dress. Unfortunately, I can't find any details about how they obtain the mesh model for novel poses in the paper. 3. The authors claimed in Abstract that the combination of photorealism and inference speed "still remains unsolved", which I think is not true. Some recent works have already achieved real-time neural character rendering with high photo-realisitc quality. For example, Ouyang et al. [a] proposed to use multi-plane image (MPI), which is also a light field representation, to render an animatable character in real-time. Ouyang et al. also applied perceptual supervision on full images to learn high-quality appearance details. Therefore, I think this submission should cite, discuss, and perhaps compare with this highly relevant work [a]. [a] Ouyang et al. Real-Time Neural Character Rendering with Pose-Guided Multiplane Images. ECCV 2022. 4. In Line 199-200, the authors mention that the blurry results trained with L1 loss is due to the one-to-many mapping problem. However, their solution, i.e., applying perceptual supervision, does not resolve this problem; in other words, the one-to-many problem still exists after applying the perceptual loss. I guess this is the main reason for the jittering texture in the supplemental video (00:14-00:29). 5. The novel pose sequences for the long skirt case (supplemental video 00:56-00:59) are too short. Typos: Line 278: In Fig.5, **We** --> In Fig.5, **we**

Questions

None

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors discuss the limitations and potential societal impact in the supplemental document.

Summary

The paper presents a real-time approach for generating controllable and photorealistic digital human avatars. The main contribution of the paper is to learn the texture of the avatar using a light-field representation. In contrast to other continuous representations (e.g., NeRF), the proposed method can predict the color of a ray using a single MLP call. This allows faster rendering of the images which in turn allows using perceptual losses during training and yields significantly better synthesis results. The paper mainly extends the Deep Dynamic Characters (DDC) method and replaces its TextureNet with the proposed light-field representation. The experiments are performed on the DynaCap dataset where the proposed method is shown to outperform other real-time methods while being on par with the non-realtime methods.

Strengths

- The use of the light slab method for learning the texture details of the avatar is novel and makes sense. I believe this has the potential to be used in other methods as well where faster rendering speed is required. - Since the color for each ray can be predicted in one shot, the proposed method allows rendering full images during training. Hence, perceptual losses can be used and are shown to help significantly (LPIPS: 18.51 vs 12.85) - The proposed method achieves results on par with the state-of-the-art while being 100x better. - The paper is well-written and easy to read, though some typos need to be fixed

Weaknesses

### Novelty - My main concern with the paper is its limited novelty. The paper builds on DDC and mostly replaces its TextureNet with the proposed light-field-based MLPs. It is unclear whether better results are due to better engineering or the proposed light-field-based approach. Is the implementation of DDC and the proposed method exactly the same with texture nets being the only difference between the two? ### Missing Experiments - I would have liked to see an experiment in which light-field is replaced with vanilla NeRF while having everything else exactly the same.

Questions

### Checkerboard artifacts: - What are those checkerboard artifacts in Fig. 4? With such artifacts it is very easy to identify that the images are computer generated, hence it raises a question regarding paper's claim of `photorealistic` synthesis. Perhaps tone down the photorealistc claim if these artifacts are not preventable. ### Experiments - The best setting of the proposed method is with $L_{perc}$, I wonder why the ablation studies are performed without this loss? What happens if we use $L_{perc}$ with `Single surface + viewing direction` setting? ### Overview figure Why the red dot is at a different location in the feature maps? Shouldn't it be consistent with the temporal normal maps?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations of the paper are not highlighted in the paper. The authors should add the during the rebuttal. Potential limitations are: 1. Requirement of an initial 3D scan of the person as required by DDC, which is not easy to obtain without having access to sophisticated hardware. 2. Long training sequences. 3. Use in deep fakes. 4. Privacy concerns, etc.

The surface light field representation

Dear Authors of Manuscript 3760 and Reviewers, In the course of my review, particularly while examining the "360° Viewpoints" section (Sec. 3.2), I noted a potential correlation between the proposed parameterization and the 'surface light field' concept discussed in [1]. The current manuscript does not cite [1], which makes it challenging for me to fully evaluate this aspect. I would appreciate it if the authors could provide further clarity on the distinctions and connections between their work and [1]. [1] Wood, Daniel N., Daniel I. Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H. Salesin, and Werner Stuetzle. "Surface light fields for 3D photography." In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 487-496. 2023. Kind regards, AC