AutoDecoding Latent 3D Diffusion Models

Summary

In this manuscript, a novel method for unconditional and text-conditional generative models of 3D shape and texture representations is proposed. More specifically, the authors propose to train a 3D diffusion model of radiance and RGB fields that can be trained from 2D image and object mask supervision. The method comprises two stages, where in the first stage, an auto-decoder is trained, while in the second, a 3D UNet modeling a denoising diffusion process is trained on the latent features. At test time, random latent representations can be sampled and denoised to generate 3D radiance volumes that can be rendered from arbitrary viewpoints.

Strengths

- The authors propose a valid method for training 3D diffusion models from posed 2D images. The key idea to first train an auto-decoder, and then in a second stage a 3D UNet describing a 3D denoising diffusion process is interesting and, with the success of recent latent 2D diffusion models in mind, an important field of study. - The authors perform an extensive evaluation on 5 different datasets. To be able to run experiments on these different datasets, multiple automated masking and/or filtering steps needed to be performed (compare e.g. L. 252). - The manuscript is well written, the technical explanations are sound, and the use of mathematical terms and symbols is correct. - The manuscript contains helpful and well-organised figures such as the overview figure 1

Weaknesses

- Experimental Evaluation in Table 1: Why does pi-GAN achieve such a high FID value of 52.71? Could the authors share some intuition on why the score is so bad? This seems not to be coherent with comparable prior works, for example GRAF [47] or GIRAFFE [43] report an FID of 34/20 on this dataset. Note that not improving over GAN-based approaches in FID on these single-object datasets is OK I believe, but it is important that the numbers are accurately reported. - Qualitative Comparison: I believe the manuscript would benefit from a qualitative baseline comparison, e.g. for the methods reported in Table 1. It would further be interesting to discuss to a greater extent the differences of the results from the different generative model types, i.e. compare GAN-based results to the proposed diffusion model. - The results are still quite limited, both qualitatively and quantitatively. I do appreciate that the authors tackle this hard task "feed-forward" of unconditional/text-conditional 3D generation, but e.g. the texture from the samples in Fig. 5 are very limited. I believe at least a text-based comparison and discussion to test-time optimization-based methods such as Dreamfusion is relevant here, as they lead to significantly better results. Potentially the proposed method could also be extended with an additional test-time-optimization step in future work.

Questions

- Can the authors explain why the FID metric is so bad for the pi-GAN method? - Can the authors explain what are the main reasons for the limited quality for the texture predictions in Figure 5? - How important is it that the datasets contain multiple views (as opposed to only one) of the same object?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

- The authors have discussed limitations as well as potential negative societal impact. As stated above, I believe the manuscript could benefit from a comparison/discussion against test-time-optimization-based methods due to the limited quality of the shown results.

Summary

This paper presents a 3D generation framework that generalize to large-scale 3D dataset and articulated objects. The method comprises two parts. The first part is a 3D auto-decoder, reconstructing 3D objects from multi-view images or monocular videos. The second part is a latent diffusion model for unconditional or text-conditioned 3D generation. Extensive experiments are performed on 5 datasets, including the largest 3D dataset objaverse.

Strengths

- I am impressed by the workload of this work. Not only are diverse static 3D objects are supported, but also articulated objects like 3D human heads. - Experiments are solid. A total of 5 datasets are used, including Objaverse, which is the largest and the most challenging 3D dataset. Qualitative results on tables and chairs are reported and compared with 3D-aware GAN and 3D diffusion methods. - Using the normalized IQR and median to normalize the latent features is an interesting trick. It is crucial for latent diffusion methods to deal with the latent normalization.

Weaknesses

- The method itself is not novel, which I think is a minor issue since the main contribution of this work I think is the scaling up and generalizing the articulated objects. - The quality of generated objects is inferior. I think this is limited by the reconstruction part. The resolution is limited to 64^3 due to the computational complexity of 3D volumes. I am wondering whether a CNN refinement/ super resolution module would help (e.g. EG3D, DiffRF)? - The text-conditioned generation can only correctly generate the overall color and object category. Most detail descriptions are ignored. Other than the caption quality, is there any other possible reason for that, like the conditioning method design?

Questions

N/A

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This paper proposes a diffusion model that learns to generate 3D objects, using only multi-view images or videos for training. It first trains a 3D convolutional autodecoder to embed the dataset; this maps latent vectors via a 3D feature space to voxelised scenes, and is trained for reconstruction of volumetrically-rendered images. Then, it trains a diffusion model over the feature space of this autodecoder, to enable a-priori generation; there are experiments on which layer of 3D features are best to use. The method is demonstrated on a variety of datasets, both synthetic and real, including rigid objects such as chairs, and non-rigid objects such as human faces.

Strengths

The proposed pipeline is novel, its components are clearly described, and most design decisions are well motivated/justified (and appear sound). There are insightful discussions on how the autodecoder architecture affects performance. The evaluation is rather comprehensive, covering five datasets of somewhat different character. Some of these are synthetic and use ground-truth camera poses, while others use imperfect poses from SfM. Generation results on synthetic images (chairs & tables) are quantitatively better than baselines – both FID and KID are lower than pi-GAN, EG3D, and DiffRF. Qualitative results here also look good. Generation results on real images are found to be quantitatively of higher quality (lower FID/KID) than those generated by naively sampling in the latent space of the autodecoder (without any diffusion process). There is an ablation study covering various important design decisions in the autodecoder (mainly architectural choices), and an additional set of experiments investigating which feature layer of the autodecoder the diffusion is performed over, and how many diffusion steps are used.

Weaknesses

While it may be accurate that "no prior work demonstrates the ability to generalize to such large-scale datasets" (L282), I feel there should be some attempt at a quantitative comparison here – e.g. retraining the best of the prior works on these datasets to see how well or badly they perform. In particular, Objverse, CelebV-Text and MVImgNet are all extremely recent, so it may simply be that the prior works have not been tested on those datasets (as they were not yet available), but would still work to some degree. This is particularly important to put the (rather high!) FID scores of tab. 3 in context. An alternative that would go some way to mitigating this problem would be to use one of the more-constrained by still photographic datasets used in pi-GAN or EG3D (e.g. human/cat faces or cars), and evaluate how well the proposed method performs on this. Qualitative results on MVImgNet (only given in the supplementary) are not very impressive – it is often impossible for this reviewer to determine the class of the generated objects. Similarly, the results on face generation (with a target) look significantly lower fidelity than other recent methods like EG3D. The qualitative text-conditioned generation results in fig. 4 & 5 are not particularly impressive. Moreover, the 'ground-truth' text labels are from a pretrained captioning model, and thus noisy. It would be valuable to include an experiment with high-quality captions, so the impact of caption quality vs model power can be understood. The technical contribution is a little small for NeurIPS – both latent diffusion and autodecoding of 3D shapes are pre-existing techniques, and their composition does not seem to be particularly complicated. This is somewhat mitigated by the detailed experiments and extensive discussion of design choices.

Questions

Why is an autodecoder approach chosen, instead of (for example) VQ-GAN or autoencoding? See above under weaknesses regarding performance comparison with baselines on real-world data – this is my largest concern. How are the 'driven' face animations in fig. 2 generated? I didn't see this described anywhere in the text.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

There is adequate discussion of both limitations and broader impact.

Summary

The paper proposes a 3D autoencoder to learn a latent volumetric space on the training dataset, which can be decoded into a radiance volumetric representation for novel-view synthesis, and then learn a 3D diffusion model on the latent volumetric representation. The latent volumetric space is acquired by training NeRF on multiple views (or frames).

Strengths

- Diverse datasets, including synthetic and real-world, rigid and articulated objects, are used for evaluation.

Weaknesses

1. Some claims in the paper are not verified by the evidence. See *Questions* for examples. 2. Only novel-view synthesis is demonstrated. Since the authors use a volumetric representation, the method is restricted by the voxel resolution. The visualized images look small and coarse. It will be better if the authors can show some 3D results (e.g., extracting meshes from radiance fields). Thus, compared to prior and concurrent works (see *Questions*), this method does not seem to be more scalable or extendable. Especially, the voxel resolution is a bottleneck. 3. Camera poses still seem to be necessary in this work, either automatically being annotated in synthetic data or estimated for in-the-wild data. It is a little tricky to say that this work does not need "3D supervision". The authors can do such an experiment to convince the readers that the proposed method is robust to camera poses: comparing a model trained with estimated poses on synthetic data and one trained with GT poses. Minor typo: - L212: One of our key observation -> One of our key observations?

Questions

1. In L41-L43, the authors claim that "our approach is thus designed to be robust to" different sources of poses. However, I do not find relevant explanations in the method or experiment section. Can the authors elaborate on it? 2. In L150-L151, the authors claim that "intermediate representations such as feature volumes or tri-planes, as it is more efficient to render and ensures consistency across multiple views". However, several prior or concurrent works (e.g., 3D Neural Field Generation using Triplane Diffusion, 3DGen: Triplane Latent Diffusion for Textured Mesh Generation, Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction) show that feature volumes or tri-planes can work well. Can the authors provide any ablation study to support the claim? 3. In L62, "To identify the best scale" is unclear, as the word "scale" is not mentioned before. 4. It is unclear which datasets are used for training and evaluation. In L71-76, it is unclear whether the authors train 3 models on 3 datasets separately or they train a model on 3 datasets progressively. And in Sec 4.1, it is unclear whether all datasets are just for evaluation, or some of them are used for training. The authors should clarify their training and evaluation protocols.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have adequately addressed the limitations.

Summary

In this paper, the authors propose an approach to learning a latent diffusion model for 3D assets from image (or video) supervision. The core is that they use an autodecoder architecture, which learns embeddings in the latent space by decoding them into a volumetric representation for rendering. Then, the authors identify the appropriate intermediate volumetric latent space and introduce a normalization technique to learn a latent diffusion model. In addition, this method can use either existing camera information or no camera information during training. The experiments demonstrate the proposed method is able to generate both static and articulated 3D objects.

Strengths

- The proposed method has a broader application domain and is extendable to large-scale datasets. It can be applied to various datasets, including rigid static and articulated moving objects. - The proposed modifications are effective in improving the 3D autodecoder. - The authors propose a robust normalization trick to train latent diffusion models on every intermediate volumetric latent space.

Weaknesses

- The authors miss a proper discussion with an important related work, GAUDI [a]. GAUDI first learns a 3D autodecoder from images and then trains a diffusion model on the latent space as a prior for generation. There are several similar points between this work and GAUDI. However, GAUDI is not even cited. - The learned 3D voxel grids are treated as "Canonical Representation." However, how to properly define a canonical view of large-scale 3D datasets? For example, in Objaverse, the object poses are not well-aligned. Even for a single category (e.g., chair), the objects are randomly placed. Will this case influence the performance of the proposed method? Do you have any observations? - Considering the autoencoder-based method needs to regularize the bottleneck, the autodecoder can also regularize their bottleneck as done in [c, d]. Have you tried to train a 1D diffusion model (see [e]) in the latent space of the autodecoder to see the generation ability? - Missing comparison with some important baselines, such as GET3D [f]. I also suggest the authors train GET3D and DiffRF for Table 3 for a fair comparison. - I have some questions or suggestions according to the writing: - In Line 1, the authors assume diffusion models are all latent diffusion models. I suggest the authors polish their writing to avoid this misuse. - In Line 57, the authors mention, "First, our autodecoders do not have a clear "bottleneck." Is the bottleneck here a latent space? If it is the case, the embedding space is the latent space of the autodecoder. - In Line 147, it is suggested to add some credits to prior work on trilinear interpolation for volumetric rendering, such as Plenoxels [b]. - Can you elaborate more on the direct latent sample? - It is suggested to move the "Hash Embedding" part to the main paper. Otherwise, the vanilla autodecoder approach can not be scaled to large-scale datasets due to the large size of the embedding. - It is suggested to add [a-e] to the reference. - In LDM [46], they use a normalization technique on the latent space. The proposed robust normalization seems to be an extended version of this. [a] GAUDI: A Neural Architect for Immersive 3D Scene Generation. NeurIPS 2022. [b] Plenoxels: Radiance Fields without Neural Networks.CVPR 2022. [c] Demystifying Inter-Class Disentanglement. ICLR 2020. [d] Rethinking Content and Style: Exploring Bias for Unsupervised Disentanglement. ICCV 2021. [e] Latent Point Diffusion Models for 3D Shape Generation. NeurIPS 2022. [f] GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images. NeurIPS 2022.

Questions

Please refer to the weakness section. I will adjust my rating according to the rebuttal.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The limitation and broader impact part look adequate to me.

Post-rebuttal

Thanks for the detailed rebuttal. This resolves some of my concerns, and overall I remain positive about this paper.

Post-Discussion

We would like to thank all the reviewers for taking the time to reply to our responses. We are glad that: they found our rebuttal detailed (R.Nn3Q), extensive and informative (R.nmDk); reassured or improved their positive outlook for the paper (R.xDwK, iNSR, Nn3Q); and had their concerns addressed (R.2ZWE, R.xDwW). We will update our manuscript, adding all the additional content promised in our responses.