Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape

Summary

This paper introduces a novel generative model for generating 3D textured shapes, given a single example as input. Building upon recent work on diffusion models and tri-plane representations for shape representations, the proposed model can generate variations of the input shape. The method is evaluated across a variety of metrics of quality and diversity and compared to previous work on this topic. An ablation study is included, showing the impact of each of the components of the model. Finally, important applications are shown, including generating 3D shapes with BRDF properties, as well as different edition possibilities.

Strengths

- This paper introduces a novel generative model for 3D textured shape generation, capable of synthesizing unseen variations of a 3D texture shape given a single example as input. Generative models trained on a single example (image, texture, video, etc) are a long-standing problem in the literature, and this paper proposes, to the best of my knowledge, the first method for 3D textured shapes. - The proposed method is sound, the design decisions are solid and well-justified, and borrow from the literature of both neural rendering and single image diffusion models in interesting ways. - Some ideas in this paper may be valuable for many downstream applications, including generative models trained on datasets of 3D textured shapes, or for dreamfusion-like models, where the representation that is learned is a radiance field. - The quantitative evaluation proposed in this paper is comprehensive, measuring not only quality but also diversity. - The applications proposed in this paper demonstrate that the method can be used for controllable generation and edition. Further, it can also synthesize BRDF properties, which makes this model applicable for downstream computer graphics pipelines. - The paper is well structured and it is mostly well written. The quality of the figures is very high. - The triplane convolution is smart and I believe it amy have an impact on many future works. - The supplementary material provides valuable insights on the quality of the results and the impact of different individual components. - Extensive implementation details are provided, and code is included in the submission, increasing the paper's reproducibility.

Weaknesses

- The paper writing is sometimes overly convoluted and the same arguments are repeated too many times. In my opinion, this makes the paper at times a bit hard to read. (See the Questions section). - In some cases, showing more examples could have been benefitial to this paper. - While limitations are somewhat adequately discussed, it would have been interesting to see failure cases. - The applications (4.4 and 4.5) sections are limited in the amount of examples shown and it makes it hard to fully understand what level of control is actually provided to the user. - The related work section can be, in my opinion, improved. In particular, I believe that this paper should mention an important use-case for single-sample generative models, which is texture synthesis. This is even more important as this paper is working with Textured 3D shapes, and most of the examples shown contain repetitions (eg bricks or windows). I suggest including important work like "Non-stationary texture synthesis by adversarial expansion", Zho et al., Siggraph 2018; "“Self-organising textures", Niklasson et al. 2021; and "SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps", Rodriguez-Pardo et al. TVCJ 2022. - Some components of the model are detrimental to the diversity of the generated samples. This is a traditional trade-off in many generative models, where increased diversity can lead to less realistic outputs. However, I believe that this paper leans heavily towards realism, hindering diversity. I believe that some level of control on this tradeoff should be allowed to the user, and changing the receptive field of the model is, in my eyes, the only possible way to achieve this. This could be discussed in the limitations section or some future work could be hinted to address this.

Questions

- What are the results of this method with unstructured shapes? Most of the results shown contain repeating patterns (like columns, windows or bricks). What would happen with a human 3D shape, for instance? - Can this method generate tileable texture shapes? For example, it would be valuable to generate a tileable version of the shape in Figure 6 (a). - Regarding the resolution of the method, it is mentioned that the maximum resolution across one dimension is 256 for the input shape, and that the receptive field is 128, while it is also mentioned that the receptive field is 40% of the input resolution. Could the authors clarify this? - Does this model allow for texture transfer or for structural analogies? See "Non-Stationary Texture Synthesis by Adversarial Expansion", "Drop the GAN" or "Neural Photometry-guided Visual Attribute Transfer" for examples of this. It would be very powerful to be able to condition the generation of one or more of the tri-planes with a structure from another image. - Is there a middle ground between using and not using the $\epsilon$-prediction? I am concerned that this introduces a trade-off between diversity and quality that may be limiting the impact of the proposed method. - Can the authors show more examples of results of this method? I think it would be valuable to see more results on controlled generation, retargeting and PBR generation. - What is the impact of using only a 2-level U-Net for the denoising, rather than Writing improvement suggestions: - I suggest the author mention the need for relightable assets less frequently. I think the introduction motivates the problem well, however, it is mentioned more times during the paper, hindering its readability. - On the first paragraph of the introduction: "often time-consuming and tedious", as the current text is written, is referring to "artistic sensibilities", which I would disagree that are "time-consuming and tedious". I understand where the authors are going with this argument, however, the current phrasing is a bit confusing.

Rating

8: accept, good paper

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Ethics concerns

The authors have mentioned that "I certify that there is no URL (e.g., github page) that could be used to find authors' identity." on their submission. However, I came across this URL which is not anonymized: https://github.com/Sin3DM/Sin3DM . I am not sure this complies with ICLR code of conduct and I would ask the ACs to look into this. My review has not been influenced by this in any way.

Summary

This paper aims at training a diffusion model to generate tri-plane that could be decoded into 3D objects. It first compress the 3D meshes into tri-planes by training a auto-encoder. Next, it trains a diffusion model to generate the encoded triplane. By altering the input noise, the model could achieve retargeting, outpainting and local editing.

Strengths

1. The edited geometry after altering the noise is satisfactory. It has rich local variation while retaining the global structure. 2. The writing is clear.

Weaknesses

1. After training, the model could only generate 3D contents with the same global structure. Since we already have the basic 3D mesh at the beginning, it seems that the application is a bit narrow and limited. Besides, achieving this goal needs to train two models, an auto-encoder and a diffusion model. The outcome does not seem to match the effort. 2. The novelty of this work lies at the usage of tri-plane as the input of diffusion model and the introduction of tri-plane convolution. Both aspects do not offer a solid contribution. Using tri-plane as a compact 3D representation has been explore by many prior works. The contributino tri-plane convolution also lacks complete analysis, such as discussion and comparison with similar modules that could fuse three axis information.

Questions

1. What would happens if multiple objects instead of single object are encoded and fed into diffusion model in training? Would it not be able to generate realistic samples? 2. What does retargeting mean in this context? I could only get the meaning of outpaining and local editing. 3. How do the input noise and target tri-plane feature match each other when training the diffusion model on only one input shape? One input shape could only correspond to one tr-plane feature maps, while different noise is sampled in each iteration.

Rating

5: marginally below the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Summary

This paper introduces a diffusion model trained on a single 3D textured object. The goal here is to learn a generative model that can regenerate local patch wise patterns of the single object. The key idea is to train a path-based diffusion model on encoded latents of the single objects and use a triplane-base decoder to represent the 3D content. For the decoding part, the authors propose a novel triplane-aware convolutional architecture and 3D convolutional based encoder. The authors provide results on various 3D assets and provide an ablation study to support their claims.

Strengths

The related work section covers relevant topics and helps to set the work in context. The authors tackle a novel problem that I was not aware of before. The results look plausible and claims are supported.

Weaknesses

One point that remains unclear to me is how the triplane based representations can handle larger extensions of objects, e.g. in the example of the building. How can the triplane represent those? Do you increase the size? The authors claim that the model learns a distribution over patches. It is unclear to me how you choose the size of patches/ receptive field and how you identify that it is nor overfitting. I think there is more evaluation needed to show that it is not overfitting. Maybe an ablation on the receptive field size can lead to interesting insights.

Questions

In my view, the diversity metrics measure the average distance between generated samples and it seems like the generated samples should be very different from each other to have an improved diversity. I’m unsure if this is a good metric. The goal you want to achieve is a good similarity of local details while having a diversity in the global structure. Maybe a patch-based distance vs. a global feature can help with that. Please comment on that.

Rating

6: marginally above the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Summary

This method introduces a way to generate new textured 3D shapes from a single example. To accomplish this, the input shape is represented using triplane feature maps, which encode signed distance as well as color. These feature maps are used to train an auto encoder, and a diffusion model is then trained on this auto encoder latent space. Then, by modifying the noise map, e.g., changing the resolution or masking regions, during the diffusion inference process, novel 3D shapes whose internal distribution resembles that of the original are generated.

Strengths

The overall idea of using latent diffusion models to aid in learning the internal distribution of a 3D shape and the specific design choices in the method---e.g., the shape representation and the steps taken to prevent overfitting---are clever. The results shown also look nice and consistent with minimal artifacts.

Weaknesses

It is mentioned several times that a unique benefit of the proposed method is that inputs and outputs meshes. This is not the case, as instead the 3D representation is triplanes encoding signed distance as well as texture color. While this can indeed be converted to a mesh using marching cubes, it's not fair to claim that the method outputs a mesh. This brings me to my main concern with the paper. While the authors cite and briefly discuss [Li et al. 2023], no qualitative or quantiative comparisons are provided, with the explanation that Li et al. do not produce parameterized meshes. While I agree that there are merits to the proposed representation, [Li et al. 2023] seems like the most natural point of comparison---many of the experiments are nearly identical to this in this work. For this reason, I cannot recommend acceptance as is. Typos: Abstract: "example as long" -> "example has long" 4: "The decoder first refine..." -> "The decoder first refines... gathers..." Figure 3: replicating along which axis?

Questions

How consistent/realistic are the normals generated in Figure 7? It would be interesting to show them used in relighting.

Rating

5: marginally below the acceptance threshold

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

General response

We appreciate the reviewers finding our proposed method novel, presentation clear, and experimental results strong. We have addressed all concerns raised by the reviewers in our rebuttal and we hope the authors can engage in follow-up discussions if any other concerns still remain. We have submitted a revised version of the paper with revised texts are highlighted in blue color. Specifically, in the revised version, the major changes include, - we add quantitative and qualitative comparison to [Li et al 2023] (see Table 1 and Figure 14) (__reviewer Udpj__); - we fixed a bug for computing texture quality score and updated Table 1 and Table 2 accordingly; - we include more examples for retargeting and controlled generation Figure 15, and show failure cases in Figure 16 (__reviewer cGW2__); - we clarify all the points that led to confusion (__reviewers Udpj, s6cL, cGW2__) We address each reviewer's specific concerns in separate responses below.