Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation

Summary

This paper proposes to use the tactile sensing signal (height map and normal map) to improve the 3D generation quality, especially the geometric details. The authors use a 3D mesh generation guided by a normal-conditioned ControlNet to ensure the consistency between the visual textures and the tactile textures. They also develop a multi-part editing pipeline to generate objects with different texture parts. Experiments and ablation studies demonstrate the effectiveness of the proposed method.

Strengths

The origniality of the paper is good. To my knowledge, this is the first work to use the tactile sensing for 3D generation. The generation quality is satisfying. The generated meshes include good geometric details, which align well with the input tactile signals. The paper is well written and structured. The visualizations clearly demonstrate the quality of the generated meshes.

Weaknesses

The connection between the tactile sensing and the 3D generation is not strong and critical. The tactile signals are only used to generate the normal maps of certain textures. It may be replaced by simpler alternatives such as texture reterival from a normal map database. It needs further elaboration why this combination is necessary.

Questions

1. What is the generation time of the multi-part geneation? 2. Can the authors add an experiment of real image to 3D mesh with real corresponding tactile signals?

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Limitaions have been addressed in the paper, which include complex geometry generation and slight seams.

Summary

This paper proposes a lightweight 3D texture field that ensures the consistency between visual and tactile textures while preserving photorealism. The experiments demonstrate that quantitative and qualitative results show good generation quality.

Strengths

1. The authors pioneered the use of tactile sensing to enhance geometric details. 2. The exposition is good. The paper is easy to understand. 3. They created a TouchTexture dataset, comprising 16 everyday objects, contributing a new dataset to the community.

Weaknesses

1. This paper suggests that existing methods struggle to achieve fine-grained geometric details. However, methods like Neuralangelo and PermutoSDF can recover highly detailed geometric information. 2. The TouchTexture dataset presented in Figure 3 and the supporting materials seemingly do not capture the "local geometric intricacies." 3. There are only very limited objects been shown in the main paper and supplementary material. Are those objects cherry-picked? It would be great if more results on in-the-wild object can be provided to show the generalisation ability of the model. 4. Ablation Study is an important part in paper and it would be more convincing with both quantitative and qualitative experiments, and cannot be simply summarized with only several sentences. If there are any figures and tables in the article, please indicate the specific table number or figure number in the analysis and analyze according to the specific visualization results. - [Neuralangelo: High-Fidelity Neural Surface Reconstruction. CVPR2023] - [PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces using Permutohedral Lattices. CVPR2023]

Questions

See my previous sections.

Rating

4

Confidence

4

Soundness

2

Presentation

2

Contribution

1

Limitations

I'm not confident about voting for accepting this paper because of the potential similarity to existing methods and the limited novelty. There is more engineering effort than novelty. The novelty might be limited.

Summary

This submission addresses the long-standing challenge of enhancing geometric details in results produced by text-to-3D and image-to-3D pipelines. The approach introduces a novel method that leverages tactile normal modality to synthesize high-fidelity geometric details. Additionally, it employs attention maps during the diffusion process to segment input images based on text prompts, allowing for the synthesis of multiple textures across various regions. The results demonstrate that this method effectively recovers geometric details and ensures alignment between geometry and color.

Strengths

1. The approach is innovative in utilizing tactile normal modality to enhance geometric details. 2. The introduction of a newly collected tactile dataset, TouchTexture, is beneficial to the research community.

Weaknesses

1. The paper claims compatibility of the proposed method with both text-to-3D and image-to-3D pipelines. However, DreamCraft3D, selected as a text-to-3D pipeline, requires both an image and a text caption as inputs. Although DreamCraft3D can be considered as a 'text-to-image, text & image-to-3D' process, it differs from a purely text-to-3D approach. Thus, the compatibility between the proposed method and purely text-to-3D pipelines remains questionable. 2. The pipeline overview in Figure 4 indicates the need for a reference image and tactile input. The paper does not address how to select an appropriate tactile input for the reference image, nor how to ensure the tactile details are compatible with the object in the image. 3. The paper presents an intriguing text-guided segmentation strategy that leverages attention maps during the diffusion process based on text prompts. However, lacking expertise in diffusion-based segmentation, I am unfamiliar with the efficiency and success rate of this method, but I am positively impressed by it. 4. The generalization and diversity of the proposed method are also of concern. The objects presented in the results are not very complex, and the dataset comprises 16 popular categories. It remains unclear how well the method would perform on more complex objects, such as those in sci-fi or fantasy genres.

Questions

1. I am curious about your criteria for selecting 3D generation baselines. For image-to-3D tasks, to my knowledge, more advanced baselines such as 'InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models' and 'TripoSR: Fast 3D Object Reconstruction from a Single Image' offer superior quality and may yield stronger results. For text-to-3D tasks, 'RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D' is a noteworthy pure text-to-mesh method. 2. The proposed method in the paper first refines normals and then optimizes colors based on the refined normals. Does this imply that the geometric details introduced by your method are not derived from the original colors (i.e. not very necessary to original colors) of the objects? Additionally, could the color optimization process potentially disrupt the alignment between the original colors and the refined colors, potentially failing to meet the initial requirements? 3. I am curious about the generalization and diversity capabilities of the proposed method. Can it handle more complex objects? Furthermore, with more powerful baselines, can this method achieve higher quality results on more complex cases?

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors acknowledge their limitations in the main paper and address potential social impacts in the supplementary materials. Regarding the first limitation, the implementation of new 3D generative models, such as 'Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image' and 'Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention', is recommended. For the second limitation, utilizing a more powerful computer graphics tool could be beneficial. Concerning the social impact, issues related to deepfakes and the potential for misinformation are noteworthy. The authors assert that humans can currently distinguish their synthesized objects from real ones, a claim with which I concur. Although it is of low priority, it would be preferable for the authors to include a comparison between a generated object and a real one; a single case would suffice.

Summary

This paper proposes a method for generating 3D assets with detailed geometry through inputs from a tactile sensor. More specifically, given a bump-map as input from a tactile sensor (just a small patch is enough), the method uses it as regularization while maximizing the likelihood using a normals-conditioned stable diffusion model. The albedo is also optimized alongside the normals, though limited results are shown. Additionally, using diffusion based segmentation maps, different parts of the image can be given different textures. While each component of the method itself is not too novel, the sum total is and the results are pretty good.

Strengths

1) This is a novel and unexplored task and the proposed method provides the community with a good baseline to build upon. 2) The results of the method are quite convincing. The ability to edit only certain parts of the image with desired textures is particularly nice. 3) The paper is well written with each component explained pretty well.

Weaknesses

1) There are no results on the albedo provided, it would be great if the authors could explain why they were omitted. I strongly urge them to include it in the rebuttal. 2) I may be mistaken, but it seems this method works only for repetitive textures (though the diffusion model is able to change it through optimization). It would be great if the authors could provide the results of an experiment where the object has two very different textures but those textures are only learnt through the diffusion prior. For example, in the cactus pot case, this would correspond to optimizing L_{tactile} only for the pot and let the diffusion prior decide what the normals for the cactus must look like (or vice versa).

Questions

Just reiterating what I already mentioned in the weaknesses: It would be great if the authors could provide the results of an experiment where the object has two very different textures but those textures are only learnt through the diffusion prior. For example, in the cactus pot case, this would correspond to optimizing L_{tactile} only for the pot and let the diffusion prior decide what the normals for the cactus must look like (or vice versa). I believe this experiment would give us insights on how the diffusion prior itself would perform if a part of the image already has detailed normals (via L_{tactile})

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes