X-Ray: A Sequential 3D Representation For Generation

Summary

The paper presents X-Ray, a new 3D sequential representation inspired by the penetrating quality of X-ray scans. This technique converts a 3D object into surface frames at different layers, ideal for creating 3D models from images. Experimental findings show that the X-Ray approach outperforms existing methods in improving the precision of 3D generation, opening up new possibilities for research and real-world applications in 3D representation.

Strengths

* The motivation is interesting. Exising methods indeed cannot completely generate objects that include both visible and hidden surfaces. While the proposed method can the hidden interior of the object can be fully reconstructed. * The compatibility of X-Ray data structures with sequential 3D representations in video formats opens up new opportunities for using video diffusion models in 3D generation. * Great experimental results. The paper outperforms competitors by achieving the best results.

Weaknesses

* The author asserts that "General, accurate, and efficient 3D representations are three crucial requirements for 3D generation." However, the citation is missing or the author should provide evidence to support the claim that 3D representations need to be general, accurate, and efficient. * The ablation study is insufficient. The author should conduct ablation on X-Ray frames, for example, by using video or image diffusion techniquesm or the impact of frame complexity. * The author claims to have synthesized a complete interior mesh in Figure 1, but I did not observe any other results besides cars or analysis in the experiments. * I think the method used is the X-ray edge method because the dataset quality does not allow for a complete X-ray scan. For instance, the 3D CAT data has empty spaces within.

Questions

* The authors need to provide a clear explanation of X-ray and complete internal mesh, as some training data contain empty spaces within. * The authors should conduct additional ablation studies.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors acknowledged limitations but did not address the potential negative societal impact of the proposed technology.

Summary

The paper addresses the problem of missing mesh interiors in image-to-3D generation. The proposed representation, X-Ray, adopts ray casting to encode both visible and hidden surfaces into a video format. With this representation, the authors enable single image-to-3D mesh generation, including the inside of the mesh, through a video diffusion model. Experiments demonstrate the effectiveness of the proposed method in 3D generation from a single image, both quantitatively and qualitatively.

Strengths

(1) The task of single-view image to 3D mesh generation including interior surface of mesh is extremely challenging and well-motivated. As far as I know, this is the first paper that generates a mesh considering the interior surface. (2) The idea of using ray casting to scan the interior of the mesh and convert it into video frames is novel. I really like this approach. (3) The results show that the proposed method achieves SOTA performance on single image-to-3D generation and point cloud generation with a short inference time compared to existing baselines. (4) The writing is easy to understand and the design choices are well presented.

Weaknesses

(1) Considering Computed Tomography, it is expected that the performance will significantly increase as the number of layer $L$ increases. However, the experiments analyzed in Section 5.2 show that performance improvements are very minimal when $L$ is larger than 8. If additional computing resources are available, it would be interesting to see the performance when the frame resolution is low, but $L$ is extremely high (not mandatory). (2) The entire training pipeline is quite long and memory-intensive. It was conducted on 8 NVIDIA A100 GPU servers for a week.

Questions

(1) When training, it seems necessary to normalize the object so that the rays from a specific camera position can capture the entire object, making it easier to train. I'm curious whether this preprocessing step was applied to the Objaverse mesh. If this process was included, it would be beneficial to add this information to the paper. (2) How much GPU memory is required during inference?

Rating

9

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors mention the limitations related to the number of sequence layers and the missing parts when X-Ray is truncated, as discussed in Section 6 and Appendix A.5.

Summary

The paper proposes a new representation that encodes a 3D mesh into the ray intersection points from a single view point. The position, color, and surface normal at the intersection points are stored as the representation. Poisson reconstruction is used to recover the 3D mesh from the representation. A cascaded diffusion model is trained using the extracted representation from 60k objaverse meshes. Evaluation is conducted on three other datasets.

Strengths

The paper proposes a new representation, which is compact, efficient, and tailored to single image to 3D applications. It demonstrated better performance compared to other methods in the paper in the metrics. The visual quality of the results are also better. 3D generation is a very relevant topics and the paper will be interesting to many readers.

Weaknesses

1. The paper does not discuss how to extend the proposed representation to include multiple viewpoints to provide better encoding/decoding quality. Even though the representation records the intersection points along a ray, the intersection points from front and top views would be very different. 2. One limitation of X-ray is that the limited number of intersection points (8 in the paper) makes it difficult to encode complex shapes like hairs and carefully designed meshes with all internal structures. For example, taking a real engine, the number of surfaces a ray can pass through would be far more than 8 (eg., the wires, the bores, shafts). The paper mentions the limitation, however, it does not seem clear to me from the paper how the limitation can be addressed efficiently. 3. Another related limitation is the potentially nonuniform number of intersection points along different rays from the same viewpoint. For example, on a human head, the rays passing through the hair region would need a much higher number of intersections to avoid losing information than those through the face region. The nonuniformity extends to across objects. To model one object that has a large number of intersection points, it might need to increase L used for all objects in order to learn a diffusion model. This property makes the presentation inefficient. 4. In order to generate all interior structures of a 3D object, we need meshes that are carefully designed with interior details. This kind of dataset is difficult to get and if existed would increase L, making the method inefficient.

Questions

1. The proposed representation is related to the pointmap used in DUSt3R (CVPR2024). From my understanding, at a high level, it extends pointmaps to multiple points along a ray. To me, it is an interesting connection and can be mentioned in the paper. 2. The core of the representation is finding the intersection points and creating the tensor representation. Currently the paper only mentions that it calls a function in trimesh. I would suggest describing the operation in more detail -- for example, I do not know how the Hit is constructed. Does the method try to aggregate intersection points at similar depths at the same index along the L-dimension? Or is Hit simply a padding mechanism? 3. In Sec 5.3 quantitative comparison, Table 1 and Table 2 show metrics that evaluate the quality of the geometry. How is the photometric quality of the proposed method compared to other methods, e.g., PSNR or SSIM on the conditioning viewpoint (and other novel views)? 4. How significant is the effect of the predicted normal, and is it fair to compare rendering-based methods like LRM using the extracted meshes in Table 1? My understanding is the quality of the Poisson reconstruction can be affected heavily by the quality of the normal, which may be poorly modeled by volumetric density. Should the paper also show the same metrics directly on the point clouds or depth maps?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The paper mentions the limited number of intersection points. However, it does not seem clear how to address the limitation potentially without significantly increasing the number of L.

Summary

This paper introduces X-Ray, a new 3D representation designed for efficient generation of 3D objects from single images. The key idea is to represent a 3D object as a sequence of 2D "surface frames" capturing hit/miss, depth, normal, and color information along rays cast from the camera viewpoint. This sequential representation lends itself well to generation using video diffusion models, enabling the synthesis of both visible and hidden surfaces. The authors propose a pipeline consisting of an X-Ray diffusion model to generate low-resolution surface frames from an input image, followed by an X-Ray upsampler to enhance resolution. They evaluate their method on single-view 3D reconstruction and unconditional 3D shape generation tasks, reporting quantitative results on standard benchmarks.

Strengths

- The paper tackles the important challenge of reconstructing complete 3D models, including hidden surfaces, from single images. This is a significant limitation of current rendering-based approaches, and the authors' focus on this problem is well-motivated and timely. - The X-Ray representation, while its novelty requires further substantiation, offers an intuitive way to encode 3D surface information in a sequential manner. By focusing solely on surface details rather than volumetric data, the representation has the potential to be more memory-efficient than voxel grids or dense point clouds, especially for objects with complex internal structures. - The authors smartly leverage recent advancements in video diffusion models for 3D generation. This is a promising direction, as video diffusion models have shown impressive capabilities in synthesizing high-quality and temporally coherent sequences of images. Adapting these models to the task of 3D generation through the X-Ray representation is a reasonable and potentially fruitful approach.

Weaknesses

- The paper does not provide a convincing argument for the novelty of the X-Ray representation. While its sequential capture of surface information is intuitive, a thorough comparison to existing techniques is lacking. Specifically, the authors should clearly differentiate X-Ray from methods like depth peeling, multi-view depth images, multi-plane images (MPI), and notably, the PI3D representation (Liu et al., CVPR 2024), which also leverages diffusion models for 3D generation. * Liu, Ying-Tian, Yuan-Chen Guo, Guan Luo, Heyi Sun, Wei Yin, and Song-Hai Zhang. "Pi3d: Efficient text-to-3d generation with pseudo-image diffusion." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19915-19924. 2024. - The paper repeatedly emphasizes the efficiency of the X-Ray representation without providing concrete evidence or analysis. The authors should quantify their claims by comparing the memory footprint and computational costs of X-Ray to alternative representations like voxel grids, point clouds, and neural implicit representations (e.g., NeRFs) for objects of varying complexity. - While leveraging video diffusion models is promising, the paper does not clearly articulate how the specific properties of the X-Ray representation are exploited within the diffusion process beyond being a sequential data format. Do the hit/miss indicators or the ordered nature of surface frames influence the model architecture or training? Would similar performance be achieved with alternative sequential representations as input to the diffusion model? - The evaluation heavily relies on reconstruction metrics (CD, EMD), even when assessing a generative model. While these metrics are relevant for the single-view reconstruction task, they do not capture the generative capabilities of X-Ray. The authors should expand generative evaluation to diverse categories beyond ShapeNet Cars. The paper could assess generative quality by evaluating the diversity and realism of multiple shapes generated from the same input image. - The evaluation would be also strengthened by: (1) Including recent SDS-based single-view 3D generation methods as baselines; (2) Providing visualizations of generated shapes for the unconditional generation experiment; (3) Dedicating a section to analyze failure cases, visually showcasing problematic inputs and outputs.

Questions

- Could the authors please elaborate on the key differences between the X-Ray representation and existing techniques like depth peeling, multi-view depth images, and multi-plane images (MPI)? Also please include the discussion with PI3D. - To support the claims of efficiency, could the authors provide a quantitative analysis of the memory footprint and computational cost (encoding, decoding, generation) of X-Ray compared to voxel grids, point clouds, or NeRFs? This analysis should consider objects of varying complexity and resolutions. - Could the authors please include generative metrics for the single-view 3D reconstruction experiments on GSO and OmniObject3D?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

- The authors merely state limitations without explaining their causes, impact, or potential solutions. E.g., saying "X-Ray frames become sparse" is not enough. How does this sparsity affect generation? How can it be addressed?

Response to rebuttal

Thanks for the authors' response. I have read it as well as other reviews. I believe this paper presents a new problem (single image to mesh generation including interior surface) that the academic community has not yet solved, and I consider it the first paper to suggest a new direction for research. Additionally, the proposed method (using ray casting + video diffusion model) is novel and has been proven through experiments to be superior to existing methods, which I believe can serve as a good reference for future research. Regarding the concern about the number of $L$, which I and other reviewers were worried about, the authors claimed to address it through an autoregressive Large-Language Model (LLM), but this part is not explained in detail, so it's hard to understand. However, given the current limitations in computing resources, the performance is sufficiently guaranteed even with $L=8$, to the extent that it surpasses other baselines, so I don’t think this is a major issue. In the revised version, it would be good to make revisions to address the concerns of the other reviewers, and particularly, I strongly recommend adding a section on the efficiency of the X-Ray representation (Rebuttal to Reviewer xHzv, Response 2). Since a new representation has been proposed, if a comparison with other representations is organized in a table, readers will be able to easily grasp the efficiency of the X-Ray representation at a glance. I appreciate the additional experiments conducted for the rebuttal, and for the reasons mentioned above, I will maintain my previous rating of 9.