One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization

Summary

This paper introduces a novel approach for converting a single image into a 3D textured mesh without the necessity for per-shape optimization. The method employs Zero-123 for generating novel views of the input image, which is then utilized by a generalizable MVSNeRF to reconstruct the 3D shape. To tackle issues like inconsistent multi-view predictions from Zero-123 and inaccurate camera poses, the authors have developed an elevation estimation module and specialized training strategies, including 2-Stage Source View Selection and Groundtruth-Prediction Mixed Training.

Strengths

1. The paper proposes a method that reconstructs 3D shapes from a single image, eliminating the need for per-shape optimization. It does this through the effective application of feed-forward SparseNeuS. 2. The authors discuss and address several challenges associated with lifting Zero-123 predictions to 3D, such as inconsistent multi-views and inaccurate camera poses. The introduction of 2-Stage Source View Selection, Groundtruth-Prediction Mixed Training, and an Elevation Estimation Module appears to be effective solutions. 3. The results demonstrate that One-2345 enhances both the quality and efficiency of the single-image to 3D conversion. Additionally, the authors adeptly extend this to text-3D conversion using a pre-trained text-image model. The ablation studies provide strong evidence for the efficacy of the introduced modules and training strategies.

Weaknesses

1. The paper mentions the use of SparseNeuS for reconstructing 3D shapes from multi-view predictions by predicting blending weights. The colors of the 3D points are computed as the weighted sum of projected colors. However, since SparseNeuS usually takes real images as input and this work uses potentially inaccurate multi-view predictions, I am concerned that directly aggregating colors from these predictions could lead to blurry textures and artifacts. I suggest the authors investigate the effects of aggregating or refining colors more thoroughly. 2. There is a noticeable discrepancy between training and inference. During training, the 2-Stage Source View Selection module generates multiple nearby views from ground truth anchor views for reconstruction. However, during inference, there are no ground truth anchor views available, and Zero-123 must rely on its own predicted anchor views, which may not always be accurate. The paper does not address these challenges or offer solutions. Additionally, the “2-Stage Source View Selection and Groundtruth-Prediction Mixed Training” section could benefit from clearer explanations. 3. SparseNeuS employs post-optimization after the feed-forward pass. It would be helpful to know if One-2345 adopts a similar strategy, and, if so, how much improvement can be attributed to such a lightweight post-optimization. 4. The overall concept of integrating a feed-forward MVSNeRF with Zero-123 is not groundbreaking. Moreover, the Groundtruth-Prediction Mixed Training, though practical, lacks novelty. However, the motivation for employing a feed-forward approach to elevate Zero-123 to 3D shape reconstruction is sound.

Questions

see the strengths and weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper does not discuss limitations.

Summary

The authors tackle the problem of 3D reconstruction from a single image, which is challenging due to the lack of 3D information. They propose a novel method that combines a 2D diffusion model, Zero123, with a cost-volume-based 3D reconstruction technique, SparseNeuS, to generate a 360-degree 3D textured mesh in a feed-forward pass. They also estimate the elevation of the input shape and introduce several training strategies to improve the consistency and quality of the 3D mesh. Their contributions are: * A novel method that leverages 2D prior models for 3D modeling without per-scene optimization. * An elevation estimation module that computes the camera poses required by the reconstruction module. * A series of essential training strategies that enable the reconstruction of 360-degree meshes from inherently inconsistent multi-view predictions.

Strengths

* The work tackles 3D reconstruction from a single image, which is challenging and useful. The work can handle potentially any object category and generate a full 3D mesh from a single image. * The proposed method achieves high-quality 3D reconstruction in a feed-forward manner without optimization, which is faster and more efficient. The work does not require per-scene optimization but relies on a single feed-forward pass to generate the 3D mesh and as such it is much faster than existing methods. * The work leverages 2D prior models for 3D modeling, which is somewhat novel. It uses a 2D diffusion model to synthesize multi-view images of the input, and then uses multi-view 3D reconstruction techniques to obtain a 3D mesh.

Weaknesses

* The evaluation section is noticeably light. The authors used 20 shapes from GSO and Objaverse to report F-Score and CLIP similarity. The number of shapes (20) is too low for any meaningful comparison considering that the method was trained with 46k 3D assets. * The paper is framed that the work solves the image to 3D problem, which is true to some extend. In reality the work solve the problem of 3D lifting a single object, which is a much more constrained problem. I would advise the authors to change their wording to a more accurate description of the task. * The work mentions that it used 46k Objaverse assets for training, i.e. the whole Objaverse-LVIS dataset . All figures have Objaverse-LVIS assets and they keep repeating in the figures (backpack, super mario, horse, minion etc). A quick search show nearly all of them, for example this is the backpack: https://skfb.ly/6XCoS . All other can be found with search on the website https://objaverse.allenai.org/explore . On the contrary, there a minimal results on assets not from objaverse which rises the question whether the reconstructions we are seeing in the paper are assets from the training set. * The work has showcases a lot of licensed assets like pokemon, Super-mario and others. These assets require a licensing agreement with the respect companies that hold the rights. It might be sensible to replace them.

Questions

* Would it be possible to do a human evaluation with hundreds of 3D assets comparing your results with that of Shap-E? This would enhance the quality of the current evaluation. * Would it be possible to report numerical scores for hundreds of generated 3D assets rather than 20? To the same extend, can you report what the assets used were and whether they exist in Objaverse-LVIS? * Objaverse created a licensing issue when it came out with many artists requesting to retract their 3D assets from the dataset. Did the authors respect this by checking the no-AI flag in Sketchfab for both training and evaluation? + Comments in Weaknesses

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

Yes to some extend

Summary

The proposed method overcomes the challenges of lengthy optimization time, 3D inconsistency, and poor geometry that are common in existing methods. This method uses a view-conditioned 2D diffusion model, Zero123, to generate multi-view images from a single input image, and then lifts these images to 3D space. The 3D reconstruction is based on an SDF-based neural surface reconstruction method, with several training strategies proposed to enable the creation of 360-degree meshes. This method is faster, produces better geometry, and generates more 3D consistent results than existing methods. It has been evaluated on synthetic data and real-world images, demonstrating superior mesh quality and runtime. Additionally, it can be integrated with text-to-image diffusion models to support the text-to-3D task.

Strengths

The main idea of this paper is to combine the view synthesis method Zero123 with the multi-view stereo (MVS) method SparseNeuS for 3D generation. This simple approach provides interesting insights and results, as it enables fast 3D generation that visually outperforms sophisticated 2D diffusion+NeRF methods like Dreamfusion, and also appears to be superior to other data-driven 3D generation methods such as Point-E and Shape-E. The contribution of this paper is valid considering the advantages of runtime and better visual results.

Weaknesses

- While simple and effective, the idea of the paper is not that eye-opening in the sense that it seems to be a natural extension or improvement of Zero-123. - The performance of the proposed method is upper-bounded by the Zero-123. If Zeros-123 fails, it seems that there is no way for this method to generate reasonable output as well. In some sense, the problem of multi-view inconsistency is not solved but bypassed via the use of SparseNeuS. - The area is advancing fast. Follow-ups of DreamFusion such as Magic3D and Fantasia3D have already achieved big improvements. It's not sure how the proposed method compares with more recent 2DDiffusion+NeRF methods.

Questions

Going forward, which is a better path for 3D generation, novel view synthesis + mvs (like in this paper) or 3D data-driven (like in Shape-E)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The issue of multi-view inconsistency is not addressed but bypassed. It would be nicer to have a method that is more fundamental to addressing this issue.

Summary

This paper tackles the task of single image 3D reconstruction, where given a single input image of an object, the proposed method aims to generates a 360 degree 3D textured mesh. Compared to prior works tackling a similar problem setting, this paper primarily aims to improve the quality of the reconstruction and reduce the time costed during inference time. The proposed method contain 3 modules: (1) an off-the-shelf view-conditioned 2D diffusion model to generate multi-view images from a single input image, (2) a camera pose estimator, (3) an existing SDF-based multi-view reconstruction method to lift these images into 3D space. The method is evaluated qualitatively and quantitatively on synthetic and real data, and outperforms most baselines.

Strengths

Originality: - This paper is well positioned in the literature. It properly summarizes and tackles existing limitations of related works with a similar task setting. - This proposed method is a smart combination of existing methods, and the proposed design choices are well motivated. Clarity: - The writing is clear and easy to follow. Quality: - The qualitative and quantitative results outperform baselines and achieves SOTA. It still has a significant room for improvements. But it’s acceptable considering the challenges of the task. Significance: - The proposed method presents a novel idea for the literature -- incorporate a feed-forward model for diffusion-based methods.

Weaknesses

- The novelty of the proposed method is relatively limited, as it heavily relies on prior works. Specifically, two out of the three modules of the proposed method (Zero123 and SparseNeuS) are built upon existing approaches. However, the authors do demonstrate originality through their design choices and the underlying reasoning for the combination of existing ideas. - The proposed method contains several off-the-shelf or disconnected modules. It could have been beneficial to provide more discussion and analysis on how errors of one module affect the final performance. For example, pose estimation is a task that is not completely solved, how will the pose estimation error affect the pipeline? For inference of real world images, how will the inaccuracy of segmentation masks affect the final quality? - Although the quality of the final results achieved by the proposed method can be considered state-of-the-art, there is still significant room for improvement. From the qualitative results, most reconstructed 3D model is far from the quality of the input image. It can be helpful to expand the discussion of limitation to analyze why these artifacts can happen (e.g. blurriness, distorted shape).

Questions

In general I find this paper above the acceptance bar. The questions below aim to elicit additional clarification and insights of the paper: - If allowing an additional test-time optimization during inference time, will it further improve the quality of the outputs? (Or will it get harmed by the inconsistent multi-view images?) - The authors mentioned the proposed method suffers less from the multi-face issue (the Janus problem). Is it a benefit of the prior work Zero123 or from original design of the proposed method? - Minor: L.98 generates -> generate Description in L.150 “uses a spherical camera” and L.203 “a spherical camera model” is ambiguous.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper discusses failure cases in supp, including inconsistent multi-view images from Zero123, and artifacts on the back side. I encourage the authors to also explicitly mention in the main paper.

Summary

This paper introduces an efficient method for single-image 3D reconstruction that significantly improves upon previous techniques by generating textured mesh in one feed-forward pass. The authors employ a view-conditioned 2D diffusion model, Zero123, to create multi-view images from a single input view and then lift these images into 3D space using SparseNeuS. The proposed technique is much faster than existing methods, while reserving 3D consistency.

Strengths

The proposed method enables the generation of 3D meshes in one feed-forward pass after training. Due to its training-based, one-pass approach, the generation speed significantly outperforms many concurrent optimization-based methods. The quality of writing in the introduction and related work sections is commendable. The related work section covers the most recent work in the field and is well-organized.

Weaknesses

The authors propose a multi-stage method primarily built upon the combination of multiple existing works. The work seems more engineering-oriented than technically innovative. Many aspects of the methodology, such as elevation estimation, 2-Stage Source View Selection, and Groundtruth-Prediction Mixed training, focus on bridging various components borrowed from existing baseline models. Additionally, as the method relies on Zero123 and SparseNeuS, will it inevitably inherit their limitations? Can this method circumvent the failure cases of Zero123 or SparseNeuS? The method 12345 is trained on 1,156 categories from the Objaverse-LVIS dataset. How does 12345 perform when the images fall outside these 1,156 categories? Many concurrent image-to-3D works can generalize to uncommon objects due to their large diffusion models. Could 12345's training in aligning Zero123 and SparseNeuS diminish the zero-shot capability? The authors provide limited visualization cases in their ablation study. Including quantitative results to support the ablation study would be beneficial, as 3D generation is typically unstable and hard to reproduce. The organization and presentation of the methodology section could be improved. The method involves multiple stages, but the authors do not strictly adhere to the method structure order and insert discussions throughout. This makes sections 3.1 and 3.3 difficult to follow. From the visualization, the mesh generation quality is not so impressive. Could the authors explain their motivation for learning SDF and RGB from two separate MLPs? Many NeRF papers typically predict these from the same network.

Questions

See the Weaknesses above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Despite the concerns raised in the weaknesses section, a positive review score is given in recognition of the effort to formulate a single-pass pipeline and for the considerable credit on the superior generation efficiency.

Updates on color prediction experiments

Thank you for your insightful suggestions. We have previously initiated an experiment in which we employ a MLP to aggregate projected colors, with the aim of directly predicting point colors rather than their linear weights. Recent experimental results indicate that this approach may help mitigate certain artifacts, such as white spots that can occur when exporting meshes. We agree that pursuing more sophisticated strategies for color prediction holds promise as a valuable avenue for future research.

Response to the rebuttal

Thanks to the authors for the detailed rebuttal. I have no further questions at this point, and I hope the authors can improve their future version as promised. I have raised my score to weak accept.