In this work, we introduce Unique3D, a novel image-to-3D framework for efficiently generating high-quality 3D meshes from single-view images, featuring state-of-the-art generation fidelity and strong generalizability. Previous methods based on Score Distillation Sampling (SDS) can produce diversified 3D results by distilling 3D knowledge from large 2D diffusion models, but they usually suffer from long per-case optimization time with inconsistent issues. Recent works address the problem and generate better 3D results either by finetuning a multi-view diffusion model or training a fast feed-forward model. However, they still lack intricate textures and complex geometries due to inconsistency and limited generated resolution. To simultaneously achieve high fidelity, consistency, and efficiency in single image-to-3D, we propose a novel framework Unique3D that includes a multi-view diffusion model with a corresponding normal diffusion model to generate multi-view images with their normal maps, a multi-level upscale process to progressively improve the resolution of generated orthographic multi-views, as well as an instant and consistent mesh reconstruction algorithm called ISOMER, which fully integrates the color and geometric priors into mesh results. Extensive experiments demonstrate that our Unique3D significantly outperforms other image-to-3D baselines in terms of geometric and textural details.
Paper
Similar papers
Peer review
Summary
This paper introduces a novel method to reconstruct 3D from a single image. The method is two-stage: (i) 4 images corresponding to the orthographic views are first generated along with normal maps, (ii) then, the multi-view maps are used to initialize and optimize a mesh using differentiable rendering techniques which are adapted to this sparse-view setting. This second stage contrasts with recent image-to-3D methods which typically train large feed-forward reconstruction models. Extensive comparisons are conducted on (i) samples from original papers for a qualitative evaluation and (ii) on the standard Google Scanned Objects (GSO) dataset for quantitative results. In both cases, the proposed approach performs significantly better than the presented baselines.
Strengths
[S1] **Novelty.** The proposed approach is interesting and novel in different aspects. The first stage aims at generating multi-view images and normal maps from a single image; this idea is not novel per se and was introduced/studied by prior works like Wonder3D. However, it is worth noting that it technically differs a bit from Wonder3D and there is an additional effort to generate high resolution outputs. The second stage about 3D mesh reconstruction using differentiable rendering is elegant, effective and rather novel. In particular, it contrasts with recent image-to-3D methods which typically train large feed-forward reconstruction models tailored for this task. [S2] **SOTA performances.** For both the qualitative and quantitative evaluations, the proposed approach showcases results that are significantly better than prior works. Visually, the method not only generates more detailed textures but also more accurate geometry. [S3] **Sound experiments.** The authors conducted extensive experiments that are sound and validate the proposed method. They not only compare the method to an exhaustive set of state-of-the-art competitors both qualitatively and quantitatively, but also conducted ablation studies on some method components.
Weaknesses
[W1] **Unrigorous technical presentation.** While the high-level overview of the approach is clear, some parts of the technical presentation lack clarity and contain incoherences. For example: - Eq (1) looks wrong: a depth map can be obtained by integrating gradients of normals not the normals themselves - Eq (5) is incoherent and lacks clarity: is $i$ an integer or an image? If $\mathcal{I}$ is a set of images, $i \in \mathcal{I}$ is an image, but $V_M(v, img)$ is undefined. The rightmost part of the equation $\sum_i V_M(v, \mathcal{I})$ does not make sense either. This part need revisions, $i$ can not be at the same time: an integer indexing the ground-truth views $\mathcal{I_m}$, a sample view image from $\mathcal{I}$ or an integer indexing the sample views $\mathcal{I}$. I would define this process for a single target views $I_t$ given the source views $I_s^1, ... I_s^N$, and then define the loss in Eq (6) using the sum over the pseudo ground-truth target views. This lack of rigour harms readability and resolving these issues would greatly strengthen the paper quality. [W2] **Missing ablation studies.** I would expect more ablation studies to better understand the impact of each component that are different from prior works. Currently, the two ablation experiments correspond to two very technical aspects which are a regularization term (called Expansion) and the explicit target technique. Experiments assessing the impact of the following components are missing: - the image-to-multiview stage (e.g., compare results by replacing it with Wonder3D image-to-multiview stage) - the super-resolution (e.g., compare results with/without super-resolution network) - the multiview-to-3D stage (e.g., compare results by replacing it with Wonder3D multiview-to-3D stage)
Questions
Questions: - Eq (2): how is the ground-truth mask computed? - What happens when the input view does not correspond to the frontal orthographic view, e.g. say with elevation=20 and azimuth=45? - L171 what are the details behind edge collapse/split/flip? Remarks: - the terms "wild image", "wild views" do not sound correct, the appropriate term would be "in-the-wild image" but this typically corresponds to random real-world images that one could find on internet or social medias, which is not the case here. I would suggest removing this aspect which is not necessary for the paper storyline - the term "ExplicitTarget optimization" is not crystal clear out of context, I would suggest finding another term that clearly conveys what is under the hood, e.g. visibility-aware supervision
Rating
7
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
Yes, there is a brief limitation section. Including visuals illustrating failure cases would greatly help the readers better understand the model limitations.
Correction to Typo
Sorry for the messed up Latex in Equation (1) above, here is a corrected version. > Here $d(i,j)$ denotes the value of coordinate (i,j) in the depth map, while $x$ is the integration process along the vertical line $y=j$, $\vec{n}(x)$ denotes the vector of the input normal field at the position $x$. $n_x$ is the component of the normal field $\vec{n}$ along the direction of the x-axis (which is a scalar function). > The confusion might arise because the letter 'x’ is used in two different contexts. To clarify, we’ll make the following adjustments: We will use the formula $d(i,j)=\sum_t=0^i n_x(t,j)$, where `d` and `n_x` are considered as 2D arrays in the discrete version. This change aligns better with how the variables are handled in the code.
Thank You for Your Thorough Review and Invitation for Further Discussion
Dear Reviewer, Thank you for your patience and thorough review. We have addressed the issues you raised with detailed responses. We welcome you to engage in further discussion with us. Your insights are invaluable, and we are eager to clarify any points or provide additional information as needed. Looking forward to your feedback and any further questions you may have. Best regards, Authors
Summary
The paper focuses on single-image-to-3D. Given a single image, it first finetunes Stable Diffusion Image Variations models to generate orthogonal multi-view RGB/normal images and ControlNet-Tile models to enhance resolution. Then, it proposes the instant and consistent mesh reconstruction algorithm (ISOMER). It has three stages: 1) Use the estimated depth (from predicted normal maps) of the front and back views to initialize the geometry. 2) Use differentiable rendering to optimize geometry with mask and normal loss. 3) Optimize geometry and vertex colors to fit the view-weighted color, where each view’s weight is the square of cosine between vertex normal and view direction. The total time required is claimed to be within 30 seconds. It compares with some recent approaches on 30 GSO objects and achieves better performance in terms of both visual and geometry quality.
Strengths
- **High-Resolution Appearance at Input View:** It demonstrates improvements in the resolution of the generated appearances, especially at the input view. Unlike previous works that produced lower-resolution multi-view outputs, resulting in a lack of clarity and detail, this paper utilizes diffusion models to achieve effective super-resolution. This allows the generated views to maintain high resolution and exhibit more details (reimagined though). - **Enhanced Geometry:** The paper showcases impressive geometry, particularly in examples of garage kit figures. Technically, this work better leverages predicted normal map information to optimize geometry and color, surpassing previous models like Wonder3D, which also predicted normal maps. This enhanced utilization of normal maps carves more geometric details. - **Many Details and Supporting Materials:** It provides many network parameters, predicted meshes, and rendering videos, which are valuable for assessing the performance and effectiveness of the proposed methods.
Weaknesses
- **Limited Qualitative Results to Frontal View, Worse Side Views:** The majority of qualitative results presented in the paper are focused on the frontal view, as it is prioritized by the approach. However, for non-frontal views, there are issues such as visible seams and artifacts. For example, the video in supplementary material shows several instances where characters have duplicated ears. Although the frontal view quality is enhanced, this improvement comes at the expense of other views, which suffer from more noticeable artifacts. - **Potential Bottleneck by the Two-view Initialization of First Step** Following the previous point, those artifacts are potentially influenced by the algorithm’s initialization process since only front and back views are used. Artifacts like dual-ear, if introduced in the first step of ISOMER, may not be easily corrected via subsequent refinement. What if the input view is not exactly frontal and more ill-posed? The artifacts can be even more severe. - **Narrow Evaluation Scope, Limited Geometry Complexity:** The paper predominantly evaluates small cartoon figures and animal models in terms of qualitative evaluation, with limited representation of real-world objects encountered in daily life. Real-world objects may have more complex geometry, where four orthogonal views are not enough to cover. Even for simple objects like a mug, the method may fail to generate a solid bottom and an empty interior given that they are unseen. - **Pose Sensitivity and Distortion Issues:** The network outputs exhibit significant distortions when input images are not taken from a frontal view with the elevation equal to zero. For instance, a machine example in the supplementary video shows clear distortion when the input image is taken from an elevated side angle. This indicates a lack of robust data augmentation during training, leading to heavy restrictions on input poses. - **Oversimplified Texture Handling of Occluded Region:** The approach to unseen vertex coloring is overly simplistic, using flood-fill colors to interpolate occluded areas. This may result in incorrect color transitions for complex geometries with significant occlusions. Even for simple cases, the bottoms or tops of objects are often unseen from 4 orthogonal views. Vertex colors in those regions are roughly smoothed out. - **Small Evaluation Set:** The evaluation set consists of only 30 objects, which is insufficient, especially considering all of those baselines are not purely optimization-based, whose inference time costs per image are no longer than a few minutes. A few hundred objects will be necessary. For example, GRM [63] uses 250 shapes for single-view reconstruction evaluation, providing more robust and reliable assessments. - **Unclear Writing:** The paper contains several sections where the writing is unclear. Please refer to the points listed in the Questions. - **Lack of Quantitative Ablation Studies:** It does not include any quantitative evaluation studies.
Questions
1. Please properly cite, acknowledge, and compare with previous works if similar ideas are applied. For example, in the Multi-view Image Generation, two key points (L499-502) seem to be the same as ImageDream [15]. And, it also uses IP-Adapter (L515). A reference U-Net is also used in Zero123++ [46]. Please cite related work and explain the differences when introducing the proposed method. This helps readers understand how ideas are inherited and improved. 2. No notations in Eq. 1 are explained. I can understand the point. But it is not reader-friendly. Please delete it or explain it. The formula itself is also over-simplified. 3. Wrong references of figures, and algorithms. L181, L188, L557, etc. 4. In Algo. 1, the variable cnt is useless and not explained. The set ‘colored’ is initialized as empty but assumed as initialized by C (L13). The authors may consider putting it more clearly: For invisible vertex coloring, it applies the flood fill algorithm, where each vertex is colored by the average color of the neighboring colored vertices. Use proper terms and make your writing concise. 5. Is Expansion (L181) referring to applying normal-based loss L_normal? 6. L204: the result for vertex v → the predicted color of vertex v. Make it more clear. 7. Time cost of each ISOMER step.
Rating
5
Confidence
5
Soundness
3
Presentation
1
Contribution
2
Limitations
Many significant limitations of the approach are not discussed. Please refer to the Weaknesses.
Responses to questions
Question 1: > We greatly appreciate your insightful feedback. In light of your suggestions, we will revise the introduction in the updated paper to clearly delineate our approach. Similar to 3D generation methods like Wonder3D and InstantMesh, our work draws inspiration from ImageDream and Zero123++ for multiview image generation. However, we employ two denoising models and a super-resolution model to accomplish multi-view generation with notable distinctions as follows: > (a) Orthogonal View Generation: Our first denoising model, similar to ImageDream, generates four orthogonal views instead of perspective views, which simplifies the subsequent reconstruction process and enhances multi-view consistency. The choice of orthogonal views facilitates a direct geometric correlation between pixels. Moreover, we integrate learnable class embedding to encode view information, enhancing the model's ability to understand and process multi-view inputs. > (b) IP-Adapter Utilization: Contrary to being used in the Multi-view Image Generation step, our method employs an IP-Adapter in the Multi-view Image Upscale step. We incorporate a controlnet model with IP-Adapter to allow for the enhancement of multi-view details and achieve the targeted resolution. > (c) Normal Prediction Setting: In the Normal Prediction module, we adopt a denoising model with a reference U-Net, akin to the concept in Zero123++. However, unlike Zero123++, which shares all weights between the reference U-Net and the main network, we utilize an independent pre-trained reference U-Net. We freeze all parts within the reference U-Net except for self-attention (Lines 544-545) to preserve the generalization ability from the pre-trained model. Question 2: > Thanks a lot for pointing out! Here $d(i,j)$ denotes the value of coordinate $(i,j)$ in the depth map, while $x$ is the integration process along the vertical line $y=j$, $\\vec{n}(x)$ denotes the vector of the input normal field at the position $x$. $n_x$ is the component of the normal field $\\vec{n}$ along the direction of the x-axis (which is a scalar function). > The confusion might arise because the letter ‘x’ is used in two different contexts. To clarify, we’ll make the following adjustments: We will use the formula $d(i,j)=\\sum_{t=0}^i n_x(t,j)$, where `d` and `n_x` are considered as 2D arrays in the discrete version. This change aligns better with how the variables are handled in the code. Question 3: > Thanks for pointing out the issue. We will correct these mistakes in the revision. Question 4: > We thank the reviewer for raising this question, and we deeply regret the lack of detailed comments on the appendix's algorithm which has caused confusion. The variable cnt is crucial as it determines the number of while loop iterations at Line 11 (L11). It is used to record the number of iterations before reaching Line 21 (i.e., all colors are applied), and afterward, the iteration repeats cnt times to guarantee the color completion process is completed. > Besides, we acknowledge the mistake at Line 3 (L3), where the initialization of 'colored' should indeed be based on the Inv. In the invisible vertex coloring, a flood fill operation is applied on the array 'colored' to fill the colors, while an ongoing laplacian smoothing is performed on the array C[i] to achieve a smoother color transition for a more aesthetically pleasing result. We appreciate your correction and will ensure future versions have clearer documentation to prevent misunderstandings. Question 5: > The "Expansion" in our model serves as a regularization technique directly applied to the parameters, similar to weight decay, rather than functioning as a loss term. At each step, vertices are moved a small distance in the direction of their normals. Question 6: > Thanks for pointing out the issue. We will modify this statement in our revision following your suggestion. Question 7: > Thanks for your thorough comment. The entire ISOMER process takes approximately 10 seconds. Within this timeframe, the Mesh Initialization step accounts for about 2 seconds, the preliminary Mesh Reconstruction takes around 3 seconds, the Mesh Refinement step consumes approximately another 5 seconds, and the final Mesh Colorization is completed in less than 0.1 seconds. We appreciate the reviewer for pointing out those issues. We will thoroughly revise our paper. Thanks again for your time and in-depth suggestions.
Thank You for Your Thorough Review and Invitation for Further Discussion
Dear Reviewer, Thank you for your patience and thorough review. We have addressed the issues you raised with detailed responses. We welcome you to engage in further discussion with us. Your insights are invaluable, and we are eager to clarify any points or provide additional information as needed. Looking forward to your feedback and any further questions you may have. Best regards, Authors
Summary
This paper proposes a novel method for converting a single image to 3D. The method mainly consists of two stages: multi-view RGB and normal generation, and multi-view guided mesh optimization and texturing. The key innovation of the paper is a multi-view-normal-based 3D mesh reconstruction module. Specifically, given a single input image, the method first generates multi-view RGB images by fine-tuning a 2D diffusion model. It then employs a ControlNet-tile model and a super-resolution model to increase the resolution of the multi-view images from 256 to 2048. Additionally, a multi-view normal model is fine-tuned to generate the corresponding multi-view normal maps. Following this, a module called ISOMER is proposed to first convert the multi-view normal maps into a 3D mesh by fusing the normal maps to depth and then applying Poisson surface reconstruction. The mesh geometry is then refined according to a specialized multi-view normal loss. Finally, the mesh vertex color is optimized through a similar approach.
Strengths
1. The proposed method is well-motivated. Previous methods attempt to directly convert multi-view RGB images to 3D through a feed-forward model, which is harder to train and requires more computing resources. The proposed method leverages the multi-view normal information from a 2D diffusion model to directly optimize 3D geometry, avoiding expensive training. 2. The proposed multi-view normal to 3D module is novel and interesting. 3. The paper is generally well-written and easy to follow.
Weaknesses
1. While I agree that the proposed method is technically sound and may be a nice supplement to the community, my major concern is that the experimental section is quite lightweight. For instance, the quantitative experiment is only conducted on 30 objects from the GSO dataset. Since the proposed method and the comparing baseline are not expensive to run compared to SDS-based methods, it is not acceptable to base quantitative results on only 30 objects. It's very easy to get biased results, and the conclusions are likely to change if we choose another set of 30 objects. As a result, I strongly urge the authors to follow the conventions of previous papers and rigorously compare the methods on the entire GSO dataset with careful alignment between prediction and ground truth before calculating metrics. Additionally, it is highly suggested to include evaluations on other datasets, especially some real-world object datasets, and user studies. Otherwise, I cannot support the acceptance of the paper without convincing justification. 2. The high-level pipeline of "multi-view normal generation with 2D diffusion models and 3D reconstruction with normal-based optimization" is not first proposed in this paper. For example, Wonder3D shares a very similar high-level pipeline, which greatly limits the contribution or novelty of the paper. While the paper proposes a novel and interesting reconstruction module, ISOMER, it has not been carefully analyzed separately. For instance, I would like to know whether the performance gains come from better multi-view prediction or a better multi-view normal-to-3D reconstruction module. An interesting experiment to include would be a direct comparison between the reconstruction modules of Wonder3D and ISOMER given the same multi-view normal maps. Only after a detailed ablation study can readers choose a better multi-view normal generation module and a better reconstruction module. 3. The multi-view generation part lacks significant novelty, mainly consisting of existing known techniques related to 2D diffusion models. 4. The ablation study is quite lightweight, and many important experiments are missing: (a) The ISOMER module includes multiple stages. I suggest the authors include both the intermediate and final results of the module. A quantitative evaluation would also be beneficial. (b) The method utilizes multiple 2D diffusion models to upscale the multi-view RGB (and normal?) images. It would be interesting to see the multi-view results before and after upscaling (resolutions at 256, 512, and 2048) and their impact on the final 3D models. (c) For the "explicit target," only the geometry results are shown. What is the effect on texture? 5. The paper claims that "generate tens of millions of faces within seconds" as an advantage. However, I don't believe so. First, it's not difficult for existing methods to generate dense meshes efficiently. Also, dense meshes themselves are not required by applications but rather detailed and sharp geometry. In fact, many downstream applications prefer meshes with more compact faces and cannot tolerate tens of millions of faces. 6. Line 170 states, "Finally, the mesh is corrected after iteration through edge collapse, edge split, and edge flip to maintain a uniform face distribution and reasonable edge lengths." This introduction is too brief as it involves many operations but lacks the motivation and implementation details for each operation.
Questions
1. When converting normal maps to depth maps, how do you handle discontinuity issues? For example, there may be occluded regions, leading to sudden jumps in the normal map. 2. The method uses two models to upscale the resolution, first from 256 to 512 and then to 2048. Do we really need two models? Can we directly upscale from 256 to 1024? Or can we change their order? 3. It seems that the mesh vertex colors are directly computed as a weighted sum of the projected colors. Am I correct? Will this cause any inconsistency issues or other artifacts? 4. Line 133: "we adopt a channel-wise noise offset strategy" is not very clear to me. Could you provide more details? 5. Line 160: "not yield a real normal field which is irrotational. To address this ..." is not clear to me. Could you explain this further? 6. Line 181: What does Figure 3(b) refer to? 7. Line 224: "By examining the epipolar lines corresponding to each horizontal ray, we identify 13k instances of illegitimate data." This is not very clear to me. Could you provide more details? 8. Line 241: What does "second level of training" refer to?
Rating
6
Confidence
5
Soundness
2
Presentation
3
Contribution
3
Limitations
The authors briefly mention the limitations in the final section.
More Responses
We will add more details about edge collapse, edge split, and edge flip in the revision as follows. >Edge Collapse: This operation is used to avoid and heal defects in the mesh. It involves selecting an edge within a triangle and collapsing it to the other edge, effectively merging the two triangles into a single triangle. This process can help to eliminate narrow triangles that might be causing issues in the mesh, such as those that are too thin to accurately represent the surface they are approximating. Edge collapse can prevent the creation of topological artifacts and maintain the quality of the mesh. > Edge Split: This is the opposite of edge collapse. In edge split, an edge that is longer than a specified maximum length is divided into two, creating new vertices at the midpoint of the edge. This operation is used to refine the mesh, ensuring that the local edge length is kept close to the optimal length. It helps to maintain the quality of the mesh by avoiding edges that are too long, which could lead to an inaccurate representation of the surface. > Edge Flip: Edge flip is an operation that adjusts the connectivity of the mesh to improve its quality. It involves flipping an edge within a triangle to connect two non-adjacent vertices, effectively changing the triangulation of the mesh. This can help to maintain the degree of the vertices close to their optimal value, which is typically six for internal vertices (or four for boundary vertices). > The goal of these operations is to improve the mesh quality while avoiding defects and ensuring that the mesh accurately represents the target geometry. Question 1: > Thanks for your questions. The edges of sudden jumps will have a steep normal, and integrating over this normal gives a large depth difference. So this does not have an observable negative impact on the algorithm. Question 2: > Thanks for your insightful questions. The 256-to-512 model is specifically tuned to integrate information from multiple views, which is crucial for ensuring consistency across different perspectives. The 512-to-2048 model, on the other hand, is optimized to concentrate on the finer details of the reconstruction, enhancing the overall quality of the output. Since the multi-view aware 256-to-512 bears a relatively higher computational load compared to the 512-to-2048 model, it is part of our strategic design to balance computational load with multi-view consistency and accuracy. This dual-model strategy is crafted to enhance efficiency in both training and inference, making our method more practical for real-world applications. Question 3: > Yes. Since it is computed as a weighted sum, we rarely encounter inconsistency issues. However, It is indeed a promising direction to explore more advanced coloring methods in future work to further enhance the robustness and quality of our results. Question 4: > Thanks for pointing this out, we'll be more detailed in the revised version! For a noisy latent with shape [B, C, H, W], we further add a [1, C, 1, 1] shaped N(0, 0.1) gaussian noise to it, thus enhancing the generalization of the network and avoiding the zero terminal SNR problem. Question 5: > We thank the reviewer for raising this concern. For a legitimate normal field, any closed line integral should be zero, which is the meaning of being irrotational. However, predictions from neural networks cannot possibly meet this condition. Question 6: > Thanks for pointing out the typo. We will correct it to Figure 5(b). Question 7: > Thanks for your valuable comments. We will add more details as follows. > Because it is four views that are in the same plane, there is an obvious pairwise polar geometric relationship, i.e., any pixel in one view corresponds to a horizontal straight line in the neighboring view. Thus if a non-null pixel, does not have any non-null pixels on the corresponding straight line in the neighboring view, then the data is ilegal. This problem is usually caused by objects in the data that have no thickness, i.e., they are observable in one view and happen to be invisible in another view. Question 8: > It refers to Multi-view Image Upscale, with a detailed description in Appendix B. We thank the reviewer for pointing out these typos. We will carefully proofread the manuscript and sincerely hope that you will find the revision satisfactory. We appreciate your time and insightful comments.
thank you
Thank you for the detailed response and additional experiments. After reviewing your comments, I have the following questions and concerns: 1) It appears that Wonder3D+ISOMER performs slightly better than Wonder3D alone but still falls short (a lot) of Unique3D. Does this imply that the primary improvement in Unique3D comes from the multi-view generation (2D diffusion models) rather than the reconstruction model (ISOMER)? Could you explain the significant performance gap between Wonder3D+ISOMER and Unique3D? Additionally, would combining Unique3D's multi-view prediction with Wonder3D's reconstruction model result in better performance than "Wonder3D+ISOMER"? I'm asking because I want to understand whether the improvement is due to the reconstruction method or the multi-view prediction. Based on the current results, it's difficult to determine. 2) The authors claim that their multi-view module offers higher resolution, better multi-view consistency, and faster generation speed. However, the last two points are not supported by experiments. How do you quantitatively measure multi-view inconsistency? 3) When directly calculating the point color as a weighted sum of projected 2D pixels, why doesn't this method suffer from inconsistency, especially at the boundaries or overlaps between multiple views? 4) I still don't fully get the point. When there is a sudden depth change (e.g., due to occlusion), how can simply integrating the normals yield the correct depth? I don't think there would be steep normals; rather, there should be multiple segments of normals reflecting the surface properties of each region separately. 5) Regarding the generation of dense meshes, I agree that generating sharp features requires a sufficient number of faces. However, the ability to export a large number of triangles doesn't necessarily indicate that the method can generate sharp and detailed geometry. For instance, existing methods can increase their resolution to 512 or 1024 when using the Marching Cubes algorithm, which will produce many more triangles, but the underlying geometry remains unchanged. My point is that you should only claim the generation of sharp details (with verification) as an advantage, not just the generation of a large number of triangles. 6) Where is the "Quantitative Evaluation of the ISOMER Module"? I couldn't locate it in the rebuttal PDF. 7) Regarding the "irrotational normal", what is the motivation behind "introducing a random rotation to the normal map before integration? The process is repeated several times, and the mean value of these integrations is then used to calculate the depth, providing a reliable estimation." How does this solve the problem? 8) In the rebuttal, the qualitative examples provided are limited to one or two instances, which is not very convincing or helpful for understanding. Please avoid this and include more examples in your revision.
Response to questions
We sincerely appreciate your constructive and thorough questions. To address your concerns, we present the point-to-point response as follows. 1. It appears that Wonder3D+ISOMER performs slightly better than Wonder3D alone but still falls short (a lot) of Unique3D. Does this imply that the primary improvement in Unique3D comes from the multi-view generation (2D diffusion models) rather than the reconstruction model (ISOMER)? Could you explain the significant performance gap between Wonder3D+ISOMER and Unique3D? Additionally, would combining Unique3D's multi-view prediction with Wonder3D's reconstruction model result in better performance than "Wonder3D+ISOMER"? I'm asking because I want to understand whether the improvement is due to the reconstruction method or the multi-view prediction. Based on the current results, it's difficult to determine. > Thank you for your question. We tested the Unique3D multi-view + Wonder3D reconstruction, and the results are as follows: | Method | PSNR↑ | SSIM↑ | LPIPS↓ | Clip-Sim↑ | Chamfer Dist.↓ | Vol. IoU↑ | F-Score↑ | | --- | --- | --- | --- | --- | --- | --- | --- | | Wonder3D | 18.0932 | 0.8995 | 0.1536 | 0.8535 | 0.0261 | 0.4663 | 0.6016 | | Wonder3D+ISOMER | 18.6131 | 0.9026 | 0.1470 | 0.8621 | 0.0244 | 0.4743 | 0.6088 | | Unique3D+Wonder3D | 19.1688 | 0.9219 | 0.1107 | 0.8732 | 0.0153 | 0.5232 | 0.6576 | | Unique3D | 20.0611 | 0.9222 | 0.1070 | 0.8787 | 0.0143 | 0.5416 | 0.6696 | > We analyzed the results: > 1. Our multi-view training part differs from Wonder3D in data filtering, training strategies, and network architecture. For example, about 37% of the data in Wonder3D's data list cannot pass our data filtering (mainly due to issues like thicknessless surfaces, one-sided visibility, or too small projected areas), which may have led to different learned preferences. > 2. We also observed that the main reason for Wonder3D's low scores is the overly flat or thick predictions (often problematic with elongated objects like shoes or flat objects like books), which is a significant factor in Wonder3D's lower scores. Such errors are less observed in our multi-view predictions. > Based on the above analysis, it is consistent with your prediction. It can be said that the main improvement of qualitative comparisons comes from the accuracy of multi-view prediction. These indicators are not sensitive to geometric details and are more focused on evaluating basic geometric structures. Therefore, ISOMER has relatively limited improvement under these indicators, but we have observed that the visual effect of Wonder3D+ISOMER is much better than Wonder3D. We are also actively looking for a better metric to evaluate geometric details. 2. The authors claim that their multi-view module offers higher resolution, better multi-view consistency, and faster generation speed. However, the last two points are not supported by experiments. How do you quantitatively measure multi-view inconsistency? > Thank you for your insightful question. The faster generation speed refers to the comparison between generating multi-view at 512 resolution directly and combining the generation at 256 resolution with upscaling to 512. Theoretically, the computational load of the latter is reduced by 55% compared to the former (since the computational load of 512 is four times that of 256). Regarding multi-view consistency, it is compared to the directly apply super-resolution from 256 to 2048. Since directly apply super-resolution does not include information from other views, it introduces more inconsistencies. We will clarify this in the revised version to avoid any ambiguity in understanding. 3. When directly calculating the point color as a weighted sum of projected 2D pixels, why doesn't this method suffer from inconsistency, especially at the boundaries or overlaps between multiple views? > This is why the reconstruction method includes a Reconstruction Stage and a Refine Stage in our method. The Reconstruction Stage, which does not contain ExplicitTarget, quickly produces a model that approximates the correct shape but is limited by multi-view consistency. ExplicitTarget further addresses this issue in the Refine Stage. A straightforward understanding can be seen in Fig. 1 of the Rebuttal material, where ExplicitTarget is used for direct coloring. If the coloring is replaced with multi-view normal maps, it becomes the optimization target for each step in the Refine Stage. Without ExplicitTarget, the optimization would face inconsistencies due to multiple views, whereas ExplicitTarget does not have this problem. At the contour edges of each view, the weights used in the ExplicitTarget calculation are based on the angle between the normals, so that the typically lower weights at these edges. Additionally, since the adjacent views are orthogonal in a four-view setup, a vertex's color is mostly influenced by only one view, preventing incorrect superposition.
Remaining responses
4. I still don't fully get the point. When there is a sudden depth change (e.g., due to occlusion), how can simply integrating the normals yield the correct depth? I don't think there would be steep normals; rather, there should be multiple segments of normals reflecting the surface properties of each region separately. > The training data includes steep edge normals (the multi-view normals in the data are derived from depth maps, as direct surface normals would yield incorrect data due to many surfaces being wrong direction in Objaverse). Therefore, we observe such normals during generation, although they may not be accurate (as shown in Appendix Fig. 8). However, the accuracy of the initial depth estimation is not crucial for the final result (as demonstrated in Appendix Fig. 9); what matters is the topological holes, not the accuracy. The topological holes are reflected by the accuracy of the normal maps. We expect to explore more accurate initialization methods to improve the final method's accuracy in the future. 5. Regarding the generation of dense meshes, I agree that generating sharp features requires a sufficient number of faces. However, the ability to export a large number of triangles doesn't necessarily indicate that the method can generate sharp and detailed geometry. For instance, existing methods can increase their resolution to 512 or 1024 when using the Marching Cubes algorithm, which will produce many more triangles, but the underlying geometry remains unchanged. My point is that you should only claim the generation of sharp details (with verification) as an advantage, not just the generation of a large number of triangles. > We fully agree with your statement. A large number of triangles are merely a prerequisite for sharp details, not indicative of sharp details themselves. As shown in Fig. 3 of the main text, existing methods fail to achieve good sharp details in geometry, largely because they use methods like Marching Cubes to extract meshes. For example, replacing the reconstruction in Wonder3D with ISOMER improves the results. Methods like OpenLRM, CRM, and Instant-Mesh use no more than $384$ resolution Marching Cubes or Flexible Cubes, as these algorithms require evaluating $512^3$ SDF values at $512$ resolution, which needs over 24GB of GPU memory and nearly a few minutes of runtime. A $1024$ resolution Marching Cubes theoretically takes over ten minutes and >100GB of GPU memory. In contrast, ISOMER can achieve sharp detail reconstruction in just a few seconds. 6. Where is the "Quantitative Evaluation of the ISOMER Module"? I couldn't locate it in the rebuttal PDF. > The "Quantitative Evaluation of the ISOMER Module" refers to the indirect quantitative comparison between Wonder3D and Wonder3D + ISOMER. We will add a direct comparison based on ground-truth multiview normals in the revised paper. Compared to the ground-truth multiview normals, we believe the input for the reconstruction operation should be the generated multiview from the front view rather than the accurate ground-truth multiview. Therefore, we consider comparing using multi-view normal generated Wonder3D, which has inconsistencies across multiple views, to be more relevant to the actual task of this reconstruction algorithm. 7. Regarding the "irrotational normal", what is the motivation behind "introducing a random rotation to the normal map before integration? The process is repeated several times, and the mean value of these integrations is then used to calculate the depth, providing a reliable estimation." How does this solve the problem? > "Irrotational normal" implies that the integral values along different paths should be the same, but since the predicted "normal map" is not "irrotational," these integral values actually differ. We choose straight lines as the integration path because it is the most straightforward to operate. Different straight lines yield different integral values, so we take their expectation as the final integral value. This is why random rotations are needed to calculate the mean value. We plan to explore using a direct depth prediction model to accomplish this task instead of calculating depth from normals in the future. 8. In the rebuttal, the qualitative examples provided are limited to one or two instances, which is not very convincing or helpful for understanding. Please avoid this and include more examples in your revision. > Thank you for the suggestion! We agree with you, but due to the limited time for the rebuttal and the addition of extensive experiments and code, many experiments were not fully completed. We will include at least six representative examples for each qualitative experiment in the revision to enhance the paper. Thank you very much for your insightful questions and suggestions, and we welcome further discussion!
Summary
This paper introduces Unique3D, a framework aiming to generate 3D meshes from single-view images with high quality and fidelity. Driven by the observation that 2D image pixel and normal priors with higher resolution can be crucial in generating intricate textures and complex geometries, Unique3D integrates a multi-level upscale process to progressively improve the resolution. It also proposes an instant and consistent mesh reconstruction algorithm called ISOMER to lower the computation complexity and improve reconstruction quality.
Strengths
Unique3D addresses the limitations of previous methods by generating high-resolution 2D images and normal maps and optimizing the color mesh according to these 2D guidances. The idea of increasing resolution in 2D is straightforward and powerful as the experiments suggest. The pipeline is efficient and can generate mesh within a short time frame (about 30 seconds per mesh). This makes it practical for many real-world applications. The paper provides solid and detailed results for the effectiveness of its methods and delivers rather convincing results. The proposed ISOMER is novel and could inspire further studies in object color mesh reconstruction. The paper includes a thorough analysis of the ISOMER algorithm which provides good insights into the module.
Weaknesses
1. This pipeline features an image diffusion model, two super-resolution modules, and a normal diffusion model. All work in a sequential manner without interaction. This may bring significant compounding errors in multi-view/multi-resolution consistency before reconstructing mesh with ISOMER. The author should include a more thorough analysis on error patterns with particular focus on multi-view/multi-resolution consistency. 2. The subsection "ExplicitTarget Optimization for Multi-view Inconsistency and Geometric Refinement" (L183-) is hard to read and understand. Improvements in writing and equations are needed. 3. Based on my understanding of the ISOMER algorithm, it could be prone to inconsistencies, especially around the 2D boundaries of the generated multi-view images. Could you provide more insights into this aspect? Also, how well is the model working with non-front-facing input views? Adding some discussion for these could further improve the coverage of the experiments and help people understand the limitations. 4. 30 random objects (L257) are too few to be a meaningful quantitative evaluation.
Questions
See weakness 1, 3, 4.
Rating
6
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
Limitations are discussed very briefly in the conclusion section. Addressing the questions in the above weaknesses could involve further discussions on the limitations. Societal impacts are included in the supplementary materials.
Thank You for Your Thorough Review and Invitation for Further Discussion
Dear Reviewer, Thank you for your patience and thorough review. We have addressed the issues you raised with detailed responses. We welcome you to engage in further discussion with us. Your insights are invaluable, and we are eager to clarify any points or provide additional information as needed. Looking forward to your feedback and any further questions you may have. Best regards, Authors
Thanks for the detailed rebuttal! Comment 1/2: If the error mainly originated from multi-view inconsistency, why are there more seams and artifacts like dual ears at the sides instead of on the front view? Comment 3: It was stated that the unseen features pose a challenge for all methods. However, I don't fully agree with this. For example, I think methods like CLAY (Rodin Gen-1) or One-2-3-45++ can address this issue effectively with 3D supervision, so such challenges wouldn't arise in these cases. Comment 4: Are the input views in the test rendered at elevation=0? Are rotations only horizontal? Currently, this method shows noticeable distortion given non-zero elevation input views, which doesn't occur in works like InstantMesh.
Respones to More Questions
Comment 1/2: > Thank you for your insightful observation regarding the presence of seams and artifacts. As mentioned, our code implementation was inspired by the Wonder3D's code, where we assigned different weights to different views during the ExplicitTarget calculation: 2.0 for the front, 1.0 for the back, and 0.8 for the other sides. This weighting strategy helps to mitigate artifacts on the front view, aligning more closely with human preferences by maintaining higher quality in the most visible areas. We believe this approach enhances the overall visual quality, especially in cases where multi-view images are inconsistent. Comment 3: > Your perspective on the handling of unseen features by methods like CLAY (Rodin Gen-1) is insightful. Indeed, CLAY's explicit 3D representation and explicit 3D supervision effectively address the issue of unseen features, producing robust structures in invisible areas. While CLAY may not ensure high consistency with the input views, it excels in generating detailed unseen regions. Integrating such strengths into our future work is a promising direction for us. > Regarding One-2-3-45++, its non-open-source nature prevents direct comparison. However, strikingly similar to One-2-3-45++, Instant-Mesh also employ Zero123++ for six-view generation and subsequent SDF generation with multi-view differentiable rendering supervision, face challenges such as generating "a solid bottom and an empty interior of a mug". For instance, Instant-Mesh fails to produce a empty interior of a mug due to the inability to observe the interior from surrounding views. This limitation suggests that One-2-3-45++ might also struggle with similar issues, as it does not inherently circumvent the problem of unobserved regions. > CLAY is an excellent job that can't be ignored. However, it is important to note that CLAY was published on arXiv a week after our submission deadline, hence it was not included in our current discussion. Our work focuses more on geometric details and external visual effects of objects, contrasting with CLAY's emphasis on overall structural generation. We will incorporate a discussion of these related works into the refined version of our paper. Comment 4: > We acknowledge your concern about the potential distortion observed in non-zero elevation input views. In the rebuttal appendix, additional tests on object rotation sampling follow $azimuth \in U[-180, 180], elevation \in U[-30, 30]$, rather than $elevation=0$. We have examined the test cases and did not find significant distortion in the non-zero elevation input views. We will supplement the multi-view visualization results under these conditions in the revised version. It is worth noting that if random rotations are not included during model training, the trained model will exhibit noticeable distortion in the non-zero elevation input views. Thank you for pointing this out! Thank you very much for your questions and suggestions, and we welcome further discussion!
Further discussion on distortion and elevation
Comment 4: Pose Sensitivity and Distortion Issues: "For instance, a machine example in the supplementary video shows clear distortion when the input image is taken from an elevated side angle." > We have thoroughly reviewed the "a machine example" you mentioned and have not found any significant distortion in the generated results. The input view of this example has an azimuth of approximately 60 degrees and an elevation of about 15 degrees. The generated model also precisely aligns with these angles, exhibiting an azimuth of around 60 degrees and an elevation of 15 degrees, which perfectly matches the rotation angles of the input object. Furthermore, the generated geometric structure is entirely consistent with the input image. On the contrary, this example actually demonstrates the robustness of our model in handling objects with rotation.
Thanks for the rebuttal
Thanks for the detailed answers. I have carefully read the rebuttal and my concerns were mostly addressed. One missing aspect is a discussion about failure cases and I strongly recommend adding some in the revised paper. In general, such an analysis is easy to build, it provides strong insights about the method performances and it can really drive the next iterations. Overall, I will keep my rating.
Thank You
Dear Reviewer, We sincerely appreciate your thoughtful feedback and the time you have taken to carefully review our rebuttal. We are pleased to hear that our detailed responses have addressed most of your concerns. Regarding your suggestion to include a discussion on failure cases, we fully agree with the value such an analysis would bring to the paper. It not only enhances the robustness of our method but also provides crucial insights for future improvements. In the revised version, we will incorporate a comprehensive analysis of failure cases. This will include: * Analysis of inconsistent predictions across views. * Generation of geometric structures in unseen areas. * Simple colorize algorithm. * Inaccurate predicted normal maps. We believe that this detailed analysis will significantly strengthen our paper and contribute to the advancement of the field. Once again, thank you for your constructive feedback. We look forward to incorporating these improvements and hope to meet your expectations in the revised manuscript. Best regards, Authors
Thanks for the response. I find most of my concern addressed and will keep my score leaning towards acceptance.
Decision
Accept (poster)