Summary
The paper explores the feasibility of using procedurally synthesized datasets of 3D shapes to optimize existing large reconstruction models (LRMs), that procure 3D shapes in the form of NeRFs. The data is constructed by 1) sampling several shapes from a set of parametric primitives (cubes, spheres, tori, etc.) with random positions, scales, and rotations; 2) applying randomly sampled textures (from a preselected set of textures); 3) possible application of additional augmentations (changing the curvature of objects, and allowing concave and thin structures). Numerous experiments on existing Objaverse and proposed Zeroverse datasets show that given the correct degrees of the proposed augmentations, existing LRMs can be trained with the proposed synthetic data to produce comparable although slightly worse results, which suggests that existing scarcity of semantically rich 3D data might not be the main limiting factor for the development of reconstruction models since this data is not necessarily needed.
Strengths
* Since the experiments are very computationally demanding, the presented results are valuable (reproducing them is computationally prohibitive for most academic labs).
* A lot of ablation experiments explore the effects of the proposed augmentations suggesting the existence of an optimal amount of 3D shape augmentation for proper training of LRMs.
* The paper is well-written and easy to follow.
Weaknesses
* In terms of the metrics, models trained with the proposed synthetic data are worse but the authors claim that qualitatively it is hard to see the difference. At the same time, there are almost no qualitative comparisons of reconstructed objects from models trained on real/synthetic data in the main text, supplementary, or on the provided anonymous webpage. It would be better to show these and let the readers conclude if they are close.
* One of the main ideas of the paper that the LRMs do not require semantically rich realistic data for training is somewhat successfully challenged in the same paper, given that the LRMs trained for a longer time improve even further, increasing the gap between the proposed synthetic and conventional data.
* Another issue, mentioned by the authors is that the amount of the used augmentations for optimal performance may depend on the test dataset, so the domain gap between the synthetic data and considered test sets may differ, which limits, to some extent, the generalization ability of the proposed models.
Questions
Have the authors tried to perform any similar experiments for significantly different data synthesis pipelines? For example, the space of primitives can be extended with existing non-primitive shapes or with shapes generated by existing shape generation methods. If the semantics do not matter, the primitive set should not be crucially important, but at the same time non-primitive shapes used as base shapes will contain some relevant data patterns needed to capture high-frequency details in other non-primitive shapes from the test sets. Maybe this way, the authors would not need to find a set of perfect parameters for the augmentations to make the method work.
See limitations.
Limitations
While I do not think any ethics review is necessary for this paper, I think it may make sense to disclose the full computational budget for this project (for example in GPU hours spent) to warn any potential readers of the costs of such works.